texts: How to alter word2vec wikipedia model for n-grams?

texts:
I have a very little data, so my word2vec model does not perform well. My intention is to identify words similar to technical terms such as 'support vector machine', 'machine learning', 'artificial intelligence' etc.

I am interested in knowing if I can use the Google's wikipedia model for this. But according to my model most of the words I will be dealing with are n-grams. Hence, how can I utilise this Google's wikipedia model that is based on unigrams to achieve my task?

I am happy to provide more examples if needed :)


texts: face landmark detection cnn loss not converging in tensorflow

texts:
I am trying to build face landmark detection model using simple regression.I used celeba dataset which has 5 points hence 10 output units.I used grayscale and normalized image as input.
Here is my model
    self.model = models.Sequential()
    self.model.add(layers.Conv2D(32, (4, 4), activation='relu', input_shape=(218, 178,1)))#l1
    self.model.add(layers.Conv2D(64, (4, 4), activation='relu'))#l2
    self.model.add(layers.Conv2D(32, (4, 4), activation='relu'))#l3
    self.model.add(layers.Conv2D(16, (4, 4), activation='relu'))#l4
    self.model.add(layers.Conv2D(16, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(16, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(32, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    self.model.add(layers.AveragePooling2D((3, 3)))
    self.model.add(layers.Flatten())
    self.model.add(layers.Dense(16))
    self.model.add(layers.Dense(10))

Here is my loss function
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001),loss='mse')

Here loss is stuck around 18 and doesn't go below that. I tried various configuration of CNN architecture like adding and removing layers and I also tried with changing learning rate  but no use.
Please anyone point me in right direction. How can I debug this network.
(For this I only used first 100 images of dataset)


texts: How much memory will be required to train EfficientDet D7 for object detection?

texts:
I am planning on doing some object detection once I collect my image database and annotate the images. I'm going to use the TensorFlow 2 detection Model Zoo, as it offers a great selection of pre-trained models to fine-tune. I'm using TensorFlow 2.3.0 and Python 3.8.
The images that I will be collecting will be downsized to 1064 x 1064, to get a good size to retain resolution. I was looking at using the EfficientDet D7 1536x1536 model; the images will be padded to the max dimension to make the algorithm work. Using a batch size of 1 image for training, how much memory will be required to train this algorithm? I know it is a large algorithm that will take a lot.
I have access to the following machines:

Machine 1: Intel i7 - 5960x 3 GHz 8 core, GPU = Nvidia GeForce GTX TITAN X, RAM = 32 GB
Machine 2: Intel i7 - 5960x 3GHz 8 core, GPU = Nvidia GeForce GTX 1080, RAM = 64 GB

Will one of these machines be sufficient to work with this model? At this point, I plan to train without Google Colab and use the GPUs.


texts:
I am an absolute beginner in data science and I had this (possibly stupid) question on my mind, while reading a problem in Kaggle: Say I'm given IDs of some clients, IDs of products that they sell, and quantity of the product sold, and I'm asked to predict the quantity of a product, given the client ID, and the product ID.

Now, say the client IDs are in the range 10000 - 50000 and the product IDs are in the range 1-10.

Suppose, for a moment that the client IDs were random integers from 1-1000000 and the product IDs were random integers from 1000-2000. This isn't supposed to make the slightest change in the results, is it? After all, IDs are mere tags.

But thinking data-wise, I've just bloated up two entire columns in my data to a higher scale, and these are two features as well.

So how do I think about this? How do I factor in features which are IDs? How do I 'normalize' them?

I hope I'm not being vague here. I just don't know a better way of phrasing this question.


texts:
Few questions on categorical data. Need suggestions / pointers:

How can we check for correlation between categorical features and target or between the features themselves?
How about correlation between cat and num features?
How can we find cases where the target has a dependency on a combination of categorical features, but may not show corr with individual features?
Can we handle scenarios where, test data has a categorical feature value that is not present in training data?


texts: Unable to make CNN model with same output and input dimensions

texts:
I'm trying to develop a simple CNN model that takes in a RGB images and returns the same 3 channel as output.
Eg: 3 x 128 x 128 is what I'm giving as input
K.set_image_data_format('channels_first')
def resBlock(x, channels, kernel_size=[3, 3], scale=0.1):
    tmp = Conv2D(channels, kernel_size, kernel_initializer='he_uniform', padding='same')(x)
    tmp = Activation('relu')(tmp)
    tmp = Conv2D(channels, kernel_size, kernel_initializer='he_uniform', padding='same')(tmp)
    tmp = Lambda(lambda x: x * scale)(tmp)

    return Add()([x, tmp])

feature_size = 128

input_shape = (3, 128,128)
inputs = Input(shape=input_shape)

# x = Concatenate(axis=1)([inputs])
x = Conv2D(feature_size, kernel_size=(3,3), activation='relu', input_shape=input_shape, padding='same')(inputs)
for i in range(6):
    x = resBlock(x, feature_size)

x = Conv2D(feature_size, (3, 3), kernel_initializer='he_uniform', padding='same')(x)
x = Add()([x, inputs])
model = Model(inputs=inputs,  outputs = x)

This is what I have currently, however, it throws an error:
ValueError: Operands could not be broadcast together with shapes (128, 128, 128) (3, 128, 128)

It fails in this line: x = Add()([x, inputs])


texts: In random forest, can features with high Gini importance show insignificant p-values in t-tests?

texts:
In have two classes in my data, and one of my features with Gini importance 0.023 (8th highest out of 95 features) shows a p-value of 0.44 when doing a t-test on the values from the two classes. I'm wondering if this makes sense. Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.

EDIT: If it's helpful, all of the Gini importances range between 0.0011 to 0.0295, and the p-values ranged from 5.13e-109 to 1. These p-values were seen for both t-tests and rank-sum tests.


texts: How to input different sized images into transfer learning network

texts:
I have been looking online for a solution but have a difficult time finding a clear enough solution. I want to know how to use transfer learning (VGG16 for example) on images that have different sizes than the images the network originally trained on (so instead of inputting images of size (224,224,3) I want to input images of size (32,32,3)).

I initially thought about just padding those images but the network may look into the black pixels and think that they mean something, and I realize that might hard the accuracy and also when I tried to do that my colab notebook collapsed.

This is my VGG-16 code:
def vgg16_model(img_rows, img_cols, channel=1, num_classes=None):

model = VGG16(weights='imagenet', include_top=True)

model.layers.pop()

model.outputs = [model.layers[-1].output]

model.layers[-1].outbound_nodes = []

x=Dense(num_classes, activation='relu')(model.output)

model=Model(model.input,x)

#To set the first 8 layers to non-trainable (weights will not be updated)

for layer in model.layers[:15]:

   layer.trainable = False
for layer in model.layers[16:]:
  layer.trainable=True
model_new = Sequential()
for layer in model.layers[:-1]: # just exclude last layer from copying
    model_new.add(layer)
model=model_new
model.add(Dense(256,activation='relu',input_shape=(1000,)))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(10,activation='softmax'))
#model.add(Dense(1,activation='softmax'))

# Learning rate is changed to 0.001
#sgd = SGD(lr=1e-2, decay=1e-6, momentum=0.9, nesterov=True)
sgd = SGD(lr=lr,decay=decay,momentum=0.95, nesterov=True)
adam=Adam(lr=1e-5, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0001, amsgrad=True)
#model.compile(optimizer=adam, loss='binary_crossentropy',metrics=['accuracy'])
model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['accuracy'])

# checkpoint
filepath="weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

return model

I would greatly appreciate if someone could write the corrected version of this code in order to enable me to insert pictures of size (32,32,3).

Thanks a lot in advance!!


texts: Genetic Optimization, Heuristic regarding choosing the number of generations and population size

texts:
I have a simple model with some fitness function that I'm trying to max out.
This model have ~20 variables, each about ~15 options.
Is there a heuristic formula or a study of some sort that can guide me regarding the best:

Population size
Generations

There are other options of course such as Mutation rate / Crossover rate but I mainly interested in the 2 above.


texts:
I am building lost sales estimation model for out of stock days etc. using XGBoost. I am using simple logic of training model on data of normal days with ample inventory (when sales and demand are same) and then using trained model to predict demand on out of stock days. For model building I am splitting the normal days data into train and test datasets.

However I am getting a peculiar problem of very highly overestimated sales values of out of stock days. prediction for both train and test days are fine but only out of stock days prediction gives problem. Any tips what might be going wrong and how I can debug the problem in gradient tree type of model.


texts:
The following is from Understanding Machine Learning: Theory to Algorithm textbook:

Definition of PAC Learnability: A hypothesis class $\mathcal H$ is PAC learnable
if there exist a function $m_H : (0, 1)^2 \rightarrow \mathbb{N}$ and a learning algorithm with the
following property: For every $\epsilon, \delta \in (0, 1)$, for every distribution $D$ over $X$, and
for every labeling function $f : X \rightarrow \{0,1\}$, if the realizable assumption holds
with respect to $\mathcal H,D,f$ then when running the learning algorithm on $m \ge m_H(\epsilon,\delta)$
i.i.d. examples generated by $D$ and labeled by $f$, the algorithm returns
a hypothesis $h$ such that, with probability of at least $1 - \delta$ (over the choice of
the examples), $L_{(D,f)}(h) \le \epsilon$.

1) In the function definition $m_H : (0, 1)^2 \rightarrow \mathbb{N}$; what does a) 0 and 1 in the bracket, b) the integer 2, and c) $\rightarrow \mathbb{N}$ refer to?


texts: What are the benefits of having ML in js?

texts: What are the benefits of having ML in JavaScript I.e. the deeplearn.js (now tensorflow) stuff, as opposed to implementing the ML steps in a python backend?


texts: Why do we need to concatenate in a U-Net?

texts:
You might be familiar with the U-Net, a machine learning network deceived for image segmentation.
It's basically an encoder/decoder network with some direct links between encoder and decoder segments:

I would like to better understand the reason behind those gray lines!
What is the idea behind copying one part (cropped) of the outputs of the convolutional segments  (before max-pooling) and concatenating it to the inputs to the individual up-convolutional layer?


texts:
I've been searching for a long time to answer my question, but I haven't found anything. So I hope you can help.
I'm searching for an opportunity to add a regressor to my prophet model in python. I know the 'model.add_regressor' method but they don't solve my problem. Because I will add data I don't know in the future, rather only in the past, as a regressor.
I want to predict 6 time steps (monthly data, so 6 months into the future).
So, know anybody an opportunity to solve this?


texts:
I want to understand Timeseries shape similarity algorithm ( Shape-based distance aka SBD). I can't understand the statistics behind it and why it is better than DTW or other similarity measure. I'm trying to read this paper http://www1.cs.columbia.edu/~jopa/Papers/PaparrizosSIGMOD2015.pdf but can't understand anything in this regard. Any explanation with some example is really appreciated.

I started with learning about cross-correlation, normalized cross-correlation, Auto-Correlation, Normalization and Time Shift.

https://anomaly.io/understand-auto-cross-correlation-normalized-shift/

I couldn't understand the concept of sliding cross-correlation which is used to identify shape based similarity. What is "sliding" in this type of correlation and what difference it's making?


texts: Visualizing a large number of variables in python

texts: I have a large number of variables (2000) and want to visualize them using Python. Also, I would like to see the summary statistics of these variables. It seems that drawing box plots for all of these variables would not be feasible. What are some methods that could be used to visualize this data?


texts:
The Ward method for clustering minimizes the total within-cluster variance.
So I suppose that there is a link between the Ward distances that I got with the linkage function, and the variance of the data.
Is there a way to correlate it with the variance ?
I tried to see if the maximum ward distance is correlated with the variance, by trying to calculate those two datas on a lot of collections, but the results are not very interesting :


texts:
I was asked by my supervisor to replicate a result from a former graduate student. My supervisor believes the result of that paper is not accurate and he asked me to find out why! The paper was about conducting a random forest classifier to classify some sort of diseases. I was reading through the simulation process, and I realize that the training set and the test test were generated separately (the same set of parameters were used in the simulations). In other word, the training was simulated first, then another simulation was carried on to generate the test set. My understanding for training and test set is to simulate one data set then use the same data for training and test set.

My question, is it correct to simulate the training and test set separately? Does it affect the accuracy of the classifier? any reference will be truly helpful.


texts: Assign more importance to recent records during training

texts:
My goal is to build a classification model in order to predict if a customer will buy a product or not (binary classification).
Since in the last months (let's say 3-4) I know that the advertising of the company is changed a bit, I want to put more emphasis on the newer records.
I know that it is possible to specify the sample_weights parameter in most of the classification algorithms, but I don't know how to properly build these weights.
Basically, I have the date in which the customers have (or have not) bought the product, but I want to understand if there is a specific way to build the weights using this information.
In addition, I would also know if someone knows some references or applications for this application (time-related sampling weights).
Thank you in advance!


texts: Machine learning detect changes in components

texts:
I am a student who will finish my studies next year and I want to analyze the job market in advance. I have found an interesting job where it says:

The department of assemblies and systems of the X deals with the simulation, testing and evaluation of components, assemblies and complete systems mainly from the automotive sector. In the future, machine learning methods will be used to detect changes in components or test benches at an early stage

I find the last sentence very interesting, but I don't quite understand it. If you expect a change to a component (e.g. the component starts to break), you can hard code it to detect this change. Could somebody explain to me with a simple example how machine learning can be helpful in this area?


texts: Parameters to build a churn prediction engine

texts:
Let us assume that I’m working for some company A, and my manager has asked me to build a churn prediction engine given that the product to which the churn prediction is to be done is unknown. The engine must be a generalized one, that is it should predict the potential churns for any product data that is fed into it.

I have found a few basic parameters that are required for such a generalized engine. Recency, Frequency and Monetary value can be utilized for a generalized engine to determine the potential churns in any product. What are all the other generalized parameters like the above mentioned?

Say, you don’t know Machine Learning but how would you say if a person would churn or not with a parameter other than his subscription details, his activity or the complaints he made. What other parameters that would make sense here?


texts: Is it a Problem if Training Data and Evaluation Data are Very Similar?

texts:
I'm working on a group project where we have decided to use this Microsoft WIP repository as our starting point. It's a project which compares different frameworks and models and their ability to do NER in order to filter personal data.

We have several out of the box models we're evaluating from frameworks like SpaCy and Flair. We're also training additional models based on data generated with a data generator that is included in the project.

Recently I came to realize that the data generator actually just takes a file which includes templates, for example the line below:

The name in the account is not correct, please change it to [PERSON]

Then the [PERSON] part would be replaced with a name from another file including several names.

I came to realize that there are only 125 templates. These 125 templates are then used to generate 213 training samples, 59 test samples, and 28 validation sets. Thus of the 300 samples used for training our models over half are near duplicates, for example:

The name in the account is not correct, please change it to John Doe
The name in the account is not correct, please change it to Jeff Peters

We then have an additional step where we compare all the models on a different set of data. However this "different" set of data is just 300 more samples generated from the same 125 templates.

I believe this lack of diversity in data, and similarity between training and evaluation data makes all scores obtained so far invalid.

However my project partner does not believe this would have a negative effect. While I feel strongly that the data will create a problem I'm unable to provide strong citation or references to prove this to him.

He is saying we should "deconstruct the algorithm to show that sentence structure isn't as important to a machine as it is to humans".

Can anyone confirm that what I'm describing will be an issue and invalidate our results, and potentially provide references to back the claim?

Our aim is to compare models and their general performance for NER, not just their ability to perform on this specific data-set.


texts: Text classification of an imbalanced dataset

texts: I have a dataset with size ~ 500k entries. There are 2 columns, 'product description' and 'level 1'. I am developing my model such that it learns from a training set of 350k and based on the product description for test data, it gives the values in 'Level 1'. A simple linear classifier gives an accuracy of 85% which is too low, I am aiming for 97% atleast. I think this might be because the dataset is imbalanced, the level 1 values in the training data are imbalanced. How do I resolve this? Can I make the upsampling minority and downsampling majority work here?


texts: Does OpenAI Gym or Tensorforce require a normalized action space?

texts:
I am learning to use OpenAI Gym to make a custom environment with continuous action and observation spaces and apply reinforcement learning algorithms using the Tensorforce library. The problem is that the action space must be normalized (values in the [-1, 1] interval) in order to work; otherwise, when using the required (not normalized) values, it gives an error.

These are the lines I'm using:

self.action_space = spaces.Box(np.array([4.6, 80.0]), np.array([4.8, 180.0]), dtype=np.float32) # Gives error
self.action_space = spaces.Box(np.array([-1, -1]), np.array([1, 1]), dtype=np.float32) # OK, but not desired

The error I get when using the required (not normalized) values:

File "C:\Users\User\Anaconda3\lib\site-packages\tensorforce\environments\openai_gym.py", line 378, in unflatten_action
    if np.isinf(action).any() or np.isnan(action).any():

TypeError: ufunc 'isinf' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

where action is {'2'}.

Is there something I'm missing here? Or does OpenAI Gym or Tensorforce work only with normalized action values? If so, why? And how can this be avoided?


texts: Avoiding trivial solution with sparse output

texts:
I have a network whose output is a large vector ( ~ 2000 let's say ). The ground truth output is very sparse and binary- that is out of the 2000 most of the values will be 0 and just a handful will be 1. Just to clarify, each sample will have 1's at different locations with some uneven likelihood of there being a 1 at any given neuron.

I am pretty sure that if I run this network it will quickly find the trivial solutions of outputting all 0 and just stay there.

How might you suggest to nudge the network to avoid this trivial solution?

I thought of somehow weighting the samples in such a way that each sample will give a high weight to the neurons that are equal to 1. This seems very hard to implement efficiently. How can I easily implement this in tensorflow/keras efficiently?

Another Idea is to set all the 0's to random centered at .5 and slowly decrease them towards zero as the network trains.

Other ideas?


texts:
I trained a Random Forest classifier (sklearn) and consequently computed the feature importance and consequently ranked them. The forest has 100 estimators. My top 5 features with their importances are as here:

f1 = 0.91
f2 = 0.04
f3 = 0.013
f4 = 0.007
f5 = 0.004

To avoid over-fitting, I did the evaluation using cross-validation and learning curve.

My question is that the importance for f1 seems significantly higher than other features. Does it imply incorrectness (over-fitting?) of any sort? Should I do feature selection in some other way to generalize the model better?


texts: Smart sentence segmentation not splitting on abbreviations

texts:
Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the resulting sentence split is not correct. I understand that sentencers are supposed to be simple and quick but I am wondering if there is a better one that takes into account something more than uppercased words and punctuation? Alternatively, how to make SpaCy / NLTK / ... sentencer work for such sentences?
I am interested primarily with sentencers with Python API.


texts:
There are several methods to normalize data, among them are:

min-max, z-score and scale decimal.

Can I use any one or with what criteria should I choose one of them?


texts: Multiple labels in supervised learning algorithm

texts:
I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ["celebrity", "murder"]. So basically each vector of features can have many labels (not the same amount. The first feature vector can have 3 labels, second 1, third 5).

If I would have just one label corresponded to each text, I would try a Naive Bayes classifier, but I do not really know how should I proceed if I can have many labels.

Is there any way to transform Naive Bayes into multi label classification problem (if there is a better approach - please let me know)?

P.S. few things about the data I have.

approximately 10.000 elements in the dataset
text is approximately 2-3 sentences
maximum 7 labels per text


texts: Reducing 3D tensor to 2D tensor for classification

texts:
Context
I am building a vanilla RNN that can take a sequence of N hot-encoded characters and output a single class. Here is the input X and Y into the graph:
batchX_placeholder = tf.placeholder(tf.float32, [batch_size, N, encoding_size])
batchY_placeholder = tf.placeholder(tf.float32, [batch_size, num_classes])

I also input a state that is initialized to zeros, and is iteratively updated for each character:
init_state = tf.placeholder(tf.float32, [batch_size, encoding_size, state_size])

At the end of the sequence, I have done N matrix multiplies to init_state, and it remains the same shape.
Question
Now I want to calculate loss - but how am I to do so when I'm comparing tf.placeholder(tf.float32, [batch_size, num_classes]) (the "true" class labels) and tf.placeholder(tf.float32, [batch_size, encoding_size, state_size]) (the final state)?
(I have tried reducing the final state by doing one matrix multiply to get a 1 in the last dimension, reshaping, and then doing another multiply to get the shapes to match, but it seems weird and returns weird results that don't seem to converge)


texts: How can one can aim for a career in machine learning?

texts:
I'm interested in various machine learning methods and wonder how can I aim for a career in the field.
What are most basic skills to have?

I mean the matters strictly connected to the machine learning - not programming or libraries.


texts: Table extraction state of the art (beginning of 2018)

texts:
What is the current state of the art when it comes to extracting tabular data from documents (images/ocr, not pdf/html)?

I have come to ICDAR, but the last table detection challenges were from 2015.


texts: What is Sentiment Bias? How will it affect a Lexicon Based Sentiment Analysis?

texts:
I am comparing deep learning and lexicon/rule-based models for sentiment analysis. When I was doing some research into the limitations of lexicon based models, I came across a journal article that mentioned sentiment bias. However, this article did not explain what exactly sentiment bias is.

I am providing the link here: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0202523


texts: I am looking for a suitable graph representation where the nodes/vertices are molecules with 2 main variables: structural characteristic and a real number property (for example biological potency). The network is build up by molecules by matching structural properties and it can be rather complex with many clusters of high degree vertices (ie molecules that are structurally similar to each other). However the representation should give a clear picture of potencies, although high detail accuracy is not required: some sort of classification is adequate. I played with JUNG with a graph having edge weights as the change in potency and the ISOM layout thinking that it reserves (as much as possible) the edge weights but with not good results. Something like radial layout could be ideal, as it gives a clear picture of potency distribution, but this obviously would suffer from crowding of vertices that are closer to the center.  As, obviously, my knowledge is restricted, is there some layout that you would suggest and is practical for a non expert to implement?


texts:
I need to rank human skills depending on their similarity to the input skill. So if I enter "Dutch language", I want the list like this:

0.97 Dutch
0.86 Dutch lessons
0.55 Frisian
0.50 Flemish
0.27 German language

I have a database of around 4500 human skills (ranging from "programming in C" to "baking almond cake") with 600 manually categorized. I can already find corresponding article on BabelNet and pull domain, categories and related terms.

Example skill with the data from BabelNet:

name:"photography"
categories:
  0:"Photography"
  1:"French_inventions"
  2:"Optics"
  3:"1822_introductions"
manualCategory:"art & music"
domains:
   ART_ARCHITECTURE_AND_ARCHAEOLOGY:1
compounds:
  0:"digital_photography"
  1:"landscape_photography"
  2:"photographic_developing"
  3:"motion_photography"
  4:"nature_photography"
  ...
  48:"photographic_plates"
otherForms:
  0:"still_photography"
  1:"photo"
  2:"photos"
  3:"photographed"
  4:"photographers"
  ...
  20:"Photographer"

Can you suggest me the approach or at least steer in the right direction?


texts: Can a Logsitic Regression model continue making predictions after removing predictions from the data set?

texts:
I have a logistic regression model that predicts churn (0 vs. 1). I was asked to use the model to predict on a historical group of non-churners, remove anyone who was marked as a churner, and then increase one variable while keeping the rest constant to see how the prediction changes. Interestingly, it predicts zero churns after removing this first cohort of predicted churn. Increasing this one variable seems to have no impact after the initial cohort is removed, although this variable has the greatest feature importance in the model. Is this simply functioning as expected?

Here is the sample code being used:

r1_pred = logisticRegr_balanced.predict(dfa)
dfa['Churn Prediction'] = r1_pred
print("Non-Churners: "+str(len(dfa[dfa['Churn Prediction']==0])))
print("Churners: "+str(len(dfa[dfa['Churn Prediction']==1])))
print("Percent Churn: " +str((len(dfa[dfa['Churn Prediction']==1]))/len(dfa['Churn Prediction'])))

Results in:

Non-Churners: 70611
Churners: 19609
Percent Churn: 0.21734648636665926

Then I create a new dataframe with only the survivors, and increment the "Customer Life" variable by 30 days.

dfa_30 = dfa[dfa['Churn Prediction']==0]
dfa_30 = dfa_60.drop('Churn Prediction', axis=1, inplace=False)
dfa_30.CustomerLife = dfa_30.CustomerLife + 30

r30_pred = logisticRegr_balanced.predict(dfa_30)
dfa_30['Churn Prediction'] = r30_pred
print("Non-Churners: "+str(len(dfa_30[dfa_30['Churn Prediction']==0])))
print("Churners: "+str(len(dfa_30[dfa_30['Churn Prediction']==1])))
print("Percent Churn: " +str((len(dfa_30[dfa_30['Churn Prediction']==1]))/len(dfa_30['Churn Prediction'])))

Results in:

Non-Churners: 70611
Churners: 0
Percent Churn: 0.0

Is the model no longer able to predict churn because it has categorized everything on the binary scale in the first prediction, so all that's left are "permanent" survivors?


texts:
I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture analysis for feature extraction to build a classification module.
My question is, how to decide whether these images needs pre-processing or not, such as noise removal by using median filter for example? In other words, how to determine if there is noise in this data?


texts: Discarding rare words when comparing texts - per text, per comparison, or per codex?

texts:
I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency.

first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words used are "rare" (for example, they appear  <10 times in the text). I wish to replace these rare words with tokens ("UNK", for unknown word).

Let's say I am comparing 10 texts. At what point should I count the rare words to discover which can be considered rare?
I can think of three options:

Count words for each text separately. If "Boat" is a word is rare in the first text and frequent in second one, it is replaced with UNK only in the first text, and not in the second one.
Count words for each comparison. For 10 texts I'm carrying out 45 (two-sided) comparisons. If I choose this option, I will not replace the word "Boat" in either text when comparing the first and second texts. I will however replace it with UNK when comparing the first and third texts (assume that "Boat" is rare in the third text as well).
Count words for the entire collection. Using this option, "Boat" is never replaced in any of the texts, due to it being a non-rare word in one of the texts.

What option would serve me better in finding meaningful N-gram representations of texts?

Many thanks.


texts: How to improve the binary classification model for text (News Articles) of Recurrent Neural Net with word emmbeding?

texts:
I am trying to do binary classification of news articles using Recurrent Neural Net with word embedding. Following are the parameters of the model:

Data:
    8000 labelled news articles (Sports:Non-sports::15:85)

Parameters:
    embedding size = 128
    vocabulary size = 100000
    No. of LSTM cell in each layer = 128
    No. of hidden layers = 2
    batch size = 16
    epochs = 10000

Result:
    AUC on training set = 0.60
    AUC on testing set = 0.55

As the both training and testing error is high model is underfitting and require more data. So I have couple of doubts here:

What would be the optimum data size required?
Can we change the parameters to improve AUC. By decreasing, embedding size or No. of neurons we can minimize degree of freedom.


texts: LSTM model prediction is almost constant

texts:
I am new to RNN and LSTM and currently experimenting with different settings.
When trying to model time series data in absolute terms (predicted close price), I am faced with the following problems:

Validation loss falling while training loss raising
Prediction on test set is almost constant

Any ideas why is it the case?
#df is the sample with features and target values (close+1)

# MAKE LARGE SAMPLE FOR TESTING
df_train = df[:40000]
df_val = df[40000:50000]
df_test = df[50000:]
df_train.reset_index(drop=True,inplace=True)
df_val.reset_index(drop=True,inplace=True)
df_test.reset_index(drop=True,inplace=True)

# Make x_train, x_val sets by dropping target variable
x_train = df_train.drop(['close+1'], axis=1)
x_val = df_val.drop(['close+1'], axis=1)

# Scale the training data first then fit the transform to the test set
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_test = scaler.transform(x_val)

# Create y_train, y_test, simply target variable for regression
y_train = df_train['close+1']
y_test = df_val['close+1']

# Define Lookback window for LSTM input
sliding_window = 30

# Convert x_train, x_test, y_train, y_test into 3d array (samples, timesteps, features) for LSTM input
dataXtrain = []
for i in range(len(x_train)-sliding_window-1):
        a = x_train[i:(i+sliding_window), 0:(x_train.shape[1])]
        dataXtrain.append(a)

dataXtest = []
for i in range(len(x_test)-sliding_window-1):
        a = x_test[i:(i+sliding_window), 0:(x_test.shape[1])]
        dataXtest.append(a)

dataYtrain = []
for i in range(len(y_train)-sliding_window-1):
        dataYtrain.append(y_train[i + sliding_window])

dataYtest = []
for i in range(len(y_test)-sliding_window-1):
        dataYtest.append(y_test[i + sliding_window])

# Make data the divisible by a variety of batch_sizes for training
dataXtrain = np.array(dataXtrain[:39680])
dataYtrain = np.array(dataYtrain[:39680])
dataXtest = np.array(dataXtest[:9728])
dataYtest = np.array(dataYtest[:9728])

# Checking input shapes
print('dataXtrain size is: {}'.format((dataXtrain).shape))
print('dataXtest size is: {}'.format((dataXtest).shape))
print('dataYtrain size is: {}'.format((dataYtrain).shape))
print('dataYtest size is: {}'.format((dataYtest).shape))

### ACTUAL LSTM MODEL

batch_size = 256
timesteps = dataXtrain.shape[1]
features = dataXtrain.shape[2]

# Model set-up, stacked 4 layer stateful LSTM
model = Sequential()
model.add(LSTM(512, return_sequences=True, stateful=True,
               batch_input_shape=(batch_size, timesteps, features)))
model.add(LSTM(256,stateful=True, return_sequences=True))
model.add(LSTM(256,stateful=True, return_sequences=True))
model.add(LSTM(128,stateful=True))
model.add(Dense(1, activation='linear'))

model.summary()

reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.9, patience=5, min_lr=0.000001, verbose=1)

def coeff_determination(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

model.compile(loss='mse',
              optimizer='nadam',
              metrics=[coeff_determination,'mse','mae','mape'])

history = model.fit(dataXtrain, dataYtrain,validation_data=(dataXtest, dataYtest),
          epochs=100,batch_size=batch_size, shuffle=False, verbose=1, callbacks=[reduce_lr])

predictions = model.predict(dataXtest, batch_size=batch_size)
print(predictions)

plt.plot(history.history["loss"][5:])
plt.plot(history.history["val_loss"][5:])
plt.title("model loss")
plt.ylabel("loss")
plt.xlabel("epoch")
plt.legend(["train", "val"], loc="best")
plt.show()

plt.figure(figsize=(20,8))
plt.plot(dataYtest)
plt.plot(predictions)
plt.title("Prediction")
plt.ylabel("Price")
plt.xlabel("Time")
plt.legend(["Truth", "Prediction"], loc="best")
plt.show()


texts: What is a lower bound on the vocabulary size for generating word/sentence embedding using word2vec or skip thought vectors?

texts:
I am working on a NLP related task. I have about 150 documents, each few pages long (5/6 pages long on average). After removing stopwords and other unnecessary symbols and digits, I have about 104,000 unique words. The task at hand probably require some kind of word embedding (such as word2vec) as simple bag-of-words type approach aren't working properly. However, I am concerned about the size of the data I have. I have looked at pre-trained word embedding (GloVec), however, due to the narrow focus of the domain (manufacturing) of our texts, I am hesitating to use these pre-trained vectors. That leaves me with training our own. However, the size of our data set concerns me. Hence I am just throwing this question out there: What should be the lower bound on the size of the vocabulary that we need in order to train a word embedding model (word2vec) that will be reasonable.

Any response would be greatly appreciated.

Thanks


texts: should I shift a dataset to use it for Time series regression with RNN/LSTM?

texts:
I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1

so my simple question is, should we always do this when working with time series data?

I have a time series data and I want to use it to build a regression model with RNN. I used feed forward NN first but it didn't work good enough so I decided to use RNN/LSTM. My question is should I let the dataset as it is now where every row have features and target at time t or should I shift the target column so that the features would always be at time t but the target is always shifted(t+1) ahead?


texts: Import data from google drive to Kaggle Kernel

texts: I want to import a csv file from google drive . I tried using the link in add dataset tab but it is taking some thing else as "Open". Please see the image.


texts: Large categorical dataset for regression

texts:
I need to collect several large datasets (thousands of samples, dozens of features) for regression with only categorical inputs. I already look for such datasets in the UCI repository, but I did not find any suitable one.

Does anybody know of any such dataset, or of any additional dataset repository on the Internet?


texts: Can CNNs detect features of different images?

texts:
In lecture, we talked about “parameter sharing” as a benefit of using convolutional networks. Which of the following statements about parameter sharing in ConvNets are true? (Check all that apply.)

It allows parameters learned for one task to be shared even for a different task (transfer learning).
It reduces the total number of parameters, thus reducing overfitting.
It allows gradient descent to set many of the parameters to zero, thus making the connections sparse.
It allows a feature detector to be used in multiple locations throughout the whole input image/input volume.

Here are the correct answers:

It reduces the total number of parameters, thus reducing overfitting.
It allows a feature detector to be used in multiple locations throughout the whole input image/input volume.

Why isn't the following answer also correct:

It allows parameters learned for one task to be shared even for a different task (transfer learning).
Doesn't ConvNets allow parameters to be shared, detecting the similar features of different images?


texts: Misclassifying only 1 of 3 classes in NN with Keras against RF

texts:
I'm working on the classification of a small dataset of clinical data. It is composed of around 3500 samples, with 15 features and there are 3 diferent classes (2 of them also have 2 subclasses each one). I was trying to make a 'first level' classification due to the lack of data and with random forest I'm achieving this:

I getting almost same results using SVM, so I wanted to give it a try to a really simple NN. I created a 1 hidden dense layer, 400hidden units, bathsize 16, only 200 epochs... and I'm getting better results in two classes, but one of them is really worse:

So maybe someone has encountered the same problem and can bring some light. My main question is: could this be improved? can you improve 1 class performance without adding more data? Or maybe data is not good enough for NN or its because the lack of data? I was really hoping to improve Random forest results or, at least, match them!
I'm training with unbalaced data but only like 30% 40% 30% (I have more data of the misclasssified one) but I'm using 'class_weights'.
Thanks in advance!


texts: The difference between data science and algorithm development

texts:
I see a lot of job opportunities in the field of data science but I'm not sure the difference between a data scientist and deep learning algorithm developer.
Can someone explain that to me?


texts: Why is my test data accuracy higher than my training data?

texts:
I'm using four years of data, training on the first 3 and testing on the fourth. Using LSTM w/ Keras. My test data set (which has no overlap at all with the training) is consistently performing better than my training data. How should I interpret this? It seems very unusual. Here's the trail end of the model output. You can see my training accuracy for a given epoch hovers around 80%, but test output jumps to about 86%:

Epoch 8/10
9092/9092 [==============================] - 9s 964us/step - loss: 0.9870 - acc: 0.8185
Epoch 9/10
9092/9092 [==============================] - 9s 1ms/step - loss: 0.9670 - acc: 0.7996
Epoch 10/10
9092/9092 [==============================] - 9s 937us/step - loss: 0.9799 - acc: 0.7895
Test Set Accuracy: 85.96%

predicted     0    1
actual
0          2639  238
1           211  111

Edit: Here's my code to create & compile the model:

embedding_vector_length = 32
days = 30

model = Sequential()
model.add(Embedding(2080, embedding_vector_length, input_length=days)
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
model.fit(train_x, train_y, epochs=3, batch_size=64,class_weight={0:1.,1:1})
scores = model.evaluate(test_x, test_y, verbose=0)
print("Test Set Accuracy: %.2f%%" % (scores[1]*100))


texts: Normalizing and joining of independent logistic regression model's prediction

texts:
I need to train several Logistic regression models on a different set of data (with a different set of labels):
Train data of model 1:
[X1, X2, X3] -> y1
[X4, X5, X6] -> y2
...
Train data of model N:
[X99, X101, X102] -> y8
[X103, X104] -> y9

As a model, I choose Logistic regression with a One-vs-Rest train scheme. When I get an input from the user, I need to determine which label was most probable (from all N models). So what I am trying to do is query trained models with validation data and calculate mean MU and standard deviation STD. Based on this value I will normalize predictions to be comparable between each other - is the assumption correct?
# normalization
normalized_predictions = [(pr - g["pos_mu"]) / g["pos_std"] for g, pr in zip(self.gaussians.values(), prediction)]

# example output of each Logistic regression model
output.append({
                    'answer': self.idx2label[idx],
                    'confidence': normalized_predictions[idx],
                    'hit': ""
})

...
# query and normalize - normalization is shown above
responses = query_log_reg_model_and_normalize()

# sort based on confidence value
responses = sorted(responses, key=lambda r: r['confidence'], reverse=True)

# return most probable
return responses
```


texts: AlexNet Research Paper VS PytTorch and Tensorflow implementation

texts:
I'm making my way through Deep Learning research papers, starting with AlexNet, and I found differences in the implementation of PyTorch and Tensorflow that I can't explain.
In the research paper, they define the model architecture with:

The first convolutional layer filters the 224x224x3 input image with
96 kernels of size 11x11x3 with a stride of 4 pixels (this is the distance between the >  receptive field centers of neighboring neurons in a kernel map).

The second convolutional layer takes as input the (response-normalized and pooled) output
of the first convolutional layer and filters it with 256 kernels of size 5x5x.

The third, fourth, and fifth convolutional layers are connected to one another without any
intervening pooling or normalization layers.

The third convolutional layer has 384 kernels of size 3x3x256 connected to the
(normalized, pooled) outputs of the second convolutional layer.

The fourth convolutional layer has 384 kernels of size 3x3x192

The fifth convolutional layer has 256 kernels of size 3x3x192.

Note: Fully Connected layers are not inlcuded
Here is the PyTorch implementation of the model:
# 1st Layer
nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),

# 2nd Layer
nn.Conv2d(64, 192, kernel_size=5, padding=2),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2),

# 3rd Layer
nn.Conv2d(192, 384, kernel_size=3, padding=1),
nn.ReLU(inplace=True),

# 4th Layer
nn.Conv2d(384, 256, kernel_size=3, padding=1),
nn.ReLU(inplace=True),

# 5th Layer
nn.Conv2d(256, 256, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=3, stride=2)

And the Tensorflow implementation:
# 1st layer
net = slim.conv2d(inputs, 64, [11, 11], 4, padding='VALID', scope='conv1')
net = slim.max_pool2d(net, [3, 3], 2, scope='pool1')

# 2nd layer
net = slim.conv2d(net, 192, [5, 5], scope='conv2')
net = slim.max_pool2d(net, [3, 3], 2, scope='pool2')

# 3rd layer
net = slim.conv2d(net, 384, [3, 3], scope='conv3')

# 4th layer
net = slim.conv2d(net, 384, [3, 3], scope='conv4')

# 5th layer
net = slim.conv2d(net, 256, [3, 3], scope='conv5')
net = slim.max_pool2d(net, [3, 3], 2, scope='pool5')

Note: Tensorflow applies ReLU activation to every layer
Why both implementations have a first layer with a kernel of 64 where the paper mentions a kernel of 92?
Why both implementations have a second layer with a kernel of 192 where the paper mentions a kernel of 256?
[PyTorch] Why is there a padding of 2 on layers 1 and 2?
[PyTorch] Why is there a kernel of 256 for the 4th layer where it should be a kernel of size 384


texts: How to predict constant failing of equipment

texts:
I am trying to predict the failing of an equipment that heat up the liquid in a pipeline using a heat exchanger. The heat exchanger gets build up inside the pipe and thus needs to be flushed every once in a while. There are sensors around the device that are collecting data like temperature, flowrate, pressure, etc every hour. The flushing events happen once or twice a year. The recorded dates for the events are not very specific, only to the month. There is no particular measure that the operators monitor to get an idea of the health of the heat exchanger. The flushing was done neither due to the actual status or scheduled. But the efficiency should be highest after every flushing. I thought about anomaly detection, but that applies to equipment running normal most of the time with anomaly occurs seldom. This failing is constant process and gradual. If there is a pattern the data it should be a gradualy decreasing curve.

Another idea I had was to predict the remaining useful life method. Basically rank the time periods leading to flushing event by how close in time, with closest being the most severe. So it is basically a classification problem. But the thing is the flushing date is not exact which only has the month and year. There is lot of missing values for some of the key measures which might be indicators for on and off time. Plus this is not the same problem as failure prediction because it is not failure. Even if the flushing event doesn't happen, the equipment still works. And before every time the equipment is flushed, the condition of the equipment may vary.

What is the best way to kind of quantify the deteriorate rate of the equipment?


texts: What are differentiable modules used in deep learning

texts:
I am reading this paper.

  Convolutional Neural Networks define an exceptionally powerful class of models,
  but are still limited by the lack of ability to be spatially invariant to the input data
  in a computationally and parameter efficient manner. In this work we introduce a
  new learnable module, the Spatial Transformer, which explicitly allows the spa-
  tial manipulation of data within the network. This differentiable module can be
  inserted into existing convolutional architectures, giving neural networks the abil-
  ity to actively spatially transform feature maps, conditional on the feature map
  itself, without any extra training supervision or modification to the optimisation
  process. We show that the use of spatial transformers results in models which
  learn invariance to translation, scale, rotation and more generic warping, result-
  ing in state-of-the-art performance on several benchmarks, and for a number of
  classes of transformations.

Spatial Transformers are used in CNNs to have spatial invariant transformations and consequently the process of learning would be so easier and the networks would have better performance for data with different kind of distributions (noisy data). In this paper I don't realize the meaning of differentiable module. These so called differentiable modules are used in neural networks. But what is the meaning of differentiable?


texts: What kind of statistical analyses can I do with my data?

texts:
I'm trying to analyze human intentions in clicking google ad word keywords.

In this dataset I have the usual adword details, for example

CTR = Clicks / Impressions
CPC = Cost / Clicks
CPA = Cost / Converted Clicks
ROI = Total Conversion Value / Cost
CVR = Converted Clicks / Clicks
Weighted Avg Pos = WAP / Impressions
and Ad position placement.

Basically, I can imagine a relationship between Keywords and CTR. I can do some psychological analysis, like opinion mining and emotion detection, like a vice. Still, I need to show different types of analysis.

So what kind of advanced statistical analysis methods can be applied to this data set? My data looks like this:

  Keyword |Average Position|    Average CPC|    Clicks| CTR|    Cost|   Impressions

  android app developers    ,1  ,0.5,   21.79,  8.7%,   10.99,  250


texts: Neural network regression problem, integer output neuron constraint

texts:
The problem I'm solving is a regression problem using neural networks, and the "y" value covers a very large range (let's say y represents the number of people, ranging from 0 people to 10000 people), which makes it impossible to be interpreted as a classification problem.

The only difference between the problem I'm solving right now and the ordinary regression problem is that the output neuron should produce a strict integer, not decimal.

Is there any activation function that can handle this constraint?  Or what's the appropriate way to approach to model this constraint?


texts: Why MADDPG rather than taking all cooperating agents as a single meta-agent?

texts: Since MADDPG uses a centralized critic for training, why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space? In my opinion, MADDPG is centralized enough, so it won't hurt to go one step further.


texts:
This question is from this post asking about what is the difference between one-hot encoding and leave-one-out encoding.

I now understand how to calculate the leave-one-out encoding for the training set. But how about the test set, where is Y (label) is blinded? If the encoding for test set is always 0.5, then what is the point of this encoding?


texts: Is there enough datapoints to make a reasonable predcitction?

texts:
I am planning to use this dataset (for a school project) in order to determine the most important features in predicting whether a student will receive a placement.
Further I would like to create a model which would predict if a student will receive a placement based on their own features.
Link: https://www.kaggle.com/benroshan/factors-affecting-campus-placement
However, I have noticed that this dataset only has 215 rows. So my question is:

Is this enough data points for the tasks I plan to conduct?
If so, what classification algo would be most appropriate for a dataset which is this small?

Thank you for any help in advance!


texts:
i'm currently experimenting with scikit and the DBSCAN algorithm. And i'm wondering how to combine the data with the labels to write them into a new file. I'd also like to understand how the labels array is used to filter the examples.

Please correct me anytime i say something wrong because i'd like to understand the whole process better.

For example my data looks like this:

city    x    y
A       1    1
B       1    1
C       5    5
D       8    8

So if i understand it correctly i first need to split the data i'd like to cluster. If i use the data like above it will consider the city column aswell (or even fail).

So my next array would look like this:

x    y
1    1
1    1
5    5
8    8

Now i'll use the DBSCAN on my Array and it will create a cluster model. The clusterlabels are now stored in the array foo.labels_.

As far as i know i can filter the data with those labels to get the items within the clusters. Let's assume my data is in the dataframe cities:

cluster0 = cities[foo.labels_ == 0]

What i don't understand is how this works. I somehow don't get what exactly happens here. I know that the labels are in an array where the column is a number and the value of the column is the cluster. So how do i get the correct index of my cities?

So after the clustering i'd like to export my data back into a CSV file with the following format:

city    x     y     cluster
A       1     1     0
B       1     1     0
C       5     5     1
D       8     8     2

My guess is to use to original dataframe and add another column like this:

cities = cities.assign(cluster=p.Series(labels_))

But i'm absolutely unsure if that's the correct way to achieve what i want to do.

I'd really appreciate some opinions and explanations.


texts: What kind of plot will be helpful for total count group data ( data grouped by month and year )?

texts:
I have a group data frame like this :

group_by_region_year_month=train_data.groupby(['Region Name (GO)','Year','Month'],as_index=False)

    region_sum=group_by_region_year_month.agg({"AMV": "sum"})

Here , AMV is my target variable in regression ( this is count of total vehicles).

I have data for each day from 2000-3-17 to 2017-10-11.

I want to show the variation of the count for each year and each month for each Region.

How best can I plot it?

I tried using seaborn.FacetGrid in this way:

# g=sns.FacetGrid(row="Region Name (GO)", col="AMV", data=region_sum)

But it is taking too much time .

Is there any better alternative(s) ?

Edit 1

I tried this :

sns.set(rc={'figure.figsize':(50,30)})

ax=sns.barplot(x='Region Name (GO)', y='AMV', hue='Year', data=train_data)
ax.set_xticklabels(ax.get_xticklabels(), rotation=90)

which gave me a very clumsy plot :

I did not use countplot as countplot would give me count of rows. But , I want sum of all values in a group.

Can I represent it in any better way ?


texts: How to feed key-value features (aggregated data) to LSTM?

texts:
I have the following time-series aggregated input for an LSTM-based model:

x(0): {y(0,0): {a(0,0), b(0,0)}, y(0,1): {a(0,1), b(0,1)}, ..., y(0,n): {a(0,n), b(0,n)}}
x(1): {y(1,0): {a(1,0), b(1,0)}, y(1,1): {a(1,1), b(1,1)}, ..., y(1,n): {a(1,n), b(1,n)}}
...
x(m): {y(m,0): {a(m,0), b(m,0)}, y(m,1): {a(m,1), b(m,1)}, ..., y(m,n): {a(m,n), b(m,n)}}

where x(m) is a timestep, a(m,n) and b(m,n) are features aggregated by the non-temporal sequential key y(m,n) which might be 0...1,000.

Example:

0: {90: {4, 4.2}, 91: {6, 0.2}, 92: {1, 0.4}, 93: {12, 11.2}}
1: {103: {1, 0.2}}
2: {100: {3, 0.1}, 101: {0.4, 4}}

Where 90-93, 103, and 100-101 are aggregation keys.

How can I feed this kind of input to LSTM?

Another approach would be to use non-aggregated data. In that case, I'd get the proper input for LSTM. Example:

Aggregated input:

0: {100: {3, 0.1}, 101: {0.4, 4}}

Original input:

0: 100, 1, 0.05
1: 101, 0.2, 2
2: 100, 1, 0
3: 100, 1, 0.05
4: 101, 0.2, 2

But in that case, the aggregation would be lost, and the whole purpose of aggregation is to minimize the number of steps so that I get 500 timesteps instead of e.g. 40,000, which is impossible to feed to LSTM. If you have any ideas I'd appreciate it.


texts: Latent Semantic Indexing False Positive Detection

texts:
I am using the Gensim LsiModel. I have a set of documents and fixed set of topics. Some of the documents are already categorized, others are not. The goal is to categorize the uncategorized documents with the most relevant category. I am using a similarity search as described here.

http://radimrehurek.com/gensim/tut3.html

So, I am comparing each uncategorized document to the categorized corpus to find the most relevant category. I am seeing very good performance on documents which have an appropriate category. However, it is to be expected that some documents will not have an relevant category, e.g. it is in Spanish, it is spam, or it just does not fit into an existing category. With this model every document is categorized and the best fit is the category with the highest similarity score. My question is, how can I determine when there is not relevant category? My assumption is that the similarity measures for the documents should all be low, but this is not always true. This also seems to be an arbitrary measure. Are there better ways to say a particular document does not fit well into existing categories?


texts: How marginal contributions of adding a variable in a model is calculated in determining SHAP feature importance?

texts:
I was trying to find feature importance using SHAP values in python for Isolation Forest. SHAP calculates the feature importance of a feature($i$) pertaining to a model($f$) for a datapoint($x$) using the following formula:

where N is the set of all features. Now, the terms within the summation contain $[f_x(S\bigcup\{i\}) - f_x(S)]$ where $S$ is a set of features that does not contain the feature $i$ (i.e $S \subseteq N$ \ $\{i\}$).
So, for example, if we have features $x_1, x_2, x_3, x_4$ in our model and we want to find the importance of feature 4 ($i=4$), then the term  $[f_x(S\bigcup\{i\}) - f_x(S)]$ could be $[f_x(x_1, x_2, x_3, x_4) - f_x(x_1, x_2, x_3)]$ or $[f_x(x_1, x_4) - f_x(x_1)]$ etc.
So, we fit or model $f_x$ (linear regression or decision tree or whatever it is) on all the features and get the value $f_x(x_1, x_2, x_3, x_4)$ as the prediction of the fitted model at x. But how do we calculate the other terms, for example $f_x(x_1, x_2, x_3)$. It is written in many articles that $f_x(x_1, x_2, x_3)$ could be calculated by fitting the same model "as if the feature $x_4$ were not present". But that would be computationally too heavy. How does python's shap package calculate those terms?


texts: Why Pair plot is taking up the target variable into consideration?

texts:
I am working on Haberman's cancer survival data set.

I tried to visualize the pair plot with features as AGE, Op_Year, axil_nodes_det and target variable as Surv_status. I expect 12 plots but getting 16 including plots with Surv_stautus which is my target variable
here is my code:

sns.set_style("whitegrid");
sns.pairplot(patients, hue="Surv_status", size=3);


texts:
Do you use automatic cleaning tools for data?

I mean something similar to h2o.ai's auto ml function but applied to preprocessing data.
Or do you always clean data 'by hand'.


texts: making a picture of a neural network in tensorflow

texts:
I'm new to Machine Learning and I have just gone through a tutorial explaining how to create a neural network in TensorFlow. I was wondering if it is possible to visualize the neural network I created. The output should be a picture like this
Thanks.
MRE:
import tensorflow as tf

ann = tf.keras.models.Sequential()
ann.add(tf.keras.layers.Dense(units=6, activation='relu'))
ann.add(tf.keras.layers.Dense(units=6, activation='relu'))
ann.add(tf.keras.layers.Dense(units=1, activation='sigmoid'))
tf.keras.utils.plot_model(ann, to_file='model.png', show_shapes=True,
                          show_layer_names=True, rankdir='TB',
                          expand_nested=False, dpi=96
                          )


texts: How to calculate growth function for a threshold function

texts:
I'm working on a homework problem but don't fully understand it. The problem and solution:

I don't understand the definition of the threshold function.

Does it mean to pick one feature and classify the point based on that one feature?

It's the only way I can think of to explain the solution, $N$ ways to pick a feature, for each feature there are $m+1$ ways to select the threshold.


texts: Why convolute if Max Pooling is just going to downsample the image anyway?

texts:
The idea of applying filters to do something like identify edges, is a pretty cool idea.

For example, you can take an image of a 7. With some filters, you can end up with transformed images that emphasize different characteristics of the original image. The original 7:

can be experienced by the network as:

Notice how each image has extracted a different edge of the original 7.

This is all great, but then, say the next layer in your network is a Max Pooling layer.

My question is, generally, doesn't this seem a little bit like overkill? We just were very careful and deliberate with identifying edges using filters -- now, we no longer care about any of that, since we've blasted the hell out of the pixel values! Please correct me if I'm wrong, but we went from 25 X 25 to 2 X 2! Why not just go straight to Max Pooling then, won't we end up with basically the same thing?

As an extension the my question, I can't help but wonder what would happen if, coincidentally, each of the 4 squares all just happen to have a pixel with the same max value. Surely this isn't a rare case, right? Suddenly all your training images look the exact same.


texts: Is it a good idea to train Neural Network for classification on dataset where each document has a different class i.e. no class is repeated again?

texts:
My goal is to build a recommendation model for which I want to use Neural Network (LSTM). The user will give some input keywords and the model should return the suggestions (classes) based on relevance to the input keywords. My dataset contains few thousand documents and each document has exactly one class, so 1000 documents = 1000 classses. Every document contains 50-800 words.

My question is if it is a good idea to train a neural network on such data where there is only one instance of each class? If not then how can I achieve my goal.

I have already tried Naive Bayes and it works efficiently with such data. Any help would be appreciated.


texts: Is it possible to integrate Keras and TensorFlow code?

texts: I want to use the pre-trained VGG16 model of Keras, along with another TensorFlow model. I want to take the output from one of the layers of VGG16 in Keras, put it into the TensorFlow model and train only the latter. Is this possible if I use a TensorFlow backend for Keras?


texts: How to find what values are assigned to labels that where encoded using LabelEncoder?

texts:
places = ['India','France','India','Australia','Australia','India','India','France']

  Here places are the DataFrame Series, now how can I find that which
  label was encoded with values like India = 0 , Australia = 1 ,France =
  2.
  This is ok for few labels what if there are 100's of labels available
  in a huge dataset.


texts: Supervised learning for variable length feature-less data

texts:
I have data in following form:

a: 1,2,3,2,3
a: 2,4,5,6,7,8,0,9,7,6,5,6,2
a: 7,8,9,3,4
b: 4,5,3,5,6,3,5,1,2
b: 1,6,3,2,4,5
b: 2,4,5,6,7,8,0,9,7,6,5,6,2
c: 7,8,9,3,4
c: 4,5,3,5,6,3,5,1,2
...

(in reality, each case has about 100-200 numbers, though the length is variable)

Here, a, b and c are groups (their number is fixed - taken as 3 here) and the numbers indicate a vector associated with each case. How can I apply supervised machine learning with such a data so that if I get a new series of numbers, e.g. following:

3,2,3,4,1,5,6

I should be able to determine which group (a, b or c) does this case belongs to.

Following features of each list of numbers may be important:

length of series
mean value of series
variance of series
maximum of series
minimum of series
type of distribution of series (normal or non-normal)

How can I apply machine learning methods to such data. Thanks for your insight.


texts: Autoencoder gets ~0% accuracy / doesn't train at all

texts:
So I wanted to get into the topic of 'Autoencoder', and just tested how well it would work on random vectors of size 200.

model = keras.models.Sequential()
model.add(keras.layers.Dense(50, activation='tanh'))
model.add(keras.layers.Dense(200))

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
model.fit(vectors, vectors, epochs=50, batch_size=100)

Loss is somehow negative, and continously decreases. Accuracy is pretty much 0%, with values looking like x.xxxxxxxe-07.

The architecture is supposed to be input=200, hidden-layer=50, output=200.

It really doesn't learn anything. What did I overlook, or what is the actual right approach here?


texts:
I am trying to see how diversified a portfolio is, using the portfolio holdings weights and their return streams.

For example, let's say that the portfolio has $3$ stocks, so I will have the $3x1$ W (weight) matrix and $3x3$ correlation matrix C.

The following matrix multiplication $W^T * C * W$ will result in a single number that tells you how diversified the portfolio is. The bigger the number, the less diversified and vice versa.

One challenge is that the stocks may have different time periods. For example, stock A may have $3$ months of data, stock B may be $5$ months of data and stock C may have 1 year of data. Using the naive, standard correlation function will use the common date ranges between the stock pairs and calculate correlations.

This may be a problem if the date ranges are extremely different (e.g. stock A may be $1$ month of data, while stock B and C may have 5 years of data), which will skew the result number.

Is there a name for this phenomenon? What kinds of  techniques are out there to solve this kind of problem?


texts: degeneracy of a CNN having only 1 convolution kernel down to a fully connected NN

texts:
Imagine a CNN having only 1 convolution kernel C without pooling layers etc. There are 3 layers in the NN: input layer, hidden layer, output layer. The input layer is a vector X originally flattened from a matrix containing pixels of an image, which we call image matrix hereafter. If one uses only 1 kernel (nn) to do a convolution on the image matrix of the input layer, the hidden layer is just the result of convoluting the image matrix and expanding into a vector. The hidden layer vector Xc=CX, where C is a nonsquare matrix doing the convolution (different from the convolution kernel). The output layer outputs if the image is a dog or cat.

The cost/loss function is taken as the sum of (WXci+b-yi)^2 over i, i from 1 to the number of training data, where Xci and W are vectors, b is a scalar, and yi are 0 or 1 tagging if the training images are a dog or cat. (WXci+b-yi)^2 can be rewritten as (WCXi+b-yi)^2.

Here both W and C are weights that need training and backpropping. If instead one sees W*C as a single vector, the backprop is exactly the same as in fully connect NNs. Then the calculated result of this special CNN (having only 1 convolution kernel) will be the same as that of a normal fully connected NN?


texts:
I am new to Data Science and I am trying to solve this problem. It is a problem of supervised learning.
I have a dataset that for every day of a time interval, for every defined geographic point, has a set of features and a target value. At first I thought of having a dataset of rows for every date for every geographic point, and then train the model on those rows to do a regression. Given the date and corresponding features of the test set, I would then verify the goodness of my model.
Instead, I was suggested to use the value of the (n-1)-th day as a feature of the n-th day's row. Is this a good way to take account of the temporal correlation between one day and another? Will this allow me to have good predictions also in my test set? Is this method equivalent to a sort of Sliding Window Method with an interval of one day?
| date       | feature 1      | feature 2     | target[i-1] (as feature)   | target
|------------|----------------|---------------|----------------------------|-----------
| 2019-02-01 | a1             | b1            | 123                        | 234
| 2019-02-01 | a2             | b2            | 234                        | 342
| 2019-02-01 | a3             | b3            | 342                        | 549
| 2019-02-01 | a4             | b4            | 549                        | 623
...
| 2019-02-01 | a(N-1)         | b(N-1)        | ...                        | 234
| 2019-02-01 | aN             | bN            | 234                        | 978

This also means that, in the splitting of my training set and test set, I should pick the first one as those rows corresponding to dates before a certain threshold date and the latter as those rows corresponding to dates after that same threshold. This choice is not going to affect the predictivity of my model?
Thanks in advance.


texts: How does time needed for training differ between different batch sizes?

texts:
I've constructed a CNN in Python using Numpy, which is trained with mini batch gradient descent for MNIST digit classification. When training with a batch size of 1, the time needed for 5 epochs is about 1200 s, which is only about 40% slower than with batch sizes 16 to 256. For batch size 4 and 8 it takes about 940 s and 890 s respectively. When i train it over 5 epochs with batch sizes 16, 32 or 256 the time is about 850 s (about the same for all three sizes).
I expected the time to shorten with bigger batch sizes, as seen with batch size 1 vs batch size 16, but why is the time the same for batch sizes 16, 32 and 256? The larger the batch size, the fewer the iterations. I would expect the computational cost of training a larger batch to be minimized with my vectorized code. None of the batch sizes seem to use any disk memory. Why is the time difference so small and why does the time stagnate at about 850 s for 5 epochs?


texts:
I am calculating Hellinger distance for different vectors. I initially assumed that the value returned by it in in the range of 0 to 1.

However for the following two vectors I received Hellinger score as 1.0488088481701514, which is > 1.

vector_1 = [0.0,0.5,0.7]
vector_2 = [1.0,0.0,0.0]

Now, I am curious to know the range of Hellinger distance values. Please explain me why that value exceeded 1.


texts: How do I resolve this error: "expected string or bytes-like object" please?

texts:
I was doing Lemmatisation but I got the above error. I think it's because my data isn't in string form, is that correct? If so, which part of my data isn't in string form?

This is my data source: https://www.kaggle.com/c/fake-news/data


texts: Why sparse features should have bigger learning rates associated? And how Adagrad achieves this?

texts:
I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater learning rates for the features that are less common (or more sparse) when using Adagrad.
Now, why do we even want to have lower learning rates for the features that are most common?
Why do we even care if one feature is more sparse than any other feature?


texts:
Since CNN has been widely applied in DNA sequence data, I'm wondering why CNN is not often used for predicting phenotype from SNP data, given that
 SNPs are essentially parts of DNA sequences and retain the nature of ordering.

For instance, this paper https://arxiv.org/abs/1611.09340 states that "... most of these techniques(CNNs) are based on sequence data where convolutional or recurrent networks are appropriate. When the full DNA sequence is unavailable, such as when data is acquired through genotyping, other methods need to be used..."

Please explain regarding the algorithms rather than the biology side(e.g. SNPs only explains a small portion of genetic variation, etc)

Hey guys, I just wanna remind that I'm addressing the use of SNPs(or you can include other types of variants as well)...not full DNA sequence data. It's very common to use CNN on full DNA sequences to predict variation loci. And it's easy to understand as it's natural to use CNN on sequential data.

But what I don't understand is why not using CNN (1)on SNPs (since SNPs retain the major spatial structure of DNA sequences) (2) to predict phenotype?


texts:
What are the ways to speed up the fit of the model on large files (more than 200 mb)? I tried to divide into pieces (chunks) or use dask but the effect is unsatisfactory. I use 16GB RAM and i7 2.2GHz processor 4 core. It remains to buy a cloud service or use a better computer?
My clasiffier

clf = xgb.XGBClassifier(n_estimators=500,
                        n_jobs=4,
                        max_depth=9,
                        learning_rate=0.05,
                        subsample=0.7,
                        colsample_bytree=0.7,
                        missing=-999,
                        gamma = 0.1)
# Number of variable > 400
# Number of rows > 200000


texts: Logistic Regression - Coefficients have p-value more than alpha(0.05)

texts:
I am very new to the machine learning field and have been practicing logistic regression on few sample data sets.
I have built a Model using the logistic regression algorithm. Few of the coefficients have a p-value of more than 0.05 (which is the alpha I am  considering).

R-Code for building the model and a summary of the model is given below.
model.bank.1 <- glm(y~., data=bankfull, family ="binomial")
summary(model.bank.1)

Now, before considering AIC, Residual/Null Deviance, confusion matrix and ROC for evaluating. I have observed that the p-values for some of the independent variables is more than 0.05(age has very high p-value, what does this mean?). In such a case, what should be done? Should i straight away remove all such predictors from my model ? Is there any way by which i can make p-value of these predictors less than 0.05 ?
What are the things needed to be checked before going on to evaluate the model using the AIC,Deviance, Confusion Matrix and ROC measures ?
Edit 1: I've tried standardizing the numerical columns, but there is no change in the model at all.


texts: Time series problem with LSTM is not predicting correctly

texts:
.

My Initial dataset looks like this:

Text sample of Dataset:

    TranDate    TradeAmount
0   2017-01-01  525.00
1   2017-01-02  131.20
2   2017-01-03  904.00
3   2017-01-04  900.00
4   2017-01-05  1354.38

I have to predict the TradeAmount for next day. based on past 60 days.

X = [past trade amounts of 60 days]

y = [trade amount of next day]

I have total 1122 data.

I tried with LSTM like this:

days = 60
df.sort_values(by ='TranDate',inplace=True)
data = df.filter(['TradeAmount'])

          ...............

scaler = MinMaxScaler(feature_range=(0,1))
scaled_data = scaler.fit_transform(dataset)

model = Sequential()
model.add(LSTM(50,return_sequences=True,input_shape = (x_train.shape[1],1)))
model.add(LSTM(50,return_sequences=False))
model.add(Dense(25,activation='relu'))
model.add(Dense(1))

model.compile(optimizer='adam',loss = 'mean_squared_error')
model.fit(x_train,y_train,batch_size = 950,epochs=1000,shuffle = True)

  I tried with epochs=100/500/800. batch_size = 32/256/500. It was giving a mean value of all inputs.

The above code is giving output like:

Textual Form:

    TradeAmount Predictions
1010    120.3   1464.579956
1011    5422.8  728.806152
1012    300.0   91.881477
1013    699.5   411.360626
1014    361.5   558.223267
... ... ...
1117    4194.0  1153.897339
1118    120.3   670.037781
1119    1504.9  631.024292
1120    1634.6  36.828255
1121    180.6   495.607788

Predicted value is far away from original. How can I improve this??


texts:
I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in the data. My question is: Can focal loss  be utilized for extraction and classification task to increase the accuracy? Focal loss has been applied on object detection task and for image classification task. The link is below. I want to use this on text classification task.

https://shaoanlu.wordpress.com/2017/08/16/applying-focal-loss-on-cats-vs-dogs-classification-task/

http://openaccess.thecvf.com/content_ICCV_2017/papers/Lin_Focal_Loss_for_ICCV_2017_paper.pdf


texts: How to run a model in an application using gpu (without CUDA)

texts: I am looking for a way to utilize a computer’s gpu without using cuda (or any installable software). The reason for this is I have an application where the neural network runs on a user’s computer and I can’t assume they will have the knowledge/ spend the time and effort to install CUDA. I see that it is possible to utilize the gpu using WebGL but from my reading, it sounds like due to other limitations with this approach it is not actually faster than using cpu. Does anyone know how to do this maybe using openCL or some other way to do it?


texts: I was wondering what happens when an image not in the training set is provided to the model in a multiclassification problem? Does it just classify something which is close to this image?


texts: Which data set to use to find correlation between Predictor and response variables? Test data set? Training data set? or the entire data set?

texts:
I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use linear model to fit them.

No split of training set:test set is given. I have one data set with 10000 samples. I was planning of splitting this data set in the ratio 80:20 for training and testing respectively.

Now I have a doubt on whether the correlation should be found after the data has been split or is it better to check correlation with the entire dataset? Which is the std way of doing?

P.S. I am going to use the R programming to do the same.


texts: How do we know Kernels are successful in making data linearly Separable?

texts:
When we have linearly inseparable datasets and we are using machine learning algorithms such as SVMs, we use kernels to implicitly map datapoints into a feature space that makes them linearly separable.

But how do we know if a kernel has indeed, implicitly, been successful in making the datapoints linearly separable in the new feature space? What is the guarantee?


texts:
I read this introduction about AdaBoost (http://www.cs.man.ac.uk/~nikolaon/~nikolaon_files/Introduction_to_AdaBoost.pdf), and am curious why confidence for each model is defined as

$$\alpha_j=\frac{1}{2}\log\frac{1-\epsilon_j}{\epsilon_j}$$

(from page 48 of http://www.cs.man.ac.uk/~nikolaon/~nikolaon_files/Introduction_to_AdaBoost.pdf).

Could anyone provide any source for explaining this?

Thank you!


texts:
Recently, I started working on Ridge and Lasso regularization for Linear and Logistic Regression.
My doubts are given below:

Is the penalty the same (by same proportion) for all the coefficients or is it based on variable importance?
If it is the latter I believe we can directly apply regularization rather than spending time in feature selection.
Whether the Multi-collinearity is taken care by ridge and lasso regularization?

Thank you.


texts: Suitable sample data set to test machine learning algorithms

texts:
I'm new to Machine Learning and I just came across the sci-kit package. On this interesting page there are many toy data sets used to test different clustering algorithms. Each data set has a unique pattern and some algorithms perform better than others depending on the data sets.
I want to ask why these data sets are chosen as tests for the algorithms? What are the properties for them to be suitable for use in testing? Are there any other data sets with common attributes that are used for the same purpose? Do they have certain names that I can read more about?
Thank you.


texts:
I am looking for the name of a "time dependent pie chart". I assume that this exists already, but since I don't know its name I have a hard time googling it.

Description:
Let the radius $r$ be the time. Then I get a circle with radius $r$ for every time step. This circle can then be used to represent proportions. By moving out from the middle you could see how the proportions change over time.

Similar but not the same:

Multilevel pie chart/Sunburst Chart


texts: What are Hybrid Classifiers used in Sentiment Analysis?

texts: What are Hybrid classifiers used for sentiment analysis? How are they built? Please suggest good tutorial/book/link for reference. Also how are they different from other classifiers like SVM and Naive Bayes?


texts: Antonym search for expanding search terms

texts:
I some code that identifies ngrams of interest in a small bit of text, then searches for the ngram in a larger bit of text to provide a snippet of the term used in context.  One ngram might be unordered numerical set.  Right now, if I can't find unordered numerical set, I start trimming the string from the left to see if numerical set is available in the larger text.

What I would like to do before I trim from the left, is to see if the any of the words have an antonym, like ordered for unordered.  This is because my ngram might be defined in one way, but used in the opposite.

Is there a known list or other way of finding out if there is a highly correlated antonym I might try for a set of words?

For instance, I think I could write some regexps that look for prefixes, like "un" or "dis" and see if the word that results from removing these prefixes is a valid english word.  This seems likely to be already solved, so before I try to create anything, I want to find out what might already exist.

I am currently using Python using Gensim, NLTK, and Word2Vec for the rest of my processing if it matters.


texts: Calculate the similarity between pairs of time series data

texts:
I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have similar trend & seasonality.
I would like to calculate/quantify how similar Store B,C,D,E is to Store A respectively.
I know how to calculate the simple cosine distance and Euclidean distance, and I have experience dealing with time series data (e.g. ARIMA, Prophet), but never had to deal with finding (i.e. quantifying) the similarity / distance between time series data.
Some google search suggests that "dynamic time warping" could be the method to go. I have not yet heard of this algorithm before, so not sure if this is indeed the right direction.
Additionally, can I still perform cosine distance and Euclidean distance on time series data? Is that doable?
How do I approach this issue? Any insights are appreciated.


texts: By now i thought that fuzzy clustering can be applied to any kind of data-sets but now i have heard that it can be applied only to specific data-sets that involve the concept of probability, is that true?


texts:
Is there a method/class available in Apache Mahout to perform n-fold cross validation?
If yes how it can be done?


texts: Is there a way to manually remove support vectors from a scikit-learn model?

texts:
I'm in a situation where I need to limit the number of support vectors (SVs) in my support vector machine binary classifier. A simple way to do this would be to manually remove the least important SVs. Does anyone know of a way to do this?

I've tried manually modifying the following fields of the SVM to remove SVs, but without success:

clf.dual_coef_
clf.intercept_
clf.n_support_
clf.support_
clf.support_vectors_
clf.shape_fit_

where clf is my RBF SVM.

Note that increasing regularization (by decreasing C) does not necessarily reduce the number of SVs (https://dgroppe.com/2018/01/21/increased-regularization-does-not-necessarily-decrease-the-number-of-support-vectors/)


texts: How can I extract skills from a resume using python?

texts: I am looking at how to extract a set of skills from an unstructured resume format using Python. Please note that, though I have a basic idea of NLP, I am completely new to Python.


texts: Image resizing to be done programatically or taken care by model config file in tensorflow?

texts:
For Deep Learning using Tensorflow, is it necessary to resize the images to a predefined width and height before before training using a model?
     OR

In model configuration file , I noticed this snippet? Does this mean all training images are resized automatically by tesorflow ?

  image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 600
        max_dimension: 1024
      }

My positive training site has varied sizes..

corners_40N1_0.jpg 323*283

corners_40N1_1.jpg 318*271

corners_40N1_2.jpg 341*306

corners_40N1_3.jpg 318*329

corners_45O16_0.jpg 250*280

Negative training data are of equal size.
56K7NE_80.jpg 360*360

56K7NE_81.jpg 360*360

56K7NE_82.jpg 360*360

56K7NE_83.jpg 360*360

56K7NE_84.jpg 360*360


texts: What is the best way to visualize data that is over 100% capacity

texts:
I’m trying to visualize some data that is usually under 100% (for example, workload, or activity) but on some occasions it will rise over 100, meaning that the resource has been used too much.
I was thinking on something like the rings Apple use in their activity on Apple Watch, which spins several times before reaching the final value, indicating that you have overdo. The problem is that that type of visualization requires to be dynamic and I need something static (to print on a report)
I can also use a bar chart with a defined boundary if the value exceed 100% but I’m afraid it won’t clearly represent values in small and big numbers.
So is there any type of chart that I’m clearly missing?
Any help would be appreciated.


texts: Regression on multiple datasets with a per-dataset variable

texts:
I have 10 datasets, each with the same variables (e.g., age and income) but different numbers of observations.
Let us now consider a categorical variable $X$ that can only take values $0$ and $1$ per dataset, meaning that it keeps the same value for all observations. For 5 datasets, $X=0$; for the other 5, $X=1$.
How do I create a regression model for a variable of these datasets (e.g., age) that takes into account this "meta-variable" $X$?
A simple solution would be to append a new column for $X$ to each dataset, where the same value is repeated for all observations, and then concatenate the datasets. However, I think there are better ways.


texts: How is this function (for updating a Stochastic Gradient Descent model) called without a parameter?

texts:
I'm in the middle of a Deep Learning Course offered by DataCamp and the example below was given for optimizing a SGD model:

As you can see, the function "get_new_model" requires one parameter: "input_shape".  However, in line 9 there is no parameter given.  Why doesn't this throw an error?


texts: Math behind 2D convolution for RGB images

texts:
I read many threads discussing why 2D convolutional layer is typically used for RGB images in neural network.
I read that it is possible to use 3D conv layer.
What I do not understand is the math behind it.
Say your image is 300 by 300, and the kernel_size = (3, 3) and filter = 16 for the Conv2D layer. Input_shape would be (300, 300, 3) because there are 3 channels(RGB).

Since the kernel is 2D, the convolution can only be done at 1 channel at a time. Is that correct?
Are the same kernel applied/convolved for the 3 channels? If so there should be 3 output but the dimension of the output would be (298, 298, 16). Is it averaged over the 3 channels?


texts: Do we need to add the sigmoid derivative term in the final layer's error value?

texts:
I have been studying professor Andrew Ng's Machine Learning course on Coursera. Currently, I am trying to prove the formulas for backpropagation, which is mentioned in Week 5 (in this document). Clearly, it mentions that
δ(L) = a(L) − y.

However, I have also came across this video on Youtube. In 4:30, it states that δ(3) = -(y - y^) * f'(z(3)), where f is the sigmoid function. Since L = 3 and y^ = a(L), this means that δ(L) = (a(L) - y) * f'(z(L)). I've also derived the equations and got exactly like in the video. But this has an additional "f'(z(L))" term, which is not the same as in Coursera. I have even implemented this equation in the programming assignment for Week 5, but it only works without the term.

So my question is why Coursera's formula didn't have the last term "f'(z(L))" and why the formula works without it? It seems quite irrational to not have the term in the formula. Can anyone help me to explain this, or correct me if I'm wrong? Any help would be greatly appreciated, thanks.


texts: Group_by 2 variables and pivot_wider distribution based on 2 others

texts:
Performing some calculations on a dataframe and stuck trying to calculate a few percentages. Trying to append 3 additional columns added for %POS/NEG/NEU. E.g., the sum of amount col for all observations w/ POS Direction in both Drew & A/total sum of all amounts for Drew **

Name     Rating   Amount    Price    Rate   Type    Direction
Drew     A        455       99.54    4.5    white   POS
Drew     A        655       88.44    5.3    white   NEG
Drew     B        454       54.43    3.4    blue    NEU
Drew     B        654       33.54    5.4    blue    POS
Drew     C        754       54.43    4.3    green   POS
Jon      A        454       65.23    3.4    blue    NEG
Jon      B        954       86.34    4.3    blue    NEG
Jon      B        545       34.54    4.4    green   NEG
Jon      C        454       65.45    3.4    green   POS
Jon      C        544       65.55    4.4    blue    NEU
Nick     A        675       54.33    3.4    white   POS
Nick     A        565       65.33    3.4    white   POS
Nick     B        343       54.44    6.4    blue    POS
Nick     C        656       65.33    4.3    green   NEG
Nick     C        655       94.44    3.3    green   NEU

To Get the Following Output Calculation for POS/NEG/NEU columns

Name   Rating   sum  count percent wm_price wm_rate mode_type POS NEG NEU
Drew    A                                                    .3735
Drew    B                                                    .3728
Drew    C
Jon     A
Jon     B
Jon     C
Nick    A
Nick    B
Nick    C

Here's what i got so far but im stuck implementing the pivot_wider for calculating/appending the % direction (POS/NEG/NEU) for each rating category of Each Name Any feedback appreciated!

df <- df %>% group_by(Name, Rating) %>%
  summarize(sum_rating = sum(Amount),
                count = n(),
                wm_Price = weighted.mean(Price,Amount),
                wm_Rate = weighted.mean(Rate,Amount),
                mode_Type = mode(Type)) %>%
  mutate(pct_rating = sum_rating/sum(sum_rating)) %>%
  pivot_wider(names_from = Direction, values_from = Amount/sum Amount)


texts: Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?

texts:
I am working on training RNN model on caption generation with REINFORCE algorithm. I adopt self-critic strategy (see paper Self-critical Sequence Training for Image Captioning) to reduce the variance. I initialize the model with a pre-trained RNN model (a.k.a. warm start). This pre-trained model (trained with log-likelihood objective) got 0.6 F1 score in my task.

When I use adam optimizer to train this policy gradient objective, the performance of my model drops to 0 after a few epochs. However, if I switch to gradientdescent optimizer and keep everything else the same, the performance looks reasonable and slightly better than the pre-trained model. Is there any idea why is that?

I use tensorflow to implement my model.


texts: is it wrong to use average='weighted' when having only 2 classes?

texts:
In the book 'Text Analytics with Python', the author provides model_evaluation_utils.py
In the code of the .py he does:
metrics.precision_score(true_labels, predicted_labels,average='weighted')

I have two questions regarding it:
1- per the documentation average='weighted' shall only be used when having more than 2 classes, right? Why is he using average='weighted' when using only 2 classes?
2- Why do I get different results when I run?
print('Recall:', metrics.recall_score(test_sentiments,predicted_sentiments,pos_label='positive'))
print('Recall:', metrics.recall_score(test_sentiments,predicted_sentiments,average='weighted'))

(I only have 2 classes in the data)

ps: I think that by using average='weighted' he is giving the wrong result, because the code doesn't know which one is the positive class, here is a like to the code of model_evaluation_utils.py https://github.com/dipanjanS/practical-machine-learning-with-python/blob/master/notebooks/Ch05_Building_Tuning_and_Deploying_Models/model_evaluation_utils.py


texts: I want to use phone type (e.g., "iPhone 6s", "Lenovo K8 Note", etc.) as a feature in a model.  Does anyone know of a way to obtain release date, price, etc of a phone (besides scraping)?  That seems like it would be more useful than one-hot encoding.


texts: Difference between LASSO penalty in neural network and just LASSO regression

texts:
I wonder whether those two have any significant differences.

I think in neural network, the lasso penalty put on the loss function makes the model simpler and introduces more sparsity by automatically reducing some of the features to 0. I assume that's basically the lasso regression is doing?


texts: How and When features are attached to target label

texts:
I am using Mallet CRF library and having training set sequences like below.

KFC     Bangalore  INDIA        <--- Sequence
PLACE   CITY       COUNTRY      <--- Target Label

During training/validation time, I don't have any features.
While testing time, I give features like below.

KFC                 Bangalore            INDIA                  <--- Sequence
PLACE_LIKE(1.0)     CITY_LIKE(1.0)       COUNTRY_LIKE(1.0)      <--- Features

Since there is no features in the training time, will these runtime features are useful?
I believe, the association of features and target label is happening during the training time itself so any new feature directly comes during runtime will not get associated to target label and useless, is that correct?
Can i attach Bangalore with CITY_LIKE(1.0) while training and pass lower weight feature CITY_LIKE(0.5) during testing time? Will that has impact?


texts: Relation between an underlying function and the underlying probability distribition function of data

texts:
I heard and read a lot of times the following statements and got a lot of confusion over time.

Statement 1: The goal of machine learning is to get a function from
the given data
Statement 2: The goal of machine learning is to find the underlying
distribution function of the given data

From the above two statements, I generally interpret that the underlying function is a probability distribution function of the given data.
But I did not understand the relation between the probability distribution function and the function we want to get for a particular task.
Let us consider the following example.
A random experiment $E$ has a sample space $\Omega$ and I defined two random vectors $X_1, X_2$ on $\Omega$. I am using a neural network for my task. The domain of the neural network is the range of $X_1$ and I am expecting the range of the neural network to be the range of $X_2$ with correct mapping that satisfies the data. Let $f$ be the actual function that we want the neural network to become that maps from the range of $X_1$ to the range of $X_2$. Assume that I have the joint probability distribution $P(X_1, X_2)$ for my dataset.
Now, what is the relation between the actual function $f$ we are approximating using the neural network that transforms the range of $X_1$ into a range of $X_2$ and the joint distribution $P$?