texts:
I make small CNN from scratch to classify barcodes. I have two classes: one for images with barcodes and second for all what isn't barcodes (items, animals, landscape, furniture, people). I got good training accuaracy (80%+ after 50 epochs) but my validation accuracy is constant and around 50% and it is my main problem. I am aiming in around 80% for validation accuracy. I use some technique as dropout, decrease number of parameters, data augmentation, etc. My train set is 480 images (240 - barcodes, 240 the rest) and my validation set is 120 (60\60). What should I do to achieve about 80% accuracy on the validation set. I use keras and tensorflow framework. Here my CNN:

model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(224, 224, 1), padding='same'),

tf.keras.layers.MaxPool2D(3, 3),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.2),

tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),

tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.2),

tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),

tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.2),

tf.keras.layers.Flatten(),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(loss='binary_crossentropy', optimizer="adam", metrics=
['accuracy'])


texts:
For those who do AI or Machine Learning, is it possible to analyze video? For example, here’s a tidbit from an article I’ve been reading:

  in basketball, data can help coaches determine if a player executed a jump shot well because he was skilled, the pass was good or the defense was bad. If he is skilled, he can be used to lead key strategies. If the pass is good, it identifies players that work well together and use them for the same. If the defense is bad, coaches can strategize plays that take advantage of this weakness.

How would I measure this with AI? I’d like to find the true value of a high school player and how they’d affect their college team.


texts:
I want to calculate:
True_Positive, False_Positive, False_Negative, True_Negative

for three categories. I used to have two classes Cat Dog and this is the way I used to calculate my confusion_matrix

y_pred has either a cat or dog
y_true has either a cat or dog

from sklearn.metrics import confusion_matrix
confusion_matrix_output =confusion_matrix(y_true, y_pred)
True_Positive = confusion_matrix_output[0][0]
False_Positive = confusion_matrix_output[0][1]
False_Negative = confusion_matrix_output[1][0]
True_Negative = confusion_matrix_output[1][1]

Now I have three classes 'Cat' 'Dog' 'rabbit'
Y_pred has Cat Dog rabbit
y_true has Cat Dog rabbit

How to calculate True_Positive, False_Positive, False_Negative, True_Negative?


texts: Bayesian linear regression / categorical variable / Laplace prior

texts:
I'm trying to do feature selection in the bayesian framework with a Laplace prior with the following code in Python;

Code:

#nb_predictors = len(df.columns) - 1 # we remove the target variable
nb_predictors = 7
beta = list()

with pm.Model() as model:
    # Define priors
    intercept = pm.Normal('Intercept', mu=0, sd=1/25)

    for cpt in range(1, nb_predictors + 1):
        beta.append(pm.Laplace('beta_' + str(cpt), mu=0, b=np.sqrt(2)))

    # Define Likelihood
    logit = intercept + beta[0] * df['satisfaction_level'] + beta[1] * df['last_evaluation'] \
                      + beta[2] * df['number_project'] + beta[3] * df['average_montly_hours'] \
                      + beta[4] * df['time_spend_company'] + beta[5] * df['Work_accident'] \
                      + beta[6] * df['promotion_last_5years']

    likelihood = pm.Bernoulli('left', pm.math.sigmoid(logit), observed=df['left'])

What I was wondering is what happen if I add a new categorical variable (by one hot encoding), like this sales variable. Can I still use a laplace prior and observe if the density is close to 0 (so it is probably not related with the target variable) or it makes no sense with categorical variable ? and it only works with continuous variable ?

# Define Likelihood
logit = intercept + beta[0] * df['satisfaction_level'] + beta[1] * df['last_evaluation'] \
                  + beta[2] * df['number_project'] + beta[3] * df['average_montly_hours'] \
                  + beta[4] * df['time_spend_company'] + beta[5] * df['Work_accident'] \
                  + beta[6] * df['promotion_last_5years'] + beta[7] * df['sales_low'] + beta[8] * df['sales_high']


texts: Simple Question: How to Load corpora (.txt files) in R

texts:
Context: I have 47 .txt files which are all technically small corpora. They have been lowercased and full-stops have been removed.
Question: Would anyone know how to or be able to point me to a resource that explains how to load these in R? I wish to lemmatise them and extract collocations (nearest neighbours) within a term with a window size of 3 + their counts per corpus + proportion (count of collocate/ total tokens in corpus).
Any resources/ tips would be much appreciated. It might be a simple solution. I am completely new to handling corpus data, but have tried the below code so far which did not work:
corpus <- load.corpus("corpus1.txt")


texts:
BRAT (brat rapid annotation tool) can be used for named-entity annotation:

Can BRAT be used for text classification annotation? I.e., given the text, annotate whether it belongs to some classes?


texts: convert some observations into variables

texts:
I have a table formatted like the following table:

Feature amount  ID  Location
Feat1      2    1   US
Feat2      0    1   US
Feat3      0    1   US
Feat4      1    1   US
Feat2      2    2   US
Feat4      0    2   US
Feat3      0    2   US
Feat6      1    2   US

Let's say I have 200 different IDs.
I want to convert all different features into variables and amount into observations, so I combine rows with same ID into one row. For example,

Feat1 Feat2 Feat3 Feat4 Feat5 Feat6 ID Location
  2     0     0     1    NA    NA   1    US
 NA     2     0     0    NA    1    2    US

Is there a good way to do it either in Python (pandas) or R?

Thanks in advance!


texts: How to model a Machine learning problem considering links between features

texts:
Context:

To predict employee turnover ( will an employee leave? ), I have used one of the classification algorithms (LDA) to train my dataset, and then make predictions.

The dataset is quite small (500 lines), some 20 features, the following are some examples:

Age
Years_Spent: Years for an employee has spent in the company.
Department (IT, Commercial, Management...)
Salary
Salary_Increase
Sale_Bonus
...

However, HR Experience tells us that:

For employees whose Years_Spent < 1.5, Salary_Increase is a feature that does not have any impact on turnover (Because of Salary_Increase > 0 only when Years_Spent >1.5).
Sale_Bonus will not have any impact on those who are not commercials. (Because IT guys will never recept sale bonus)

Here comes the problem:
If I set Salary_Increase = 0 for employees whose Years_Spent <1.5 and Sale_Bonus = 0 for those who are not commercials, the classification algorithm will take 0 as a very small value, so a possible conclusion could be drawn by an algorithm: "employeeA will leave because he never receives sale_bonus",  (However, in reality, employeeA is from IT department, employeeA receive never sale_bonus and employeeA will not leave because of that), as we see, the constructed model is not correct.

My question is: How to handle this kind of problem so that HR experience can be understood by classification algorithms?

Thank you for your patient reading and kindly welcome all sorts of discussions!


texts: Why would batch normalization allows us to use higher learning rate in the neural network?

texts:
I am doing some study about the BatchNormalization:
https://towardsdatascience.com/batch-normalization-8a2e585775c9

In the article, it says:

Using batch normalization allows us to use much higher learning rates, which further increases the speed at which networks train.

Could anyone please share their thoughts on why would batch normalization allow higher learning rate? Thanks!


texts: How do I approach grouped anomaly detection?

texts:
I currently have a large stream of data, with points such as HTTP request/response codes (200, 404, 500, etc.). Essentially, I want to perform anomaly detection for when too many signals that are NOT 200 are received. This means that the signal to be analyzed is dependent on groupings of data points (i.e., a single data point indicating that a 404 was sent is not good enough, only a cluster of 404's close together with respect to time means something).

Is there a good algorithm/methodology to approach this? I was previously thinking of having a moving window counter of the previous codes and basing it off of this, but I feel like the window size for example, is very subjective, and I am not sure how to tune that.


texts: How can one be assured that generative models are not memorizing dataset, and that they will generate an unique image outside of dataset?

texts:
If all GAN can do is capture the probability distribution of the dataset, then shouldn't they be similar to handing out images from the dataset?

How can we verify that the images that they generate are unique images outside of the dataset?


texts: Should highly correlated features be omitted before applying Lasso?

texts:
I would greatly appreciate if you could let me know whether I should omit highly correlated features before using Lasso logistic regression (L1) to do feature selection.

In fact, I want to use logistic regression with L1 to do prediction as well as feature selection. However, some of my features are highly correlated e.g., -1 or 0.9. Should I omit them before applying Lasso or let the Lasso decide it?

Really, I read in Mr. Raschka’s book (Python Machine Learning) that

  regularization is a very useful method to handle collinearity (high
  correlation among features).

However, this kernel  (by referring to Wikipedia) states that keeping correlated features in the model would adversely affect the feature selection but it doesn't impair the predictions.


texts: XGBClassifier and RandomForestClassifier variable importance plots are very different

texts:
I have one dataset, and decided to use XGBClassifier to get a variable importance plot from it.

I was a little surprised that some features that I assumed were insightful had no appreciable value.  With this in mind, I used an sklearn RandomForestClassifier to create another variable importance plot.

To my surprise the two were very different! Many variables that appear on one don't appear on the other.

I used as similar parameters as I could, between the two models.

Here's what they look like to give a general idea:

Has anyone had this similar experience, and if so, what kinds of reasoning for this might there be? It's a little frightening to think that many Kagglers and others really depend on these plots and to see that they can vary so much...makes me wonder.

Thanks.


texts: What are the best practises to decide whether a variable is categorical?

texts:
What are some of the systematic ways to categorise variables into categorical or numeric? I believe using only intuition in such scenarios can many-a-times lead to major irreversible errors. What are the best strategies when categorising variables?

For example, the dataframe I'm working has several categorical variables such as is_holiday that has labels for several holidays. However certain variables like visibility_in_miles suggest that those too need to be treated as categorical. part of the reason is that while most variables have hundreds of unique values, some have only 9 points.


texts: sklearn's cross_validate does not work with catboost

texts:
I would like to use cross validation with catboost. Since I do not just want to use catboost but also sampling I am using a pipeline and hence cannot use catboost's own cross validation (which works if I just use catboost and not a pipeline). So I want to use sklearn's cross validation, which works fine if I use just numerical variables but as soon as I also include the categorical variables (cat_features) and use catboost's encoding, cross_validate doesn't work anymore. Even if I don't use a pipeline but just catboost alone I get a KeyError: 0 message with cross_validate. But I don't understand why.
This is part of my code that doesn't work:

from sklearn.model_selection import cross_validate
model = cb.CatBoostClassifier(**params, cat_features=cat_features)
cv_score = cross_validate(model, X_train, y_train, scoring='roc_auc', cv=5, return_train_score=True)


texts: Bidirectional GRU: validation loss stuck on plateau diverges from well performing training loss

texts:
tl;dr: What's the interpretation of the validation loss decreasing faster than training loss at first but then get stuck on a plateau earlier and stop decreasing?

The accuracy behaviour is similar.

Background:

The task is multi-class document classification with a high number of labels (L = 48) and a highly imbalanced dataset. It's a NLP task, using only word-embeddings as features.

The model implemented is a Recurrent Neural Network based on Bidirectional GRU layer. The full implementation is as follows:

inp = Input(shape=(MAX_LEN,))
x = Embedding(num_words, embed_size, weights=[embedding_matrix])(inp)
x = SpatialDropout1D(0.2)(x)
x = Bidirectional(GRU(80, return_sequences=True, dropout=0.5, recurrent_dropout=0.5))(x)
avg_pool = GlobalAveragePooling1D()(x)
max_pool = GlobalMaxPooling1D()(x)
conc = concatenate([avg_pool, max_pool])

dense = Dense(num_classes)(conc)
bn = BatchNormalization()(dense)
outp = Activation("softmax")(bn)

adam = Adam(lr=LEARNING_RATE)

model = Model(inputs=inp, outputs=outp)
model.compile(loss=focal_loss,
              optimizer=adam)

Relevant insights:

Highly regularized: SpatialDropot, Dropout, Recurrent Dropout, Batch normalization...
Imbalanced sensible loss function: Focal loss
Adaptative optimizer: Adam

Training setup:

Epochs: 300
Learning rate: $1e-4$
Batch size: 32


texts: How to set the number of neurons and layers in neural networks

texts:
I am a beginner to neural networks and have had trouble grasping two concepts:

How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever.
How does one decide the number of neurons in each middle layer? Is it recommended having an equal number of neurons in each middle layer or does it vary with the application?


texts: Architecture for linear regression with variable input where each input is n-sized one-hot encoded

texts:
I am relatively new to deep learning (got some experience with CNNs in PyTorch), and I am not sure how to tackle the following idea. I want to parse a sentence, e.g. I like trees., one-hot encoded the parse output of each word, and feed that into a ML system. The output of each sentence is a floating-point number. As an example, the sentence I like trees. could be pre-processed and encoded as fixed-size feature vectors per token:

[[0 1 0 0 1] [1 0 0 0 0] [1 0 1 1 0] [0 0 0 0 1]]

or flattened

[0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 1]

However, the length of sentences can differ of course. From what I know there are some fixes to this. Using padding (when shorter than your defined cut-off) or cutting off the length to that cut-off (when longer). Another solution I often see mentioned is the use of an RNN/LSTM but I have no experience with them (only the basic theoretical notion).

The expected output (label) of this sentence could be something like 50.2378.

My question, then, is which architecture is best suited for this task? RNNs are popular in NLP so I am leaning towards them, but I am not sure whether a regression task fits well (or how it fits) with the architecture of an RNN. I have looked for RNNs and regression but I can only find use cases involving time series, not NLP or one-hot encoded features.

In essence, what I have is a dataset of sentences that are preprocessed to get some features per token. Let's assume for brevity sake that these features are syntactic, e.g. the word's POS tag and its dependency information (e.g. subj, obj, and so on). These features would then get one-hot encoded (I assume) to get an easy-to-use dataset. The input, thus, are sentences encoded in such a way that the information of tokens is shown, such as the flattened example above.

The output for every sentence is some floating-point number, representing some value that has been calculated beforehand in theory these will indicate the sentence's equivalence with its translation, but that is not at all important for the system. The important part is that every input is a sentence mapped to a number.

Some dummy data. The actual sentence, it's vector representation, and the output.

I like trees.                   [0 1 0 0 1 1 0 0 0 0 1 0 1 1 0 0 0 0 0 1]                           50.2378
What is that thing?             [1 1 0 1 0 0 0 0 1 1 1 0 0 0 0 1 1 0 1 0 0 1 1 0 1]               20.1237
Who are you?                    [0 1 0 0 0 0 0 1 0 0 1 0 1 0 0 0 1 1 0 1]                           1.6589
The cookies smell good today.   [0 0 1 0 1 1 0 1 1 0 1 1 1 0 1 1 1 0 1 0 1 0 1 1 1 0 0 0 0 1]     18.6295
I do!                           [0 1 0 0 1 1 0 1 1 0 0 0 1 0 1]                                     24.5489

The goal, then, is that an unseen given sentence can be pre-processed and one-hot encoded and given as input, and that an output (floating-point number) is predicted based on the trained model/function.


texts: What is the difference between (a) taking the logarithm of a set of numbers (b) dividing the set of numbers by an integer. Both appear to reduce the scale of set of numbers, so can they be used interchangeably?


texts: Why does Adam optimizer work slower than Adagrad, Adadelta, and SGD for Neural Collaborative Filtering (NCF)?

texts:
I've been working on Neural Collaborative Filtering (NCF) recently to build a recommender system using Tensorflow Recommenders. Doing some hyperparameter tuning with different optimizers available in the module tf.keras.optimizers, I found out that Adam and its other variants, such as Adamax and Nadam, work much slower than seemingly less advanced optimizers, like Adagrad, Adadelta, and SGD. With Adam and its variants, training each epoch takes about 30x longer.
It came out as a surprise to me, knowing one of the most cherished properties of Adam optimizer is its convergence speed, especially compared to SGD. What could be the reason for such a significant difference in computation speed?


texts:
I want to plot large heatmaps (say a matrix $500 \times 500$). I can do it in Python/matplotlib.pyplot with pcolor, but it is not interactive (and I need an interactive heatmap). I have tried with D3.js but what I found is aiming at displaying small heatmaps: http://bl.ocks.org/tjdecke/5558084
Naively extending this example with a bigger matrix (e.g. $500 \times 500$) can crash the web-browser.

So, can anyone point me toward a good way of displaying and interacting with large heatmaps with a web-based technology: I want to be able to interact on a web-page or a ipython notebook.


texts: Why there is only one type of artificial neuron?

texts:
I find it strange that so many deep learning tricks and improvements have been invented in the past decade but I never heard about someone trying out different models of the artificial neuron other than the one based on perceptron:
y = w*x + b

I haven't been able to find much info on this online which is surprising. I am not a machine learning expert but from the little I know, it would make sense to me to at least experiment with other options. A trivial example, what would happen if there was a layer in a network consisting of neurons where
y = w*x^2 + b

Maybe there is an obvious answer to why my suggestion above isn't a good idea but I would prefer an answer that explains why generally this is not being looked into (at least as far as i know).


texts: What's a good machine learning model for an univariate data set?

texts:
Here's my problem scenario:- I have to come up with a power equation as a function of frequency. The plot fits well with a higher order polynomial (4th or 6th) :-

$$Power = \theta_0 + \theta_1 fr^1 + \ldots + \theta_6*fr^6$$

(This is from MS Excel's trend line for a scattered plot) Frequency (x axis) ranges within fixed limits $f_1$ to $f_2$ and I have data from $100$ different devices for this kind of frequency sweep.

With this limited data set what would be a good ML model to train and generalize the coefficients that would work for any unseen device?

Thanks in advance!

Edit:-

While I can't share the exact data-set here but let me share some info about the data- $fr$ ranges from $4060(f_1)$ to $4165(f_2)$; I have an option to go granular i.e step size of $+1$ .. currently I am going $+5$

Where $Pij$ is the power value for the $i$th device and $j$th frequency sample
Question: Shall I treat each device as an example and each frequency as a feature? In that case the problem becomes multi-variate and I don't want that. I want the equation to remain exactly as stated above. The more desired option is to treat each frequency as an example, then how do I treat each of the $100$ devices? definitely not as features.. How to model the problem space into a feature vector $X$ and ans vector $Y$ and param vector $\theta$?


texts: Linear Regression with vs without polynomial features

texts: I have a conceptual question about why (processing power/storage aside) would you ever just use a regular linear regression without adding polynomial features? It seems like adding polynomial features (without overfitting) would always produce better results. I know linear regression can fit more than just a line but that is only once you decide to add polynomial features correct? My experience is with python using sklearn's libraries.


texts: How to get an intution for generating Keras CNN layers for a model

texts:
I am going through tutorial for handwritten text recognition. And to do hand written digit recognition the author has constructed a Keras model as follows:

# # Creating CNN model

input_shape = (28,28,1)
number_of_classes = 10

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=input_shape))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))

model.add(Dropout(0.5))
model.add(Dense(number_of_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),metrics=['accuracy'])

model.summary()

history = model.fit(X_train, y_train,epochs=5, shuffle=True,
                    batch_size = 200,validation_data= (X_test, y_test))

model.save('digit_classifier2.h5')

Source (here)

I am very confused that on how has the author choose these layers. I know how Conv2D works by applying filters to an image, I know what is activation function. In short I have a rough understanding of what each term means.

What I am finding it difficult is how do I know what is happening in each step of this code?
For example lets take this python code:

values_List=[11,34,43]
for index, num in enumerate(values_List):
    print(index,num)

I know that line 1 initializes a list named values_List
Line 2 iterates through this list
Line 3 prints output as (index of a number , number)

This python code is easy to understand and debug. But I am confused that if there is any error inside the keras layers. How do I proceed to debug this Keras code ?


texts:
I'm running an LSTM model to classify astrophysics time series data. I am attempting to classify each time series as either a black hole (1) or not a black hole (0). When I run my RNN, the loss is not decreasing, even after I have tried different optimizers/activations/architectures. I cannot figure out how to solve this problem. Here is the link to my Jupyter Notebook (which contains more information on the problem).
Let me know if you have any advice.


texts: Use Large Existing Dataset to Extract Information From Text

texts:
What I have:
I have a large dataset of documents and their data.
So I have the text of about 1M documents, and I know, for example the invoice number, of each one.
What I need:
Is there a way to use my dataset to train a model to take in a the text of a new document and predict what the invoice number is?
I'm just getting my feet wet with ML, and I've used ML.Net to predict the document type (from about 12 possible) from my dataset using MultiClassClassification. There, the prediction will always be from one of the 12 possible labels.
Here, I need it to use my training data to predict a new label that it wasn't trained specifically on.


texts: Creating Strings corresponding to Location Co-ordinates

texts:
What are some Python libraries which can convert a (X,Y) tuple to strings?

(1.23,4.56) yields strings “1_4”, “12_45”, “123_456”.


texts: Custom Loss Function for Mixing Sparse and Dense Features for a Prediction Problem

texts:
I have a largely uncorrelated feature space of about 40 dichotomous features, using which I'm trying to predict a continuous target variable.
Now, some of these features are very sparse (Active less than 10% of the time, with the rest as zeros). But the few times that these features are active may be really good predictors of the target. In most algorithms, these features will be mostly ignored due to how sparse they are - despite their predictive ability.
What are the ML algorithms/techniques I can use to remedy this and is there a possibility of customizing the loss function in order to give these features their due weight?
P.S. Are there any Bayesian approaches that would be more immediately reactive to these features? Would smoothing the data make any sense? And would embedding be of any use (Although, I only have 2000 training data points so using any deep learning models may not be appropriate)?


texts: Tools/tutorials for compiling corpora for NLP experiments?

texts:
I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much experience in processing large amounts of text data and whatever pipelines are involved.

Are there any tutorials on how to gather data and label it for a larg(ish) NLP experiment?

For example: BERT was originally trained on all of English Wikipedia. How do you go about gathering all of the text from Wikipedia's 5.9 Million + articles in a repository in the right format? How do you go about tokenizing such a large corpus Do things like NLTK and Beautiful soup still work on such large data sets?

If I have website or multiple website on a topic specific topic that I want to come up with some NLP models for, are there any Webscraping APIs that can pull all of that into one place?
Any tutorials, tools would be very welcome,
Thanks


texts:
There are several metrics for the quality of a graph clustering, e.g. Newman modularity. These enable you to compare two candidate clusterings of the same graph.

Does anyone know a metric that will answer the question "how modular is this graph"? For example the first of these two graphs is more modular than the second:
    o===o-----o====o    o----o===o-----o

It would be possible to choose a clustering algorithm, run it, and compute your preferred modularity metric for the best clustering found. But this is only a lower bound, so it doesn't seem very satisfactory.

The question matters. For example, the work of life scientists will be easier if the molecular organisation of life is modular than if it is not. It would be good to have a robust test - some of the discussion so far seems to involve wishful thinking.

My best attempt at this is:
- a tree is more modular if the edges near leaves are higher weight
- the modularity of a graph is the modularity of its min cut spanning tree
Does anyone know of an established answer to this question?


texts: Multi-task learning for Multi-label classification?

texts:
I have a multi-label classification problem wherein each example can belong to one of the pre-defined classes (or can belong to none of them).

I was wondering if I can somehow apply multi-task learning (MTL) to this problem. Essentially, treat each class as a separate class and use a NN based model with common feature extractor layer, and on top of it class specific layer.

My doubt is that generally there is a correlation between labels in a multi-label setting. In the MTL model, after feature extraction each class is handled separately without sharing information. Does a MTL kind of model for multi-label makes sense, given this information?


texts: What is the difference between LS-SVM and P-SVM?

texts: What is the difference between least-squares SVM (LS-SVM) and proximal SVM (P-SVM)? How does the decision boundary change in case of both of these types of SVM?


texts: What would be a good loss function to penalize big differences and reward small ones, but not in a linear way?

texts:
I have an image with the differences between 2 other images. Concentrations of black pixels mean similar regions between the images, whereas, white values highlight differences.

Thus I want a function to provide rewards when the pixels are the same (or relatively similar - a term to weight this "similarity threshold" would be nice) and penalize when the differences are bigger (penalizing more as the differences grow).

A differentiable function is much appreciated.

So in the context of machine learning and this loss function being a way to help train a generator, what kind of function do you recommend or can come up with?

Remember, the ideia is to reward similarities and penalize differences (such that "really different" equates to a bigger loss than "slightly off" or "different").

Thanks in advance to you all!


texts: How to aggregate data where instances occur over different time intervals

texts:
I am working on a problem in which I have several instances that have predictors that have activity over various different time periods (i.e. <3 months to well over 20 months.) Originally I attempted to use knowledge I have about this problem (it is an opportunity to sale conversion model) and learned that the average time for a deal to close is about 9 months, so I broke my predictors up into three month intervals. However, I took another look at the lengths of these deals and see that there are a variety of instances that have durations that are not even close to 9 months so this idea does not make sense.

The only idea I have gotten is just creating a duration column where I subtract the start and the stop date and then just do the summation for each predictor. However, I feel that the instances might get incorrectly labeled because some might have an overwhelmingly higher amount of activity than another due to the duration of the deal. Has anyone else encountered such a problem. Not sure if this is a common problem but a quick glance at google/reddit did not come up with anything (I could be asking the problem wrong.)

Comment


texts: What is the general approach I can use to predict whether a domain is related to a brand using a supervised learning algorithm?

texts: I have a number of domain names that may or may not be related to a particular brand.  For instance, if the brand is UPS, www.upssucks.com, www.upspackagesupplier.com, and www.ihateups.com might all be labeled as "related" because the website content is talking about UPS.  www.ilovepups.com and www.pushupssuck.com aren't related to the website UPS.  I want to use my trained dataset to create a prediction of whether a given domain is related to a brand using only the registered domain name as the input.  It seems like some off-the-shelf classifiers should work, but I am very new to this type of ML project.  What would be the first approach one would take to start making predictions?  I am planning on doing this in Python with scikit learn if that makes any difference.


texts: Cost function dependence on size - batch gradient descent


texts: how to correctly output precision, recall and f1score in keras?

texts:
I have a data set of images that I divided into Training and Testing folders, each divided into the two classes I am classifying. I use Keras generators to fit and evaluate the data. I found some resources online that I followed to implement precision, recall and f1-score metrics. Here is my Code:

class_mode = 'binary'
out_activation = 'sigmoid'
epochs = 1
mode = 'grayscale'
cat_or_bin = 'binary_crossentropy'
out_activation = 'sigmoid'
image_size = 224
batch = 128
channels = 1

def model_logistic():
    m = Sequential()
    m.add(Flatten(input_shape = (image_size, image_size, channels)))
    m.add(Dropout(0.2))
    m.add(Dense(out,activation=out_activation))
    return m

def recall_m(y_true, y_pred):
    y_true = K.ones_like(y_true)
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    all_positives = K.sum(K.round(K.clip(y_true, 0, 1)))

    recall = true_positives / (all_positives + K.epsilon())
    return recall

def precision_m(y_true, y_pred):
    y_true = K.ones_like(y_true)
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))

    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1_score(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))

train_generator = datagen.flow_from_directory(
    directory=dir,
    target_size=(image_size, image_size),
    color_mode=mode,
    batch_size=batch,
    classes = {'no_acc':0, 'acc':1},
    class_mode=class_mode,
    shuffle=True)

Test_generator = datagen.flow_from_directory(
    directory=dir_test,
    target_size=(image_size, image_size),
    color_mode=mode,
    batch_size=batch,
    classes = {'no_acc':0, 'acc':1},
    class_mode=class_mode,
    shuffle=True)

STEP_SIZE_TRAIN = train_generator.n // train_generator.batch_size
STEP_SIZE_TEST = test_generator.n // test_generator.batch_size

sgd = optimizers.sgd(learning_rate=0.0001, momentum=0.9, nesterov=True)

model.compile(loss=cat_or_bin, optimizer=sgd, metrics=['accuracy', f1_score, precision_m, recall_m])

    H = model.fit_generator(generator=train_generator,
                        steps_per_epoch=STEP_SIZE_TRAIN,
                        epochs=epochs,
                        verbose=1)

(loss,
accuracy,
f1_score, precision, recall) = model.evaluate_generator(test_generator, STEP_SIZE_TEST)

the output is the following for training and evaluation:

422/422 [==============================] - 384s 910ms/step - loss: 0.2392 - accuracy: 0.9303 - f1_score: 0.4661 - precision_m: 0.9502 - recall_m: 0.3174

2.7411930561065674
0.605730414390564
0.0
0.0
0.0

Why does it output zeroes for these metrics?

EDIT

Keras v2.3 actually now includes these metrics so I added them to my code as such:

from keras.metrics import Precision, Recall

model.compile(loss=cat_or_bin, optimizer=sgd, metrics=['accuracy', Precision(), Recall()])

However, the outputs are still zeroes for these metrics.


texts: LDA topic model has 0-weight topics, is that normal?

texts: While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. Is this an indication of an implementation mistake or is this normal and just an indication that I should use fewer topics?


texts:
I want to know what is metadata and what is meant by meta features?
When I google Meta Features what I get is feature selection tool called "Meta-Feature". What is the function of feature selection tools ?
Also, what I want is the definition and meaning of the meta features ?


texts: TypeError: 'GridSearchCV' object is not callable - how do I use a pickle of an SVM (Scikit-learn)?

texts:
I have created an SVM in Scikit-learn for classification. It works; it prints out either 1 or 0 depending on the class. I converted it to a pickle file and tried to use it, but I am receiving this error:

TypeError: 'GridSearchCV' object is not callable
(occurs during the last line of the program)

How can I overcome this?

Code:

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, GaussianNB
from sklearn import svm
from sklearn.model_selection import GridSearchCV
import numpy as np
from sklearn.externals import joblib
from joblib import load
import pickle

dataframe = pd.read_csv("emails.csv")

x = dataframe["text"]
y = dataframe["spam"]
x_train,y_train = x[0:5724],y[0:5724]

cv = CountVectorizer()
features = cv.fit_transform(x_train)

tuned_parameters = {'kernel': ['rbf','linear'], 'gamma': [1e-3, 1e-4],
                 'C': [1, 10, 100, 1000]}

model = GridSearchCV(svm.SVC(), tuned_parameters)

file = open("finalized_model.sav",'rb')
model = pickle.load(file)
file.close()

X = pd.read_csv("ExampleSingleEmail.csv")
model(cv.transform(X))


texts: Why are eigenvectors produced by np.linalg.eig different than the PCA components stored in the instance of the PCA object?

texts:
I am trying to understand why eVec (produced by np.linalg.eig) is different than pca.components_.T from the instance of the PCA class.  It was my understanding that the eigenvecters of the covariance matrix are the principal components after descending sort by eigenvalues.
An explanation in simple terms would be appreciated.
import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler

df = pd.read_csv(
'https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv'
)

df = df.drop(['model', 'vs', 'am'], axis = 1)

df = df.apply(lambda x: pd.to_numeric(x))

M = df.to_numpy()

Mnorm = M-np.mean(M, axis=0)

Mnorm = Mnorm/np.std(M, axis=0)
#  This is the normalized source data.

C = (Mnorm.T @ Mnorm) / (Mnorm.shape[0] - 1)
#  This is the Covariance Matrix without bias.

eVal1, eVec1 = np.linalg.eig(C)

eVal = eVal1[np.flip(np.argsort(eVal1))]
#  eVal is sorted according to the order of the eigenvalues.

eVec = eVec1[np.flip(np.argsort(eVal1))]
#  The same sort order as above is applied to the eigenvectors.

### From sklearn:

scaler = StandardScaler()

scaler = scaler.fit(df.to_numpy())

Anorm = scaler.transform(df.to_numpy())

pca = PCA(n_components=9)

pca_transform = pca.fit_transform(Anorm)

assert (Mnorm == Anorm).all().all()
#  This tests that Mnorm was probably constructed correctly.

assert (C.round(10) == pca.get_covariance().round(10)).all().all()
#  This indicates that the Covariance Matrix (C) was constructed correctly - the rounding is arbitrary.

assert (eVec.round(5) == pca.components_.T.round(5)).all().all()
#  However, eVec and pca.components_.T are not equal.


texts: error trying to build a histogram using r with the rStudio application

texts:
Using ggplot2 I am attempting to create a histogram.  I have a column that is full on the continents.  I need to add all the continents which I attempted to do with the aggregate function.

data <- aggregate(country$continent,country["continent"], sum)
hist(data)

Error in Summary.factor(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,  :
  ‘sum’ not meaningful for factors


texts: What is the difference between ImageNet and ImageNet1k? How to download it?

texts:
Some papers mention just ImageNet and some papers mention ImageNet 1k database?
What is the difference between these 2? Are they same or is the latter one subset of the former one?

I'm working on Generative Adversarial Nets. I wanted to train it on ImageNet Database. How to download ImageNet 1k? I went to ImageNet site and created & verified my account. Then there were several links. Which one to select?

Thanks!


texts: How to choose validation set for production environment?

texts:
I am using XGBoost for a time-series regression problem.

During development, i choose my validation set on last %10 percentage of data. Using timeseries split cross validation and grid-search, I got my best model on this with corresponding xgb hyperparameters.

My question is, how to choose validation set (for early stopping) on my production environment?

1) i have chosen last %10 percentage of my data as validation set, but this set is also included on training data. therefore overfit. very sensitive to noisy data.

2) my predicted data (lets say Y) changes over time, when i choose random rows within last year (%10 amount) and dont include them in training set, it gave me worse results on production than first option.

3) when i choose last week's data as validation data, not included in training set, it gave better result on 2. option. but i am not including last week's data to training procedure.

4) Or do i need a validation set on my production environment? should i set iteration count from the experiments done in development stage? (e.g i got best result on 10k th iteration, so i should limit my production-setup iteration count with 10k without using validation set at all?)

--

So, how can i choose validation set for my production environment? Best practices, or are there any tips/tricks for this?


texts: Choosing between TensorFlow or Theano as backend for Keras

texts: Keras supports both TensorFlow and Theano as backend: what are the pros/cons of choosing one versus the other, besides the fact that currently not all operations are implemented with the TensorFlow backend?


texts: Best Programming Language for Data Science

texts:
I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language for processing data?
P.S. I love statistics and programming so I think this will be fun.


texts: When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

texts:
Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update its value depending on its present value and the present value of the taking action from that state. Now

How long do I keep updating value of each cell? Shall I keep updating unless the change in the previous and the present value function is the least? I am not able to understand how to implement the stopping mechanism in the grid-world scenario(discount not considered)
Is the value function the values of all the grids in the grid world?


texts:
I am going through the tutorial at this site. Here, I can see the author is explaining the derivation of Gini Index. I want to understand the following terms

 Group
 Classes : As far as I have understood, it represents the possible values of labels in the data which we are supposed to classify. Please correct me if I am wrong.

The website here states that it is the difference between 1 and the probabilities of the classified values within the dataset while creating the split. But the first link does add some more points to the simple derivation. Can anyone please explain in layman terms the derivation of Gini Index?


texts: How do we get the baseline of each fold in cross-validation?

texts:
Given that I am using Scikit learn and cross-validation and want to compare my accuracy result for each fold with my baseline

I am using 10-fold cross-validation and how for each fold I return the baseline for that fold. is it possible?

clf= RandomForestClassifier(n_estimators=100, random_state=20)
# 10-Fold Cross validation
scores = cross_val_score(clf, features, labels, cv=10)
scores

Result: array([0.45454545, 0.63636364, 0.8       , 0.8       , 0.6       ,
       0.6       , 0.5       , 0.9       , 0.66666667, 0.33333333])


texts: how to find parameters used in decision tree algorithm

texts:
I use a machine learning algorithm, for example decision tree classifier similar to this:
from sklearn import tree
X = [[0, 0], [1, 1]]
Y = [0, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)
clf.predict([[2., 2.]])

How to find out what parameters are used?


texts: Choosing layer hyper-parameters of a CNN

texts:
Context: I'm building a CNN on MATLAB to classify wallpaper groups.

I'm using the following network type.

  CONV -> ReLU -> POOL -> CONV -> ReLU -> POOL -> FC -> DROPOUT -> FC -> SOFTMAX

Should the parameters for the first CONV or POOL layers be the same as the second (or later) CONV or POOL layers succeeding it? If not, is there a meaningful way to choose parameters for the repeated succeeding layers?

For example, for the first CONV layer you may pick a 5x5 filter. Perhaps for the second CONV layer it's more meaningful to pick a certain different size filter?


texts: What is the complexity of Mixture Hidden Markov Models (MHMM)?

texts:
I wonder what is the theoretical computational complexity of MHMM.
Is it related to the number of alphabets in sequence mining?


texts:
Is there a minimum batch size for training/re-fining a BERT model on custom data?
Could you name any cases where a mini batch size between 1-8 would make sense?
Would a batch size of 1 make sense at all?


texts: sklearn GradientBoostingClassifier provides unstable predictions

texts:
I am using sklearn.ensemble.GradientBoostingClassifier to build a rather simple model which predicts probabilities.
I have simplified the problem to having only one numerical predictor and the outcome (which is 0 and 1).
My training set has 10,000 samples.
I keep one of these samples to perform the following test. I calibrate the model 100 times and each time I drop 1% of the training samples so as to create training sets that differ but just a little bit. Then I use the resulting model to predict the probability of that 1 sample I left out.
Now the 100 predictions that I get vary a lot. I would expect, obviously for them to vary a little bit, but they can be up to +-10% (in absolute probability terms).
Is this to be expected?


texts: Understanding different approaches for layer widths

texts:
I am trying to figure out what would be the fundamental difference between these 3 types of layer widths assuming the same input size/output size and only changing the width of the layers and not the layer types or amount of layers.

Does anyone have some resources on this topic or past experience with this?


texts: What is the motivation for row-wise convolution and folding in Kalchbrenner et al. (2014)?

texts:
I was reading the paper by Kalchbrenner et al. titled A Convolutional Neural Network for Modelling Sentences and am struggling to understand their definition of convolutional layer.

First, let's take a step back and describe what I'd expect the 1d convolution to look like, just as defined in Yoon Kim (2014).

  sentence. A sentence of length n (padded where necessary) is represented as

  x1:n = x1 ⊕ x2 ⊕ . . . ⊕ xn, (1)

  where ⊕ is the concatenation operator. In general, let xi:i+j refer to the concatenation of words
  xi , xi+1, . . . , xi+j . A convolution operation involves a filter w ∈ R^hk, which is applied to a window of h words to produce a new feature. For
  example, a feature ci is generated from a window
  of words xi:i+h−1 by

  ci = f(w · xi:i+h−1 + b) (2).

  Here b ∈ R is a bias term and f is a non-linear function such as the hyperbolic tangent. This filter is applied to each possible window of words in the sentence {x1:h, x2:h+1, . . . , xn−h+1:n} to produce a feature map

  c = [c1, c2, . . . , cn−h+1], (3)

  with c ∈ R^(n−h+1).

Meaning a single feature detector transforms every window from the input sequence to a single number, resulting in n-h+1 activations.

Whereas in Kalchbrenner's paper, the convolution is described as follows:

  If we temporarily ignore the pooling layer, we
  may state how one computes each d-dimensional
  column a in the matrix a resulting after the convolutional and non-linear layers. Define M to be the
  matrix of diagonals:
  M = [diag(m:,1), . . . , diag(m:,m)] (5)
  where m are the weights of the d filters of the wide
  convolution. Then after the first pair of a convolutional and a non-linear layer, each column a in the
  matrix a is obtained as follows, for some index j:

  Here a is a column of first order features. Second order features are similarly obtained by applying Eq. 6 to a sequence of first order features aj , ..., aj+m0−1 with another weight matrix M0. Barring pooling, Eq. 6 represents a core aspect of the feature extraction function and has a rather general form that we return to below. Together with pooling, the feature function induces position invariance and makes the range of higher-order features variable.

As described in this question, the matrix M has dimensionalty of d x dm and the vector of concatenated w's has dimensionality dm. Thus the multiplication produces a vector of dimensionality d (for a single convolution of a single window!).

Architecture visualization from the paper seems to confirm this understanding:

The two matrices in the second layer represent two feature maps. Each feature map has dimensionality (s + m - 1) x d, and not (s + m - 1) as I would expect.

Authors refer to a "conventional" model where feature maps have only one dimension as Max-TDNN and differentiate it from their own.

As the authors point out, feature detectors in different rows are fully independent from each other until the top layer. Thus they introduce the Folding layer, which merges each pair of rows in the penultimate layer (by summation), reducing their number in half (from d to d/2).

Sorry for the prolonged introduction, here are my two main questions:

1) What is the possible motivation for this definition of convolution (as opposed to Max-TDNN or e.g. Yoon Kim's model)

2) In the Folding layer, why is it satisfying to only have dependence between pairs of corresponding rows? I don't understand the gain over no dependence at all.


texts: Any tips on transfer learning for a regression problem using 4D images as input?

texts:
I developed a CNN based on EfficientNet in order to predict the weight of piles of some materials in an image (the labels are the weights in kg and the input is RGBD tensors of the object).
I have two different materials (m1 and m2) which are similar but can be classified easily (you can see the difference between them).
The specific weight of each material is different (m1 is 3 times heavier than m2).
I trained my network using data of m1 only and got pretty good results: MAE of 0.12 kg per pile (who weights 0-6 kg). I used 30K tensors (24k for train and validation and 6k for test).
Now I want to take the trained network and transfer it to predict the weights of m2.
The task is the same and the images are similar (were taken in the same environment) so the data is similar.
I was wondering what would be the best way of transferring this model (and weights learnt by the model trained on m1) to predict the weights of the other material (m2).
So my question is:
What will be the best way to transfer the model trained using data of m1 to predict data of m2 (which is similar but the specific weight is much lower)?
Should I use the weights learnt using m1 as initializers and retrain the whole model to predict m2? should I freeze most of the layers and retrain the last few layers only?
How much tensors of m2 should I use? My m2 dataset also includes 30K images, should I use all of them or just a part of them?
Do you have any ideas I didn't mention?
Thanks in advance!!


texts: Is there a way to standardize latitude and longitude to be used as predictors in KNN algorithm?

texts: The predictors are latitude and longitude, and target variable is region.


texts: how should i check stability of community detection algorithm?

texts: I have a community detection algorithm that communities can't have overlap. It works based on density of edges. Now I want to see how stable this algorithm is (if I decrease internal edges and increase edges between communities). Is there a standard way to do it?


texts: Beginner math books for Machine Learning

texts:
I'm a Computer Science engineer with no background in statistics or advanced math.

I'm studying the book Python Machine Learning by Raschka and Mirjalili, but when I tried to understand the math of the Machine Learning, I wasn't able to understand the great book that a friend suggest me The Elements of Statistical Learning.

Do you know any easier statistics and math books for Machine Learning? If you don't, how should I move?


texts: What are the introductory mathematics courses that are most pertinent to machine learning?

texts:
To provide more context to this question, I have found the following list of courses in order to learn calculus:

Course | 18.01.1x | edX

Course | 18.01.2x | edX

Course | 18.01.3x | edX

I also have found the following specialisation on coursera:

Mathematics for Machine Learning | Coursera

I am wondering which of the two sets of courses is better suited to machine learning. I have already started the ones on edx but I am unsure as to whether the ones on edx are suitable as an introduction to calculus for machine learning.


texts: Why are RNNs used in some computer vision problems?

texts:
I am learning computer vision. When I was going through implementations of various computer vision projects, some OCR problems used GRU or LSTM, while some did not. I understand that RNNs are used only in problems where input data is a sequence, like audio or text.
So, in kernels of MNIST on kaggle almost no kernel has used RNNs and almost every repository for OCR on IAM dataset on GitHub has used GRU or LSTMs. Intuitively, written text in an image is a sequence, so RNNs were used. But, so is the written text in MNIST data. So, when exactly is it that RNNs(or GRUs or LSTMs) need to be used in computer vision and when don't?


texts: Neural network for Multiple integer output

texts:
I have a data set that contains 135 input features and 132 output values to be predicted. The input features are all numeric floating point values and each output value would be an integer between [0,1,2,3,4].

I am basically new to data science and machine learning, therefore I need to understand what kind of neural network model (regression or classification) would fit best for this kind of data. On one hand the output values represent different classes but I am not sure how a single neural network can predict multiple classes from the same input data.


texts: Cross validation Vs. Train Validate Test

texts:
I have a doubt regarding the cross validation approach and train-validation-test approach.

I was told that I can split a dataset into 3 parts:

Train: we train the model.
Validation: we validate and adjust model parameters.
Test: never seen before data. We get an unbiased final estimate.

So far, we have split into three subsets.  Until here everything is okay. Attached is a picture:

Then I came across the K-fold cross validation approach and what I don’t understand is how I can relate the Test subset from the above approach. Meaning, in 5-fold cross validation we split the data into 5 and in each iteration the non-validation subset is used as the train subset and the validation is used as test set. But, in terms of the above mentioned example, where is the validation part in k-fold cross validation? We either have validation or test subset.

When I refer myself to train/validation/test, that “test” is the scoring:

Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, it returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.

As an example, I found the Sci-Kit learn cross validation version as you can see in the following picture:

When doing the splitting, you can see that the algorithm that they give you, only takes care of the training part of the original dataset. So, in the end, we are not able to perform the Final evaluation process as you can see in the attached picture.

Thank you!

scikitpage


texts: How can I generate a handwritten notes given any handwriting sample and text file?

texts:
I am new to ML/DL and looking for a good way to generate a handwritten (simulated) file given 2 inputs:

A set of sample handwritten notes (for training). All notes will be of same person
A text file

Output
Text file in handwritten  form, making use of sample set
Any pointers will be good. I looked at this but it differs from my requirement in that, it does not generate for a given sample set but an arbitrary form


texts:
I am out of ideas in regards to plotting different sets of data ....

I have 10 sets of data ... Each set contains Actual Speed of vehicle and predicted Speed of Vehicle..

How can i combine all sets of data and show in a meaningful manner ?  I don't think I can simply average as they are absolute values (In one run , i was going 25mph and second run , i might be going 30 mph)//...

Is there any idea on how i can represent 10 sets of data in a meaningful manner like variance  ?

Any ideas are appreciated


texts: How to do Feature Extraction using Apache Spark

texts:
I am a newbie to Machine Learning and I have to do feature extraction for a banking application to detect / predict fraudulent transactions.

I have come across few articles on Feature Extraction using Scipy / Sci-kit learn in Python but I want to know is there any way to do the same in Apache Spark using some packages or modules within Spark

Any ideas that enable me to use Pyspark is also welcomed here

Appreciating your thoughts on this. Thanks in advance


texts:
If you have n time series of rainfall measurements every hour (x=time, y=amount of rain), and compute the distance matrix between each pair of time series based on Dynamic Time Warping, and then plot the clusters, what do the X and Y axes on the cluster plot represent?

I have figured out that the values in the distance matrix would have the same measurement unit as the amount of rain in the time series (mm or inches of rain).

In the k-medoids or k-means plot, is X still time, and Y still the amount of rain?


texts: Why Epochs take longer as learning proceeds?

texts:
I am training a deep learning model in Tensorflow on GPU (Amazon AWS) and what I observe is that in the beginning each epoch takes only less than a second but let's say after 80 epochs one epoch takes more than 8 minutes.

What could be a possible reason?


texts: I'm new in Machine learning and I'm working on a problem related to text. I know that in ML we can use features as numerical values as input to neural network, but I don't know how to use features as words. In some papers I read that we take features to be n words with some property. I really don't understand how is that possible. Please, if it is not a problem, just to tell me some good papers or textbooks or links where it is explained how to do that.


texts:
I have 2 ranking lists and I want to see how similar are they between them.
setA = {10,9,8 ,7 ,6,5,4,3,2,1}
setB = {4,5,12,14,9,8,7,6,2,1}
There are the following constraints:

1. High number (or if reverse, low) are more important than low numbers.
2. Range orders don't match In setA there are numbers not contained in setB (10,3) and (14,12). So inner join length is not len(setA) or len(setB).

I have been reading about NDCG, Kendall Tau and Spearman, but they don't seem to be adequate metrics for 1 and 2 constraints at the same time.
Q1: Am I missing something with the aforementioned metrics?
Q2: Are there more suitable metrics for this problem?


texts: Deep network not able to learn imbalanced data beyond the dominant class

texts:
I have data with 5 output classes. The training data has the following no of samples for these 5 classes:
[706326,  32211,   2856,   3050,    901]

I am using the following keras (tf.keras) code:

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(50, input_shape=(dataX.shape[1],)),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(5, activation=tf.nn.softmax) ])
     adam = tf.keras.optimizers.Adam(lr=0.5)

model.compile(optimizer=adam,
              loss='sparse_categorical_crossentropy',
              metrics=[metrics.sparse_categorical_accuracy])
     model.fit(X_train,y_train, epochs=5, batch_size=32, class_weight=class_weights)

y_pred = np.argmax(model.predict(X_test), axis=1)

The first line on class_weight is taken from one of the answers in to this question: How to set class weights for imbalanced classes in Keras?

I know about this answer: Multi-class neural net always predicting 1 class after optimization . The difference is that in that case, the class weights wasn't used whereas I am using it.

I am using sparse_categorical_crossentropy which accepts categories as integers (don't need to convert them to one-hot encoding), but I also tried categorical_crossentropy and still the same problem.

I have of course tried different learning rate, batch_size, no of epochs, optimizer, and depth/length of the network. But it always is stuck at ~0.94 accuracy which is essentially I would get if I predict the first class all the time.

Not sure what is missing here. Any error? Or should I use some other specialized deep network?


texts:
I am doing a research regarding on automatic text summarizing. So in order to weighting sentences I need to get words related to a particular field or domain like shown below.

Topic word - Car
Related words - engine, driver, road, break, accelerator

Is there any direct method that I can use like wordnet synsets.


texts: Combining Aggregated Dependent Data with Granular Independent Data

texts:
As data science newbie, I thought I was on the right tract until something doesn't seem right after plotting. A simplified structure of my dataset is as follows:

FIPS (Census Tract ID)
Drug-related death rate per 100K by county (dependent variable)
Demographic A count by Census Tract
Demographic B count by Census Tract
...

In other words, as drug-related death data is only available by county, my data consists of an aggregated dependent variable and many other granular independent variables. My questions are:

Is there a better option to prepare my data? One option I can think of is to aggregate (or average) Census Tract data to county level so there is an one-to-one, instead of one-to-many relationship. However, I feel that this approach may lose the benefit of identifying key population metrics at individual Census Tract level.
Should I transform/normalize the demographic data from counts to per 100K like my dependent variable?

Thanks in advance for sharing your wisdom!


texts: Customer Intelligence - How to measure success?

texts:
we are creating models that aim to filter new leads from our current customer base.

We started to create propensity models that calculate a percentage for each customer for a certain product group. I was wondering what are best practices to benchmark such a model and to show the success of a propensity model in advance.

Thx for your replies!


texts: With EM algorithm, can you infer the location and variance of each "peak" in a pdf? Gaussian Mixture Models?

texts:
When I plot my data into bins, there is a frequency of data points per bin, which I can plot with a histogram. Based on this probability density function, I would like to find the maximum likelihood estimation of various parameters associated with this pdf.

How can I infer the number of "peaks" of this pdf? Is this possible using a Gaussian Mixture Model (let's say using Expectation Maximization)?

I believe in this case, we are inferring the location and variance of each "peak" using Gaussian clusters.

Secondly, how does one infer the number of clusters which model the data best?


texts: Different training method for encoder-decoder model

texts:
Trying to learn the encoder-decoder model for some NLP problems.
I am referring to this Keras tutorial.
During the model training phase, this tutorial just uses the following:
model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
          batch_size=batch_size,
          epochs=epochs)

I understand this logic. But the confusion is in some other tutorials for EXACTLY THE SAME PROBLEM. For example, in Tensorflow's documentation for NMT with Attention the training_phase is very different where they use custom training loops with a custom train step and calling the step for every batch manually.
The question is are these 2 different training methods which should be used in particular cases OR its the same training method with 2 different forms of implementation?


texts:
When I use the hyperopt library to tune my Random Forest classifier, I get the following results:

Hyperopt estimated optimum {'max_depth': 10.0, 'n_estimators': 300.0}

However, when I train the model using its default hyperparameters, all of the evaluation metrics (Precision, Recall, F1, iba, AUC) return higher values compared to the tuned model. Should I still follow the tuned parameters? Or ignore the results of the tuning process, as it is not helping to improve the results?


texts: Predicting future of power consumption in repeating manufacturing process

texts:
I have this situation. We are tracking the power consumption of an industrial machine and by looking at the power consumption (in watt) we're trying to predict whenever something will break resulting in the machine needing maintenance. The machine has 1 specific manufacturing proces where the powerusage will spike up for certain periods and come down again.

I personally am very new to machine learning and don't quite know how I should structure the dataset. I have seen companies split it per component in the machine or per fase in the production process.
Currently I have this structure in mind.

Time(Unix)|Kilowatt|ProductionFase

I'm not sure if UNIX would be the right time unit or if I should just start counting up from 1 whenever the machine starts running. I'm also unsure about the productionfase part of the algorithm, since the new input data will only include time and kilowatt. Do you guys think I should switch to having it per divided per component, or just leave both options out and only look at time and kilowatt?

Other than that the labels I have in mind are LOW|NORMAL|HIGH for the powerconsumption of the machine.

Could someone verify some of my ideas or give me some tips to go in the right direction?
Thanks:)


texts: what's the difference between autoencoder and autoassociative neural networks?

texts:
Seems like Autoassociative neural networks and Autoencoders do the same thing.
Are they the same, or is there some subtlety I'm not picking up on?


texts: How can the Adaboost technique be called an ensemble learning technique?

texts: I have read that in ensemble learning we use the outputs of various classifiers to make the predictive modeling better but in Adaboost we just use one classifier and we make it a strong learner but how it is a part of ensemble learning.


texts: Suggestions on how to explain 'models' & 'algorithms'

texts:
I guess other members of this Stack have ran in to this before, but I may be wrong: Have you ever been approached and asked to explain the difference between models and algorithms? This happened to be recently and, while I feel that I explained it well, there is a colleague whom disagrees with my assessment..

How I see the two:

Algorithms: Used to train a model, that is, give it instructions and process the input.

Model: A diagram of sorts, that incorporates utilizing algorithms to train inputs. Can be reused on similar data.

Pre-Trained Model: A model, usually trained on larger data sets and can be utilized to build [your] own models with other data.

Thoughts? I'd appreciate them to see if I am just completely off base here.


texts: How important is advanced SQL for data science?

texts:
Is advanced level SQL required to be competitive as a data scientist? Is it more important for a data analyst to be good in SQL? Is it enough to be able to extract data using simple SQL queries?

I know it is faster to manipulate data in SQL than to copy data into R or Python, but are there any other advantages or disadvantages?


texts: Gaussian Discriminant Analysis (GDA) package in R

texts:
This stack exchange post - https://stats.stackexchange.com/questions/80507/what-is-a-gaussian-discriminant-analysis-gda - discusses GDA, a machine learning method for classification. I would like to implement something like this analysis in R.

I've seen posts for discriminant analysis in R using linear and quadratic discriminant analysis (http://rstudio-pubs-static.s3.amazonaws.com/35817_2552e05f1d4e4db8ba87b334101a43da.html) but nothing for GDA?

I guess my questions are, because I'm not 100% familiar with these methods, (1) whether GDA is different from or a subset of LDA and QDA, and if there is an R package / functions for GDA in R like there are for LDA and QDA (the lda() and qda() functions in the MASS package). Maybe I should be asking if I actually want to be performing GDA with my data (I'm not actually sure if the joint distribution of the features follow a gaussian dist), or if lda or qda may be better?

Thanks!


texts: Does anyone have any ideas about what would be a good way to go about building a user Life Time Value model? I have a website with two kinds of revenue, pay per click and impression from advertisments. My initial instincts are telling me I will need to use a unsupervised clustering algorithm to segment individual users by some kind of rough LTV metric. and then maybe re define a revenue distribution for each cluster. Anyway, any helpful ideas would be appreciated.


texts: Using vgg16 or inception with wights equals to None

texts:
When using pre-trained models like vgg16 or inception,
it seems that one of the benfits of using pre-trained model, is to save up time of training.
Is there a reason to use the pre-trained models without loading the weights ? (use random weights) ?


texts: Image Recognition Model with Adaptive Learning

texts:
Problem Statement: To be able to detect faces of specific people with good accuracy and tag those images with the names of the people it contains.

To do this we have about 40 images person, and 10 classes.

I have been using Facenet model ( https://github.com/davidsandberg/facenet ) and I am getting good results after training the model with 10 classes & 40 images per class. I have used 100 images for testing.

The current hurdle im facing is : Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.

Is it possible to have an adaptive learning added to my current solution?
Not sure if I made my problem clear , I could explain more in detail if needed.


texts: Should softmax cross entropy with logits always be zero if logits and labels are identical?

texts:
Apologies for this basic question. I don't have a deep understanding of this function but I was pretty sure it would be zero if logits and label inputs are identical. I found out my MLP model solves my classification problem pretty well but the cost function is never actually 0 (rather it approaches something like -2.14442e+07 but never is actually 0). I have tested what happens if set logits and labels to the same input and still its the same. I use the tensorflow tf.nn.softmax_cross_entropy_with_logits as cost function.
Thank you for your help.


texts: Comparison of machine learning approaches for a topic in a scientific paper

texts:
As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet.
The topic is a regression problem for which several machine learning approaches have been tested:
SVR, RandomForestRegressor, GradientBoosting, DecisionTrees, Artificial Neural Networks, LSTMs, K-NearestNeighbors..

It has been shown that useful results can be achieved with all approaches. Which of the methods is the best depends heavily on the parameter configuration.

It would be too much work to test any parameter configuration for any approach. Besides, I think it would be uninteresting to show dozens of plots about the parameter tuning tests, because they also depend heavily on the respective data sets.

So how do you typically compare different types of machine learning approaches in scientific papers that are heavily dependent on the parameters? Just play a bit with grid search and randomized search until you are satisfied with the parameter configuration?


texts: How to train a simple Machine learning model in batches?

texts:
I have a dataset for multiclass classification of text data. The number of samples in training data are 1,20,000. If I extract features using TF-IDF vectoriser of the sklearn library it gives about 80,000 words as features. So, when I train a sklearn model for classification using Multinomial Naive Bayes on the Kaggle platform it exceeds the RAM limit. I now want to train the model using small sample batches of 20,000 samples and combine them. How can I do this? I want that instead of training using the whole dataset I use only a small chunk of the dataset. Also, I want to use ML model and not Deep learning or Neural Network model. I want to use Multinomial Naive Bayes as classification algorithm.

Note- I have preprocessed the data to remove stopwords, punctuations and lemmatized it before using feature extraction method.


texts: Website for Datasets - Miles and Shevlin book, "Applying Regression & Correlation"

texts: I've very much enjoyed Miles and Shevlin (2015) book, "Applying regression & correlation : a guide for students and researchers." The book mentions availability of datasets, but the website provided in the book no longer exists (http://jeremymiles.co.uk/regressionbook). Is there another website that I can find the aforementioned datasets?


texts: What principle is behind semantic segmenation with CNNs?

texts:
I have been learning about segmentation models recently and was reading this paper today: Dense Transformer Networks. On page 2, it describes segmentation as:

  Given the success of deep learning methods on image-related applications, numerous recent attempts
  have been made to solve dense prediction problems using CNNs. A central idea of these methods is
  to extract a square patch centered on each pixel and apply CNNs on each of them to compute the label
  of the center pixel. The efficiency of these approaches can be improved by using fully convolutional
  or encoder-decoder networks. Specifically, fully convolutional networks [23] replace fully connected
  layers with convolutional layers, thereby allowing inputs of arbitrary size during both training and
  test. In contrast, deconvolution networks [25] employ an encoder-decoder architecture. The encoder
  path extracts high-level representations using convolutional and pooling layers. The decoder path
  uses deconvolutional and up-pooling layers to recovering the original spatial resolution. In order to
  transmit information directly from encoder to decoder, the U-Net [28] adds skip connections [14]
  between the corresponding encoder and decoder layers. A common property of all these methods is
  that the label of any pixel is determined by a regular (usually square) patch centered on that pixel.

So, I know that the models used take a single image as an input and output a segmentation mask for the entire image as an output. Are the authors here writing that the underlying operation being performed by the models is to extract patches and classify them, while this is all done in parallel? I would assume these patches correspond to the receptive field of the network, if that is what they are saying.

My overall question is just hoping for an explanation of how this statement is actually what is being performed in a segmentation model.


texts: Error term in probabilistic interpretation of least squares update rule

texts:
I have read in Stanford's CS229 course notes that to justify the least-squares update rule with probability, the following is assumed:
$$y^{(i)} = \theta^Tx^{(i)}+\epsilon^{(i)}$$
, where $\epsilon^{(i)}$ represents random noise that is distributed i.i.d. w.r.t the Normal distribution.
I understand why $\epsilon^{(i)}$ would make sense when $h(\theta)=\theta^{T}x^{(i)}$ is a trained model, but since this assumption's eventual goal is to derive the update rule, it should make sense also when $h(\theta)$ is not trained yet. However, this assumption does not make too much sense to me when the model is arbitrary and not trained at all. Is my interpretation correct? Have I missed something? If not, how do we justify $\epsilon^{(i)}$ when the model is inaccurate (not trained)?
Thanks in advance.


texts:
I'm looking at identifying trends within my data, particularly wondering if usage of my app on new sign ups is increasing over a week to week basis.  As we are constantly improving our product, I'd love to be able to identify a correlation between app usage & new feature releases.

The data I have is split into weekly cohorts.  Each week I have the number of new sign ups, and how many of those accounts are still active (e.g. users are logging into the system).

Let's say I have the following data

Week 1 - 10 new trials, 3 accounts active

Week 2 - 15 new trials, 5 accounts active

week 3 - 4 new trials, 3 accounts active

week 4 - 20 new trials, 12 accounts active

week 5 - 17 new trials, 9 accounts active

In my current approach of analysis, week 3 looks amazing because 75% of accounts are still active ... In reality though, the number of new trials is extremely small compared to other weeks.  As a result, I don't feel like I'm accurately comparing apples to apples in a week to week comparison.

Is there a way I can normalize the weekly data so that I am performing accurate analysis - or is percentage based really the best way to look at this data?

I am pretty new to this, so any help is much appreciated.

Cheers


texts: PCA - Error minimization and Variance Maximization

texts:
I'm studying the PCA algorithm and the theory behind it. I think I understood how does it work and the idea of dimension reduction of the data in order to find a new feature (component) that maximizes the variance of the data and minimizes the error.

My question is : in this algorithm , are the maximum variance and the minimum error reached at the same moment ?

In this example the magenta/black line is the solution of my PCA. So I find the 1 dimension vector that reduces my 2 dimensions dataset. I found this vector because the error (length of the red lines) is minimized and the variance (distance among the red projected points) is maximized.

So if I need to apply this algorithm , if I use either only the avg error minimization , will I reach the same result of the case in which I use the variance maximization principle ?

Thanks.


texts:
I am working since some months on a prediction from lead to a sale. Someone makes a lead on my website and I want to predict if this user will make a sale. I have these metrics on the test data.

Now when I check for example for the month May_2020 all the leads I had predicted as 1 (probability >=0.5) and compare with the real sales I receive a conversion rate of 22%. That means, only 22% of all the predicted with 1 lead becomes sale. What do you think?

Is this caused because of low precision? The precision is similar to the conversion rate or is this only luck or the precision implicates the conversion rate.
Any idea of how to reduce this big error of 78%?


texts: Correct way to use k-means cluster- python-scikit-learn

texts:
I am working with Python, pandas and scikit-learn. I have a dataset where I have the number of sales per shop. I have the date and the hour of this sales and I want to forecast the sales for the next month.

I have tried different approaches trying to forecast per individual shops, using all the data and different things but with a high error using ARIMA models.

Then I realised that may be if I train a k-means cluster I can automatically group the shops that are similar between them and then train an ARIMA model to predict per each groups that my cluster has assing.

The way I have done that is why I am posting this question, because I don't know if I have though it in a correct way.

I have grouped per shop the sales and then I have calculated the average and the standard deviation per each shop. I have trained a k-means clustering with 4 clusters, to group the shops that has similar average and standard deviation.

Do you think that is correct this procedure of grouping each shop calculate the the average and standard deviation per each one and group them in 4 clusters?. The reason for 4 clusters is for  group the shops that has similar average and similar standard deviation like divide the data in quartiles.


texts: Layer notation for feed forward neural networks

texts:
Apologies in advance, for I have a fairly rudimentary question on the notations for studying Feed-Forward Neural Networks. Here is a nice schematic taken from this blog-post.

Here $x_i = f_i(W_i \cdot x_{i-1})$ where $f_i$ is the activation function. Let us denote the number of nodes in the $i^{\text{th}}$ layer by $n_i$ and each example of the training set being $d-$dimensional (i.e., having $d$ features).

Which of the following do the nodes in the above graph represent?

Each one of the $d$ features in every example in the training set. In this case, $n_0 = d$ and $x_0$ is $(d \times 1)$.
Each example of the training set, which is $d-$dimensional. In this case, $n_0$ is the number of examples and $x_0$ is $(d \times n_0)$.

In both cases, the weight matrix $W_i$ is $(n_i \times n_{i-1})$.

On the one hand, most references like this blog-post claim it is (1), while on the other, I can also find few references such as this video which seem to claim it is (2). Which one of them carries the right interpretation?

Although seems like the back propagation algorithm can be executed in both representations, I'm quite sure it only makes sense in one of them. Any help will be greatly appreciated.


texts: Should Discriminator Loss increase or decrease?

texts:
This question is purely based on the theoretical aspect of GANs.
So, when training a GAN how should the discriminator loss look like?

Should the loss of discriminator increase (as the generator is successfully fooled discriminator)

Or should the loss of discriminator decrease?

Can someone please help me in understanding this?


texts: Voice recognition with fourier transformation with audio input in python

texts:
First of all is using the fourier transformation even a good method for recognizing different speakers? I'm not sure if it could recognize a voice if the things that are said are different. I know google and amazon have features of voice/speaker recognition in their voice assistants but what would be a good way to make that too if the fourier transformation doesn't work out?

I want to recognize voices using a neural network, to do that I need to first get a good input for the neural network but by just giving the sound recording as input I don't think it would work because it is based on frequency and time. So I found the Fourier transformation and now I'm trying to transform my audio file with Fourier and plot it.

My questions are:

How can I plot a Fourier transformation with audio input in python?
And if that is working, how can I input the Fourier transformation in the neural network (I thought perhaps give every neuron a y value with the neurons as the corresponding x value)

I tried something like (a combination of things I found on the internet):

import matplotlib.pyplot as plt
from scipy.io import wavfile as wav
from scipy.fftpack import fft
import numpy as np
import wave
import sys

spf = wave.open('AAA.wav','r')

#Extract Raw Audio from Wav File
signal = spf.readframes(-1)
signal = np.fromstring(signal, 'Int16')
fs = spf.getframerate()
fft_out = fft(signal)

Time=np.linspace(0, len(signal)/fs, num=len(signal))

plt.figure(1)
plt.title('Signal Wave...')
plt.plot(Time,fft_out)
plt.show()

but considering my input in the mic was 'aaaaaa' it does not seem right.


texts: Let's say we have two models: one is a random forest, and the other one is a xgboost model. Those two models yield similar accuracies but in the PDP plots, one model says features A, B, C have a relatively strong positive impact on the label, whereas the other model says the exact opposite (strong negative impact). In that case, what should we do? Should I be concerned and find out which model is more sane?


texts:
What is the exact definition, or at least the main idea, of  weak and strong learners? For instance, these terms are used in propositions such as "Boosting is based on weak learners".

Regarding the viewpoint that weak learners are of the type (high bias, low variance) and strong learners are (low bias,high variance). How are these terms defined without respect to specific datasets or data distributions?

For instance, is it possible to find specific datasets for which a so-called weak learner (low bias) is actually a strong learner (low variance)?
More simply, can the learning method X be a weak learner on data distribution A but a strong learner for data distribution B?


texts: Classification Dataset which is linearly non separable

texts:
I am trying to find a dataset which is linearly non-separable. I checked the Iris dataset and the UCI website says:

  The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable
  from the other 2; the latter are NOT linearly separable from each
  other

Source: https://archive.ics.uci.edu/ml/datasets/iris)

However, I need a two class dataset which is linearly non-separable.

Please give some pointers.


texts:
I am new to NLP and have a dataset that has a bunch of (social media) messages on which I would like to try some methods like latent Dirichlet allocation (LDA). First, I need to clean the data of things like punctuation, emojis, etc. I'm not sure how to go about doing this in the most efficient and accurate manner. My code right now is this:
import pandas as pd
import re

class TopicModel():
    def __init__(self, data_path = "data.csv"):
        self.data_path = data_path
        self.data = pd.read_csv(self.data_path, low_memory=False)

    def clean_data(self):
        self.remove_message_na()
        self.remove_emojis()
        self.remove_punctuation_and_lower()
        self.remove_url()
        self.remove_empty_messages()

    def remove_message_na(self):
        self.data = self.data.loc[~pd.isna(self.data['message'])]

    def remove_emojis(self):
        self.data['message'] = self.data['message'].str.encode("ascii", "ignore").str.decode("utf8")

    def remove_punctuation_and_lower(self):
        p = re.compile('''[!#?,.:";]''')
        self.data['cleaned_data'] = [p.sub("", ii).lower() for ii in self.data['message'].tolist()]

    def remove_empty_messages(self):
        self.data = self.data.loc[self.data['cleaned_data'] != ""]

    def remove_url(self):
        self.data = [re.sub(r"http\S+", "", ii) for ii in self.data['message'].tolist()]

I don't want to remove contractions, which is why I left out ' from my punctuation list, but I think optimally, the contractions would be reformatted as two separate words. I also wonder about the other punctuation marks when dealing with social media data, e.g., #. I know this question is a bit general, but I'm wondering if there is a good python library for performing the kind of data-cleaning operations that I want, prior to perform topic analysis, sentiment analysis, etc. I'd also like to know which libraries can efficiently perform these data-cleaning operations on a pandas data frame.


texts: Which phrase should be returned in case of multiple matches when comparing text?

texts:
I want to compare one sentence to some other sentences using the Bag of Words model. Suppose that my comparing sentence is:
I am playing football
and there are three more sentences that I want to compare my comparing sentence with. They are:
1. and I am playing Cricket
2. Why do you play Cricket
3. I love playing Cricket when I am at school
Now, if I compare my comparing sentence to the above three sentences by counting words, the number 1 and number 2 sentences have the same number of words that the comparing sentence has. and that is 3 (I, am , playing).
Now the question is, Which sentence is more related to my comparing sentence in this case? there are no semantic meanings involved at all.
In some places I saw, they say, it is less convoluted to return the shortest sentence in this case. What are your thoughts?


texts:
I'm pretty new the Tensorflow and Keras and I've started to build a simple GAN to analyze and then synthesize some time-series data. I would like to ask for some feedback to confirm my suspicions about a rather common TypeError error I'm getting.
Since there is no corresponding Conv1DTranspose to the other Conv1D,Conv2D,Conv3D etc. according to the documentation (tho it appears to be available in the nightly builds), I have had to make my own and stick it in a Lambda layer. Making custom layers looks quite involved so I'm assuming perhaps that's where I went awry. At least this project is not following my experience with earlier TF tutorials and setting the training flag. The traceback leads back to the TensorFlow 2.0.0 module base_layer.py so now I'm not sure if my custom layer is the cause or not.
Is my problem with the custom conv1d_transpose layer in the generator? Is the consensus I just download and work with the nightly build instead? Or does the error lay in another layer? (For example Dense doesn't have a training flay per the docs.)
def conv1d_transpose(x):
    return tf.nn.conv1d_transpose(x, filters=100, output_shape=input_dim, strides=(4), padding="SAME")

def build_generator():
    generator_input = Input(shape=(input_dim), name='generator_input')

    x = generator_input

    x = Dense(100)(x)

    x = Lambda(conv1d_transpose, name='conv1d_transpose')(x)

    x = BatchNormalization()(x)

    x = Dropout(rate = 0.5)(x)

    x = Activation('sigmoid')(x)

    generator_output = x

    return Model(generator_input, generator_output)

g(noise, training=True)

#Returns
`TypeError: call() got an unexpected keyword argument 'training'`


texts: libraries for multiple machine NN training?

texts:
As detailed here, the way to go to break NN training over multiple machines/threads, is decompose training data set on multiple chunks and send to each node, then sum results back in main node.
There is some library who already implements these techniques? Agents to install on each node?


texts:
Could Someone list the pros and cons with respect to using federated learning with the following packages:

TensorFlow federated
PySyft

Are there certain tasks which are specific to either or is one clearly better than the other?
Is there any other module which is better than these? If so could you please link them below.


texts: Typing error handling n-gram character index and vector space model

texts: Suppose I apply tri-gram indexing for my document collection, and is implementing a vector-space model to help retrieving the document. In the text it is mentioned implementing a trigram will introduce a new step in filtering the result. However, what are the problems that I need to be aware of if I implement tfidf/vector-space model? The reason I am exploring this option is to try handling basic spelling error handling, does it really work in practice?


texts:
I am looking for a machine-learning (classification) model that can adapt to new data. By that, I don’t mean completely new samples in the online-learning sense, but rather additional details about an existing sample. Over time, for an existing sample, I learn an additional feature value that was missing before. (Once it is learned it doesn't change.) When I run an updated sample through the model, I would like to get a "more accurate" prediction.

The time and order at which new data becomes available are not consistent and features can still be missing at the end. In my training data, I only know the data that is available at the end and not when which feature value became available.

Most of the features are categorical and I am wondering if it would be enough to train a model by creating many synthetic samples where I randomly replace some of the categories with a “missing category” value. It seems to me though that this approach would not make sure that additional data would lead to a more confident prediction. The model would just see it as a new and different sample. How can I make sure that an update leads to a better prediction? Any hints are appreciated.


texts: Speed of training decrease by adding more GPUs

texts: I am using the distributed Tensorflow with Mirror Strategy. I am training the VGG16 based on custom Estimator. However, by increasing the number of GPUs time of training is increased. As I check, the GPUs Utilization is about 100% and it seems the input function can feed data to GPUs. As all GPUs are in a single machine, Is there any clue to found out the problem. This is the computation graph and I am wondreing the Groups_Deps cause the problem.


texts: NLP: To remove verb and find the match in a sentence

texts:
Is there a NLP method like stemming, lemmatisation to figure out the below?

1=2
3=4

1) provide technical documentation
2) technical documentation
3) use software design patterns
4) software design patterns


texts: How to represent and work with the feature matrix for graph convolutional network (GCN) if the number of features for each node is different?

texts:
I have a question regarding features representation for graph convolutional neural network.

For my case, all nodes have a different number of features, and for now, I don't really understand how should I work with these constraints. I can not just reduce the number of features or add meaningless features in order to make the number of features on each node the same - because it will add to much extra noise to the network.

Are there any ways to solve this problem? How should I construct the feature matrix?

I'll appreciate any help and if you have any links to papers that solve this problem.


texts: How to prepare the training data for SVD-based Recommendation?

texts:
I am trying to build an SVD-based recommender system. According to my understanding, the training data should only contain the users who buy at least m items and the items which are bought by n unique users. Otherwise, if I use all users and items (including low-frequency users and items), I think the training data will be noisy.

However, here is the problem: To build the training data for SVD, I first retrieve all users who buy at least m items from the database. But there are some items bought by these users are not bought by at least n users. But after further filtering out these low-frequency items from the training data, some of the remaining users will not have item sets which contain m items because some items are deleted.

I feel that I am not on the right track. How am I supposed to build the training data for SVD?


texts:
Having working on data sets, sometimes we want to keep track of mtiple models with different architectures which work on the same data set on which we have made some transformations and preprecessing of data has been done.
So I would like to know what is the elegant way to work on multiple models which use the same data set?
Because having multiple models on the same notebook is cumborsome and recreating the same data preprocessing nd transformations on separate notebooks is also a lot of copy-paste which I think can be solved with some existing solutions which I am not aware of.
What is the industry standard when doing such tasks?
Any help is appreciated.


texts: How to drop input channels or neurons at the inference phase

texts:
I understand that dropout is used to drop some weights during training, is there a way one can conventionally train a neural network with full weights and then drop some weights at the inference?

My understanding is that during testing, for instance using MNIST data-sets, the test accuracy is computed as:

test_accuracy = accuracy.eval ({X: mnist.test.images, Y: mnist.test.labels})

How can I modify the code so that some weights are set to zero?


texts:
I deal with a cars data, but the data is non representative, there is a huge number of some cars and small number of other car as follows:
]1
I am using train_test_split to split the data before training, does StratifiedShuffleSplit can increase the performance?
If yes, how can I use it with categorical data?


texts: Is boosting resistant to overfitting for both number of iterations and number of features?

texts:
Boosting methods (such as the popular xgboost) do not tend to overfit when we use many iterations - Schapire and Freund. Are they also resistant to overfitting when we feed them with a large number of features (where some of the features are not very useful?) If so, is there a theoretic connection between these two desirable properties?

(related to
https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality)