texts: how to set gridsearchCV to ignore the errors

texts:
I want to use GridSearchCV in Python for my Logistic Regression model, and i want it to check combinations for every possible setting, but i get an error when there is time for penalty: l1 and solver lbfgs because this solver type is not supported for this penalty. My question is: is there a way to say to the gridserach if some combination isn't posible just go to the next one.
parameters = {
        'penalty'   :   ['l1','l2','elasticnet','none'],
        'solver'    :   ['lbfgs','liblinear','sag','saga']
             }

clf_A = LogisticRegression(random_state=420)

f1_scorer = make_scorer(f1_score,pos_label=1)

grid_obj = GridSearchCV(clf_A,
                        scoring=f1_scorer,
                        param_grid=parameters,
                        cv=5)

grid_obj = grid_obj.fit(X_train,y_train)

clf_A = grid_obj.best_estimator_


texts: How Mllib in Spark select variables in logistic regression

texts:
I have a question about MLlib in Spark.(with Scala)

I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but I now have to do it on Spark to be able to analyze Big Data.

How is the variable selection done? Is there any try of different variable combinations in LogisticRegressionWithLBFGS or LogisticRegressionWithSGD? Something like a test of significance of variable one by one? Or a correlation calculation with the variable of interest? Is there any calculation of BIC, AIC to choose the best model?

Because the model only returns weights and intercept...

How can I understand those Spark functions and compare to what I'm used to with SAS or R ?


texts: Anyway to know all details of trees grown using RandomForestClassifier in scikit-learn?

texts:
I am building a standard RandomForest classifier (named model, see the code below) using scikit-learn package. Now, I want to get all parameters of one Randomforest classifier (including its trees (estimators)), so that I can manually draw the flow chart for each tree of the RandomForest classifier. I wonder if anyone knows how it can be done?

Thank you in advance.

Manh

#Import Library
from sklearn.ensemble import RandomForestClassifier #use RandomForestRegressor for regression problem
#Assumed you have, X (predictor) and Y (target) for training data set and x_test(predictor) of test_dataset
# Create Random Forest object
model= RandomForestClassifier(n_estimators=10, max_depth=5) #n_estimators=1000 oob_score = True
#====
#X, y = input_X, input_y

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 4)

# Train the model using the training sets and check score
model.fit(X_train, y_train)

#Predict Output
y_pred_train = model.predict(X_train)
y_pred_test = model.predict(X_test)

#accuracy
from sklearn.metrics import accuracy_score
print(accuracy_score(y_train,y_pred_train))
print(accuracy_score(y_test,y_pred_test))


texts: when speed is not a major concern, what is the best ML algorithm for high dimensional data?

texts:
we are trying to build a predicting model using machine learning algorithms.

I have a use case where the input data have a very high dimension. Each sample point has 20000 features.

we have a decent training sample set with around 1 million training samples, if necessary, we can acquire more, say, 2 or 3 million.

We are not very speed sensitive, it's not like a recommendation system that needs to respond within a second. The application allows us to spend up to a couple of minutes for one prediction. Nevertheless, we hope the algorithm can be run in parallel mode in the future.

given above description, what kind of algorithm would you suggest?

Our biggest concern is over fitting, with so many features, it seems that we are doomed to over-fit.

We were trying to do something alone the line of nearest neighbor, but with these many features, calculating distance sounds like a mission impossible. Maybe we should do PCA to do dimension reduction first?

Any comment is welcomed!


texts:
My data looks like this:

Why is this error showing up?


texts: How can I use different loss functions for images with different labels in tensorflow?

texts:
I have two types of images, meaning two different sets of possible labels. How can I use two different loss functions based on which image I am currently training on with a batch size of 1?
Thanks !


texts: I am working on a binary classification problem. My data contains 100K samples from two different sources. When I perform the training and testing on data from the first source I can achieve classification accuracy up to 98% and when perform training and testing on the data from the second source, I can achieve up to 99%. The problem is when mix both of them, the classification accuracy goes down to 89%. Any idea how to perform the training to achieve high accuracy. Knowing that one of my features is related to the source


texts: How to add a filter/query/condition in Plotly

texts:
I want to add a condition to the column where attractionName.value_counts() >=165. How can I add a query/filter/condition to a DataFrame before I plot the data.
fig = px.bar(rwgor, x="attractionName", y="attraction score summary", color="attraction score summary", title="Long-Form Input")
fig.show()

With the above code I can plot all values in the column attractionName. I only want to plot values whose value_counts() is grater than 165.


texts: I am working on somr dataset and am implementing a deep neural network. There are some typos that I am not familiar with. strong text


texts: I'm reading a paper using GAN to process illumination in image. In the paper, the author mentioned using the SSIM loss for quality evaluation of the reconstructed image. However, in the SSIM loss, there is a hyperparam C1,C2, which depend on the dynamic value of the pixels (the author said it is 1 for their model). However, the paper doesn't mention the output layer of the Generator, which i assume that it is the tanh function. But tanh's output value would be between -1 and 1 (dynamic value = 2), which contradict with the SSIM hyperparam i have mentioned. Can anyone explain this for me?


texts: Can I apply Clustering algorithms to the result of Manifold Visualization Methods?

texts:
Some methods related to manifold-learning are commonly stated as good-for-visualization, such as T-SNE and self-organizing-maps (SOM).

I understand that when referring specifically to "visualization" means that the non-linear dimensionality reduction can provide good insights of data in its low-dimensional projection, but that most commonly this low-dimensional projection cannot be used in machine learning algorithms, since some of information of the high-dimensional structure is lost (roughly).

However, and here the question, If "clusters" are being observed in the visualization is it acceptable to apply a clustering algorithm to the low-dimensional transformed data and analyze the clusters or groups separately?

For example, I'm applying T-SNE to rather-high dimensional data (40 features) and obtaining this representation:

Disregarding the colors you observe in the picture, I would like to apply a clustering algorithm and separate the data from the found clusters (let's say 6 or 7 clusters), and then analyze the characteristics of each cluster using the high-dimensional representation of each point.

This is in synthesis: using the low dimensional to find clusters, and analyze (exploration) each cluster separately using the high dimensional representation. If I'm not able to do this, I don't see the actual point of visualizing in the low-dimensional space, in a practical sense.

I understand T-SNE preserves well local structures and less accurately global structures, is this a drawback to do why I want? Is this low-dim clustering approach more suitable for other manifold learning methods?

EDIT:
Probably a more direct way of asking this is: Can I use observed clusters in low dimensional representation to label or tag examples, and use these labels for discrimination using the original high dimensional representation?


texts: Tools suitable for semi-automatic video labeling?

texts:
So far I have been using labelme to label objects in videos I use for training, but it is quite time consuming.

Are there good tools to help with that?

I was thinking about a tool where I label some frames, and the tool would then match the labelled areas in the next frames. Note that a simple interpolation would not be enough.


texts: Predicting millions of sparse timeseries using them to help each other

texts:
This is a very general problem faced by different types of businesses. Predict the future behavior of customers over time.
Imagine that we have 1 million customers with their own features over time, forming a time series. The classic approach (statistics like ARIMA, Bayesian like prophet or purely computational like LSTM) here is not very suitable as I would consider that each series team is completely independent, many series are super sparse or have just started and to treat them individually that train a model for each series.
The more conservative approach asks a first classifier to understand what is the behavior of the time series (as a kind of grouper) so we have curves with the same seasonality and trends and then normalize so that the level is the same. We train the models for each of these groups and then we go back to the old level and we have an individual forecast.
I would like to know if there is a more appropriate technique or a canonical solution for the model to understand and predict all series more accurately, using "knowledge of the group" (of the time series that are more similar) in a more scalable way.


texts: Convolution neural network: small dataset affecting accuracy

texts:
I have dataset of 36 folders 1 image each(total 36 images) the dataset is too small but these are character images which i want to train my val_acc= 0.0229 and y_pred=model.predict(x_test) always predict class=4.

How I can improve this accuracy for such a small dataset? Why does it always predict class 4?


texts: Do precision-recall curves have a constant shape/pattern?

texts:
I know ROC curve always looks like a stair shape and that I can evaluate AUC of ROC. And I know I can compute AUC of ROC curve to compare which model is better. What I wonder is:

Does precision-recall curves have a constant shape/pattern?
Can I compute AUC of PR curve to compare which model is better?
If not, how can you compare models by PR curves?


texts: Balancing the dataset using imblearn undersampling, oversampling and combine?

texts:
I have the imbalanced dataset:
data['Class'].value_counts()
Out[22]:
0    137757
1      4905
Name: Class, dtype: int64

X_train, X_valid, y_train, y_valid = train_test_split(input_x, input_y, test_size=0.20, random_state=seed)
print(sorted(Counter(y_train).items()))
[(0, 110215), (1, 3914)]

I tried different imblearn functions:
from imblearn.combine import SMOTEENN, SMOTETomek
from imblearn.over_sampling import  ADASYN, BorderlineSMOTE, RandomOverSampler, SMOTE
from imblearn.under_sampling import CondensedNearestNeighbour, EditedNearestNeighbours, RepeatedEditedNearestNeighbours
from imblearn.under_sampling import AllKNN, InstanceHardnessThreshold, NeighbourhoodCleaningRule, TomekLinks

smote_enn = SMOTEENN(random_state=27)
smote_tomek = SMOTETomek(random_state=27)

adasyn = ADASYN(random_state=27)
borderline = BorderlineSMOTE(random_state=27)
ran_oversample = RandomOverSampler(random_state=27)
smote = SMOTE(random_state=27)

cnn = CondensedNearestNeighbour(random_state=27)
enn = EditedNearestNeighbours(random_state=27)
renn = RepeatedEditedNearestNeighbours(random_state=27)
allknn = AllKNN(random_state=27)
iht = InstanceHardnessThreshold(random_state=0)
ncr = NeighbourhoodCleaningRule(random_state=27)
tomek = TomekLinks(random_state=27)

Created different trains:
def BalancingData(function):
    X_train_resampled, y_train_resampled = function.fit_sample(X_train, y_train)
    print(sorted(Counter(y_train_resampled).items()))
    return X_train_resampled, y_train_resampled

X_train_smote_enn, y_train_smote_enn = BalancingData(smote_enn)
X_train_smote_tomek, y_train_smote_tomek = BalancingData(smote_tomek)

X_train_adasyn, y_train_adasyn = BalancingData(adasyn)
X_train_borderline, y_train_borderline = BalancingData(borderline)
X_train_ran_oversample, y_train_ran_oversample = BalancingData(ran_oversample)
X_train_smote, y_train_smote = BalancingData(smote)

X_train_cnn, y_train_cnn = BalancingData(cnn)
X_train_enn, y_train_enn = BalancingData(enn)
X_train_renn, y_train_renn = BalancingData(renn)
X_train_allknn, y_train_allknn = BalancingData(allknn)
X_train_iht, y_train_iht = BalancingData(iht)
X_train_ncr, y_train_ncr = BalancingData(ncr)
X_train_tomek, y_train_tomek = BalancingData(tomek)

Then built the Keras model:
def my_model(X,y):
    Keras_model = Sequential()
    Keras_model.add(Dense(33,activation='sigmoid', kernel_initializer='glorot_uniform', kernel_constraint=maxnorm(12), input_shape=(input_len,)))
    Keras_model.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform'))
    Keras_model.compile(optimizer = 'Adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
    Keras_model.fit(X, y, validation_data=(X_valid,y_valid), batch_size = 1000, epochs = 10, verbose = 0)
    scores_valid = Keras_model.evaluate(X_valid, y_valid, verbose=1)
    scores_train = Keras_model.evaluate(X, y, verbose=1)
    scores_full = Keras_model.evaluate(X_train, y_train, verbose=1)

call with different imb train and valid sets
my_model(X_train, y_train)
my_model(X_train_smote_enn, y_train_smote_enn)
my_model(X_train_smote_tomek, y_train_smote_tomek)
and so on

I find very minor improvement in the accuracy in allKNN. This is the initial benchmark, I know we can still tune the parameters in allKNN and try to improve it. My question is: why can't I find very much improvement without using the balancing functions the base model itself showing a better accuracy?


texts: WEKA Multilayer Perceptron with large dataset

texts:
I'm new to data mining using WEKA.

I was trying out datasets with a large dataset (2000+ attributes with 90 instances) and left the default parameters as it is.

Why is Multilayer Perceptron running long on a dataset with 2000+ attributes?
K-Nearest Neighbour does a better job in terms of speed given the same dataset.

How does the hiddenLayer in MLP affect the speed and accuracy of the training set?

What is the most recommended way in running such large dataset, or is there none?


texts: Evaluating a trained Reinforcement Learning Agent?

texts:
I am new to reinforcement learning agent training. I have read about PPO algorithm and used stable baselines library to train an agent using PPO. So my question here is how do I evaluate a trained RL agent. Consider for a regression or classification problem I have metrics like r2_score or accuracy etc.. Are there any such parameters or how do I test the agent, conclude that the agent is trained well or bad.

Thanks


texts: why do we operate with graphical models in VAE, if there are no probabilites involved?

texts:
In the variational autoencoder, I often see graphical models e.g. $P(X|Z)$ for the decoder, but when I looked at code, I don't see any random variables, I see just deterministic network, with composite functions.

I am very confused by that, can someone explain where the magic happens? I am not interested in KL divergence and other stuff. I just don't understand how graphical model > $P(X|Z)$ == composite functions.


texts:
I'm using two different functions to calculate the accuracy of my deep learning model and I am confused which one is which.

The first one is

    loss, accuracy = model.evaluate(x_train, y_train, verbose=0)
    print('Accuracy: %f' % (accuracy))
    print('Loss: %f' % (loss))

    Accuracy: 0.731495
    Loss: 1.136258

And the second way is to obtain the accuracy:

test_accuracy = history2.history['val_acc']

Accuracy is 0.731 and test_accuracy is around 0.21
Which is one is my models' accuracy


texts: How (and why) are random forests able to represent both linear and non linear data

texts:
From reading articles I've read that random forests are suited for representing both linear and non linear data quite well, but I can't find an explanation as to how and why it has his flexibility?

Any information to clarify this would be great!


texts: Understanding the ||w|| = 1 constraint for SVMs

texts:
Is it correct to say that the reason why ||w|| is set to 1 in the formula for the geometric margin is that it then is the same as the functional margin (i. e. gives the same information) while still being scale invariant?
I come to that from this site: Support Vector Machine (SVM) where it says "If ||w||=1, the functional margin is equal to geometric margin. "
If not, then why can ||w|| not be set to 0.7 oder 2.53 or 14?
I have read various blogs and pdfs on the topic. Some don't adress this at all or just say things along the lines of "this is a mathematical trick" and don't really explain it.


texts:
I am doing image classification with a CNN and I am having trouble building a network that does not do overfitting. I have in my training set 2000 images of 4 classes, while in my test set I have 3038 of the same 4 classes. My CNN is the following:

def Network(input_shape, num_classes, regl2 = 0.0001, lr=0.0001):

model = Sequential()

# C1 Convolutional Layer
model.add(Conv2D(filters=32, input_shape=input_shape, kernel_size=(3,3),\
                 strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation before passing it to the next layer
model.add(BatchNormalization())

# C2 Convolutional Layer
model.add(Conv2D(filters=64, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C3 Convolutional Layer
model.add(Conv2D(filters=128, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Batch Normalisation
model.add(BatchNormalization())

# C4 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
#Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# C5 Convolutional Layer
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

 # C6 Convolutional Layer
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

 # C7 Convolutional Layer
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Conv2D(filters=512, kernel_size=(3,3), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
# Pooling
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))
# Batch Normalisation
model.add(BatchNormalization())

# Flatten
model.add(Flatten())

flatten_shape = (input_shape[0]*input_shape[1]*input_shape[2],)

# D1 Dense Layer
model.add(Dense(4096, input_shape=flatten_shape, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D2 Dense Layer
model.add(Dense(4096, kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# D3 Dense Layer
model.add(Dense(1000,kernel_regularizer=regularizers.l2(regl2)))
model.add(Activation('relu'))
# Dropout
model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(num_classes))
model.add(Activation('softmax'))

# Compile

adam = optimizers.Adam(lr=lr)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

return model

and everytime I train and test I clearly overfit, because if I test the model I obtain a low accuracy, around 45%, and the curves of accuracy for test and training are really far apart if I plot them.

How could I improve my network in such a way it does not overfits?

Thanks in advance.


texts:
Imagine this example:

We see that the attributes color and points have the same value. What attribute should we choose for the construction of the tree?


texts: Appropriate algorithm for string (not document) classification?

texts:
I am trying to classify a large-ish number of small strings (millions) into about 10 disjunct categories. Examples of classes and strings for each class include:

email: "foo@bar.com"
phone: "55", "22334455"
personName: "John", "Q.", "Public"
organizationName: "Reuters", "IBM"
date: "Dec.", "22.10.2010"
nameAndEmail: "Larsonlarson@gmail.com" (a last name has been concatenated with an email address.)
phoneAndEmail: "55larson@gmail.com" (part of a phone number has been concatenated with an emal address)
separator: ",", "and", "or"
other: stuff that does not fit in other categories.

Carrying out classification via a set of heuristics seems tedious, so I have been experimenting with an SVM. I'm not able to get more than about 80% accuracy without doing a lot of work manually encoding features like "has an ampersand", "is uppercase", "is mostly numeric", "has a colon in the middle", etc., which sort of defeats the purpose.

I'm wondering whether part of the challenge with using an SVM is that the "salient features" of a given string, like the '@' character, are not in a fixed position in the string, so it will not get the same feature index for each string.

Does anyone have suggestions for a more appropriate approach to this task, or can anyone recommend further reading?


texts: Group similar words under one topic and assign them a title

texts:
I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do that? I tried topic modelling using MALLET. It groups similar words into a cluster. I want a heading or subject attached to them.

Is there a way to group similar words under one topic and assign them a title?

There are a number of different features that need to get one topic like picture quality camera selfie resolution that should come under camera. Need something that groups these things I enter words and it gives me the topic.


texts: what exactly happens during each epoch in neural network training

texts:
Across different epochs, which of the following is/are updated?

initial weights (initial ConvNet filter matrices, initial fully connected weights)

hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers...

The lost function calculated from the last epoch appears to be the initial value of the lost function for the current epoch. Why?


texts: Curse of dimensionality: Data set with 1 million rows, max number of features can I have approximately?

texts:
As the question entails, I would like to know as a rule of thumb, an upper limit on the number of features I could have for a data set with 1,000,000 rows/observations, before hitting the curse of dimensionality. If not a million observations, an answer with a rough estimate for the number of features for a data set with a specified size would be great too.

Thanks


texts: How can i classify only-text image(petition, invoice, page of a book) and a normal image(cat,person ) as a 2 different type?

texts: I don't think detecting text on image does work because normal images can have text on them too(like traffic sign). Is there a way to do this? I couldn't come up with an answer.


texts:
Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results.

However, scaling this to $10$ million data points requires a creatively efficient implementation since DBSCAN requires $O(n^2)$ to calculate the distance matrix and crushes my memory.

There must be some efficient sampling-based method to overcome this, ideally something similar to minHash - but I'm not sure how to approach this, and if there exists a  solution that can work on existing sklearn DBSCAN algorithm. Any ideas?


texts: How to compare the performance of different number of mixing components for EM algorithm?

texts:
I am reading about the EM (Expectation-Maximization) algorithm in a machine learning book. At the end remark of the chapter, the authors mentioned that we cannot decide the "optimality" of the number of components (# of mixtures of Gaussians distributions) based on each model's log likelihood at the end--since models with more parameters will inevitably describe the data better.

Therefore, my questions are

1) How do we compare the performance of each model using a different number of components?

2) What are the important factors that help us to decide that an EM model is sufficient for modeling the observed data?


texts: Improve performance for known unlabelled test set

texts:
I'm training a machine learning model from a training set of $1,000$ samples (but around $23,000$ features). I'm fairly content with my cross-validated results, but would like to improve it further by using information I have about the test set I will be using.
To explain that some more: I have an unlabelled test set with $500$ unlabelled samples which I know is the one my model needs to perform on. I was wondering if there are some techniques which allow one to weight features based on some criterion, for example whether they are significant in the test set (or alternatively, find which samples in the train set are "close" to the ones in the test set, and try to fit them with a higher importance than the other, more distant samples). In case there is such a technique, I'd also be interested in a way to do this in sklearn in particular (and preferably as part of a pipeline so that I could use it in cross-validation as well, since it is very important that I be able to gauge the accuracy of my model).


texts: Is Data Science just a trend or is a long term concept?

texts:
I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data science is just a mix of computer science and statistics techniques.
So I always think about this. If it is just a trend and if in 10 years from now, someone will still mention Data Science as an entire field or just a subject/topic inside CS or stats.
What do you think?


texts: How to calculate the mean of a dataframe column and find the top 10%

texts:
I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it into a DataFrame so I can use SparkSQL to select groups of players via their stats that meet certain criteria.

Once I have the subset of players I am interested in looking at further, I would like to find the mean of a column; eg Batting Average or RBIs. From there I would like to break all the players into percentile groups based on their average performance compared to all players; the top 10%, bottom 10%, 40-50%

I've been able to use the DataFrame.describe() function to return a summary of a desired column (mean, stddev, count, min, and max) all as strings though. Is there a better way to get just the mean and stddev as Doubles, and what is the best way of breaking the players into groups of 10-percentiles?

So far my thoughts are to find the values that bookend the percentile ranges and writing a function that groups players via comparators, but that feels like it is bordering on reinventing the wheel.

I have the following imports currently:

 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.SQLContext
 import org.apache.spark.{SparkConf, SparkContext}
 import org.joda.time.format.DateTimeFormat


texts: Extract company names/job titles from free text

texts:
I have a complete Hadoop platform with HDFS, MR, Hive, PIG, Hbase, etc., Python, R, Java. All data sets have a large size.

The data set A, describing the jobs of people working in a company, is composed of the following fields:

Id Person: a unique alphanumeric identifier per person.
Start Date: a date format iso entry in the post
End Date: iso size release date of the position. If the date is not given, it is the current position
Job Title: a text field containing the title and the name of the company. The text is free, non-standardized, French and / or English and can contain typos. Ex: Director Big Data Analytics with Google, Commercial Manager at [missing text] , Manager at googole ...

My question is; how can I create a feature to easily process the name of company of the job (jobtitle)?

Thank you in advance


texts:
Which is the best way to compare soccer teams from different sites? For example, one soccer team in a site is named Academica Clinceni and in another FC Clinceni and in  another one is Acs Fc Academica Clinceni.
Moreover, for a team can exist also the female version of the team (eg. Acs Fc Academica Clinceni (W) or Acs Fc Academica Clinceni (F)). Which is the best approach to distinguish these teams? For the first problem I used the Fuzzy String Matching, in your opinion is it right?


texts: Reusing a portion of a neural network (with shared weights) in Keras

texts: I have a partial neural network with several layers of various types, with weights $\theta$; let's call it $F_{\theta}$; and the dimensions of input and output arrays are the same, implemented in Keras with Tensorflow backend. Within this framework, how do train a model of the form $y = F_{\theta}(F_{\theta}(x))$, or more generally some $n$-th iteration of $F_{\theta}$? In other words, I reuse an entire subset of the neural network, with all the same weights, so the full model is deeper but does not have more parameters than the shallower model.


texts: How can I generate reasonable dummy/artificial data from a pre-existing time-series data?

texts:
I have a dataset like this, basically all numerical time-series data.

I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for reasonable looking future values?
I checked some packages like faker, Timesynth; but couldn't figure out how to produce data which considers previous data.


texts:
In NLP, there is the concept of Gazetteer which can be quite useful for creating annotations. As far as I understand:

  A gazetteer consists of a set of lists containing names of entities
  such as cities, organisations, days of the week, etc. These lists are
  used to ﬁnd occurrences of these names in text, e.g. for the task of
  named entity recognition.

So it is essentially a lookup. Isn't this kind of a cheat? If we use a Gazetteer for detecting named entities, then there is not much Natural Language Processing going on. Ideally, I would want to detect named entities using NLP techniques. Otherwise, how is it any better than a regex pattern matcher?


texts: Better understanding of Integrated Gradients

texts:
I've been trying to use Integrated Gradients to get a better understanding of the attribution of different features in my NN.
I've read the original publication(https://arxiv.org/pdf/1703.01365.pdf) and the distill explanation(https://distill.pub/2020/attribution-baselines/) and I still have a few questions which I hope you can help understand:

Do the scores I get is actual Aumann-Shapley values?
I couldn't find anything regarding the scale of the scores I get. So if feature A get a score of 0.1 and a feature B score of 1 I know B is 10 times more important. But with no limit to the score, I can't understand how I can relate to those scores as absolute numbers.
From what I can understand, Aumann-Shapley values look on the coalition factor, meaning how one feature affects the prediction as a coalition with other features. On the other hand, looking at the equations from the article it seems we look at the derivative of each feature separately. I was left confused if the integrated gradients values represent the importance of every feature alone or with regard to the other features and how they "work together"?

Thank you in advance for all the help


texts:
I am trying to understand the transformer architecture.

I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example take 8 heads.

Now for a particular layer we will have 8 different sets of (Wq, Wk, Wv), the weight matrices used to calculate the query, key and value.

Now what I want to know is whether these weight matrices are shared between the different layers i.e are the (Wq, Wk, Wv) matrices of head#1 in layer 1 same for head#1 of layers 2, 3, ....?

And if they are shared, doesn't it affect in parallelization?


texts: Standardization of variables in log-scale

texts: I have a doubt regarding standardization, I have to use a multivariate regression and one of the variables is in log-scale. Is using standardization enough to re-scale the log variable, or is it necessary to apply a transformation to a linear scale and then apply standardization?


texts: Caret and rpart - does caret automatically prune rpart trees

texts:
Question relating to the caret package 'rpart' method.

Does the method='rpart' automatically prune the tree?  If so, what rules does it follow?  If not, how does one go about directing caret to do this?


texts: Is it better to optimize hyperparameters or run multiple epochs?

texts:
Whenever I train a neural network I only have it go through a few epochs ( 1 to 3). This is because I am training them on a bad CPU and it would take some time to have the neural network go though many epochs.

However, whenever my neural network performs poorly, rather than have it go through more epochs, I try to optimize the hyperparameters. This approach has generally been successful as my neural networks are pretty simple.

But is training a neural network in this manner a bad practice? Are there disadvantages to immediately going to optimize the hyperparameters rather than running the neural network for more epochs?


texts: In supervised learning, what does "Estimating $p(y \vert x)$" mean?

texts:
I read chapter 5.1.3 of Joshua Bengio's deeplearning book, which says:
supervised learning involve observing examples of random vectors $\textbf{x}$ and associated value or vector $\textbf{y}$ and learning to predict $\textbf{y}$ from $\textbf{x}$ by estimating $p(\textbf{y} \vert \textbf{x})$.

What does $p(\textbf{y} \vert \textbf{x})$ mean?

From basic statistics, I know that $p(\textbf{y} \vert \textbf{x})=\frac{p(\textbf{y},\textbf{x})}{p(\textbf{x})}$.
How do we find $p(\textbf{y},\textbf{x})$ and  $p(\textbf{x})$?


texts: How much neural network theory required to design one?

texts:
So I have looked at some of the literature on neural networks and read some chapters, but the learning curve is so steep that I have had trouble even getting started on designing the neural network to solve my problem.

From what I understand, the architecture (or arrangement of neurons and their connections) should be designed according to the nature of the problem to be solved. Other parameters to be set according to the nature of the problem include the loss index (how the error will be calculated and if there should be a regularization term), whether or not there should be any scaling/unscaling, bounding, or conditions, and the training algorithm (such as the quasi-Newton method).

The particular type of problem I am interested in is using neural networks to figure out unknown functions (with unknown complexity) that input and output integers (as opposed to continuous values), given a large collection of inputs and outputs.

  An example function takes 4 byte inputs and returns 2 byte outputs. This is done by first taking the first 2 bytes of input and XORing them with the last 2 bytes, to produce an intermediate result. This two byte value is then XORed with a copy of itself that is shifted left by 5 bits. This result is then XORed with a copy of itself shifted right by 7 bits. Then this result is then XORed with a copy of itself shifted left by 2 bits, and this value is outputted by the function, giving the final 2-byte result. Note: More than one unique input can produce the same output.

So given a large set of inputs and outputs of an unknown function, the neural network should then optimize itself to reproduce this function given new inputs. I am not sure how to get started designing this neural network, and I am not sure fully reading through neural networks textbooks is the optimal way to get started. I am using a software library meant for designing neural networks, and I can simply set the network architecture and the parameters described above. How much theory do I need to know in order to get started solving my problem? Where should I start with learning how to design this neural network?

EDIT:
The main goal of all this is to use an existing tool to simplify producing an output (a 2-byte code, in the example above) given a new input, in a situation where the function is unknown. Neural networks seem to match the function-finding-through-trial-and-error characteristics that I need. This tool should be able to try all sorts of possible functions of increasing complexities in order to mimic the working of the actual unknown function.


texts: (Graph Convolutional Network (GCN) based recommender system maintenance issue

texts:
I have built an item-item recommender model using (Graph Convolutional Network (GCN) for an E-commerce website. Could you please help me with the maintenance of the model.

How often should I retrain the model after deployment?( 1 month, 2 months)

What are the main metrics which I should track after deployment to evaluate the model?

Any suggestions related to Train/valid/test split ratios (such as 70:15:15 or 60:20:20)

Any materials related to model maintenance/recommder systems evaluation post deployment would be of great help.

Thank you


texts: What if my validation set is worse than my training?

texts: I am running a CNN, on the 1st epoch my training set accuracy is 15% and validation set is 12%, by the 51st epoch my training accuracy is 87% and validation set is 13%. What is happening? What does it mean if my validation set is lower or greater than my training set ?? (my test set gives 12%)


texts: Doubt on retrieving tweets for sentiment analysis?

texts:
I wanted to do my first project on sentiment analysis.
My goal is trying to do sentiment analysis on tweets that mention
'Uber' and see if there is correlation with the stock price: 'UBER'.

Problem is it went public this year, so I would need tweets from this year, my question is:

Is there any way of avoiding paying for the api:
 https://developer.twitter.com/en/pricing   ? How can I get let's say 10,000 tweets of this year mentioning uber?

Or maybe, given I don't have any experience shall I instead try doing it on an already created dataset? ie https://www.kaggle.com/davidwallach/financial-tweets ? (problem here is that this data set only has 28k tweets, and if I filter for the ones only naming stock 'xyz', I will likely get only a hundred tweets...)


texts: k-means clustering data with large number of meaningless values

texts:
I am looking to perform k-means on my dataset which contains a large number of 0 values.
The last value you see is different to the others, that is simply the sum of transactions, not related to the categorical frequency count.
Example:
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 189200.579626]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.06556796]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.46e-06]

Each feature is a frequency count of how many times the associated id is seen within a category.
When I run k-means, I see that most of the data is clustered in one cluster.
+------------+--------+----------------------+
| cluster_id |  size  | sum_squared_distance |
+------------+--------+----------------------+
|     0      | 659187 |    0.999997057952    |
|     1      |   3    |    1.33333326876     |
|     2      |   1    |         0.0          |
|     3      |   3    |    0.666666716337    |
|     4      |   1    |         0.0          |
|     5      |   1    |         0.0          |
|     6      |   1    |         0.0          |
|     7      |   1    |         0.0          |
|     8      |   1    |         0.0          |
|     9      |   11   |    2.72727286816     |
+------------+--------+----------------------+

I am assuming that this is because the majority of the dataset has not been seen in a feature category and therefore has a value of 0.
What is the best way to overcome this, drop rows where a 0 is seen across each category? Are these rows meaningless to clustering?


texts:
I am about to choose ML research topic for my master thesis, but i am at a dead end. The problem is, that while reading research papers, i find solutions, but not an open problems.

For now, a came up with such ideas:

Research neural network quantization algorithms - i started to research this topic 3 months ago and it seems to me, that current papers reveals high accucary methods. I have not idea what can i to do in this field
Implement online video-processing system using quantized neural networks. It sounds also good, but it could be done with few lines of Tensorflow Lite code.
Intelligent Image Database - image database, that has image classifiers and detectors for automatic annotation. It seems to be a good idea, but i need to find here a research part. For me this tasks sounds like pure engineering task
Make some signal processing research. It could be connected with first idea as optimal hardware implementation. But i don't know where to search for suitfull specific task for ML, because a lot of dsp problems are solved with classical analytical mathematical methods.

As you can see, i always see some disadvatages in ideas. It stopping me from starting to work. Can you give me an advice how to develop one of this ideas to make it suitable both for engineering and research? Generally, how to search for research topics?


texts: (Labeled, if possible) time-series datasets for anomaly detection

texts:
I would like to create a big list of available time-series datasets for anomaly detection. I'm especially interested in the following:

The time-series data should be segmented into cycles
Ideally, these cycles should be of the same length
These cycles should be labeled as normal/anomalous

But anything goes. I will be sharing the ones I found below.


texts: Random forest with zero precision for unbalanced test data

texts:
Apologies if this is a basic question.

I have a very unbalanced dataset in which the records are labelled by one of two  classes, class1 (negative class) and class2 (positive class):

class 1: 1.5 million records

class 2: 100 thousand records

So there is essentially a 1:15 ratio between them.

Because I am dealing with time dependent observations, I have divided the train/test data in the following way:

train data:

class1: 900 thousand

class2: ~100 thousand

test data:
class1: 600 thousand

class2: 25

Thus having 62% of training data and 38% of test data, the test data being more recent.

I am using a Random Forest Classifier as a model, and have given a class weight of 1 to class1 and 9 to class2, to compensate for the data imbalance.

I read that to measure the quality of a scenario with unbalanced data, I should use a Precision vs Recall curve.

The problem is, none of the 25 data points from class2  are classified correctly, and thus the true_positives = 0.

In a case like this, how can I evaluate my model effectively?
Should I undersample class1 from my test data?
Are there more parameters besides class weights that I should try to tune?

Thank you for your time.


texts:
I want to extract each tree so that I can feed it with any data, and see the output.

dump_list=xg_clas.get_booster().get_dump()
num_t=len(dump_list)
print("Number of Trees=",num_t)

I can find number of trees like this,

xgb.plot_tree(xg_clas, num_trees=0)
plt.rcParams['figure.figsize']=[50, 10]
plt.show()

graph each tree like this. When I do something like:

  dump_list[0]

it gives me the tree as a text. But I couldn't find any way to extract a tree as an object, and use it.

https://github.com/dmlc/xgboost/issues/117#ref-commit-3f6ff43
I found this but didn't really understand what is suggested.

Progress: I tried to somehow turn

  dump_list[0]

string object into a sklearn DecisionTreeClassifier object. Still no luck.

I uploaded my notebook if you want to check it out: https://github.com/sciencelove11/Question


texts:
I have a data set of total 8000 sound samples. These are the results of my multi layer neural network, binary classifier:

Precision: [0.95 0.96]
Recall: [0.96 0.95]
F-Score: [0.95 0.95]
S: [1217. 1254.]
Accuracy training 1.0
Accuracy Test 0.95

I am happy with the test accuracy, precision and recall. But the 1.0 accuracy bugs me since its overfitting to the training-set. Is this a bad thing even the test-set accuracy is satisfactory?

Here below I load the car sound features(1) and other types of sounds features(0) from disk and assign the labels. I double checked and I dont add labels as a feature while I process and extract features from audio. sets are cut to be equal to 4100 samples each

car_features =np.load('car_features_final.npy')
car_labels =np.ones(len(car_features),dtype=int)
shuffle(car_features, random_state=12)

other_features = np.load('other_features.npy')
other_features=list(islice(shuffle(other_features, random_state=12),4100))
other_labels = np.zeros(len(other_features),dtype=int)
shuffle(other_features, random_state=12)

all_features = np.append(car_features, other_features, axis=0)
all_labels = np.append(car_labels,other_labels, axis=0)

X_train, X_test, y_train, y_test = train_test_split(all_features, all_labels, test_size=0.30, random_state=42)

clf = MLPClassifier(activation='relu', solver='adam', alpha= 0.1, hidden_layer_sizes=(300, 300, 300, 100), random_state=1, max_iter=500)
clf.fit(X_train, y_train)

y_predicted_test = clf.predict(X_test)
y_predicted_train = clf.predict(X_train)
p,r,f,s = precision_recall_fscore_support(y_test, y_predicted_test)


texts: I have read somewhere that the value of slack variables of support vectors is not 0. Does that mean the points lying in the wrong region e.g a positive point lying in the negative region will also be a support vector? I have attached one picture as well which shows that points lying in the wrong region also support vectors. I am looking for an explanation of this phenomenon


texts:
I am solving a problem connected with medicine and from each patient I get about 100 features. It's a classification diseases problem, however measurements take a lot of time and also require money.

What is the optimal size of training data? I understand that I should collect as much data as possible, however is there any empirical advice?


texts: looking for a correlation function for categorical + numerical datasets in matlab

texts:
I am looking for a function in Matlab that calculates the correlation coefficient of a data set that includes categorical + numerical data. I am quite surprised this isn't as straightforward as I hoped it would be.

Can someone point me in the right direction?


texts:
I want to create a histogram out of the range of a column named surfacearea with country data frame. I made an sql file

# In R
range <- max(country$surface) - min(country$surfacearea)

#in SQL
# part B Divide the range into 5 equal width bins
#SELECT CASE
#WHEN surfacearea  <  3415080 THEN "LOW"
#WHEN surfacearea <= 6830160 and surfacearea > 3415081 THEN "MEDIUM-LOW"
#WHEN surfacearea <= 10245240 and surfacearea > 6830161 THEN "MEDIUM"
#WHEN surfacearea <= 13660320 and surfacearea > 10245240 THEN "MEDIUM-HIGH"
#ELSE "HIGH" END AS `type`, COUNT(surfacearea)
#GROUP BY `type`
#ORDER BY `type` ASC;

I'm working on putting mySQL into r


texts: Using sklearn DictVectorizer in real-time systems

texts:
Any binary one-hot encoding is aware of only values seen in training, so features not encountered during fitting will be silently ignored. For real time, where you have millions of records in a second, and features have very high cardinality, you need to keep your hasher/mapper updated with the data.

How can we do an incremental update to the hasher (rather calculating the entire fit() every time we incounter a new feature-value pair)? What is the suggested approach here the tackle this?


texts: Why does decreasing the SGD learning rate cause a massive increase in accuracy?

texts:
In papers such as this I often see training curves with this kind of shape:

In this case SGD was used with a factor of 0.9 and learning rate decreasing by a factor of 10 every 30 epochs.

Why is there such a large decrease in error when the learning rate is changed?
Why does the validation error begin to increase after the initial drop, whereas the training error continues decreasing?
Can the same results be obtained by moving the 2nd and subsequent learning rate changes closer together? That is, why the delay in doing further drops?


texts: How can I extract the top words from a string in my dataframe column?

texts:
I have a column in my dataframe which is in a string providing description of a product. For example : This is a shirt. It is blue in color. The sizes available are large, small. The shirt is tight fit.

I want to extract the keywords from this description like shirt, blue, large, small, sizes, tight etc.

I am using python. How can I achieve this ?


texts: How to scale a variable when not knowing the maximum

texts:
I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target).

However, normalizing between 0-1 for instance means that the variable maximum value will be equal to one, and the mean to 0.

Now if I receive a new example never seen before, and this example has a value higher than the max of the training examples, how should this value be normalized ?

EDIT

As an example. If my maximum value is 150, it will be scaled to 1.0. Now if I receive a new example, with a value equal to 320, how should it be scaled ?


texts: Classifying points exactly on decision boundary

texts:
For calculating loss arose by classification we do this:

If $y (w \cdot x + b) > 0$: $\text{no loss}$

If $y (w \cdot x + b) < 0$: $\text{loss} = −y (w \cdot x + b)$

So here what about the points exactly on decision boundary? How do we classify and compute their loss? (since it would become 0)


texts: Constructing a graph where any node can have the highest PageRank among all nodes

texts:
I'm trying to solve q5.1.4 in mmds chapter5, however, I'm not sure how I can even start. The question is:

Construct, for any integer n, a Web such that, depending on
β, any of the n nodes can have the highest PageRank among those n. It is
allowed for there to be other nodes in the Web besides these n.

I would've attached my work so far but as I said I'm not sure where to start from.


texts:
Here is Colab Notebook

After 1500 episodes if batch_size=256, the RAM crashed. With Colab, I have the equivalent of 25.5 gigs of RAM. Is it normal? Or I don't have enough ram? I thought at a certain point, the RAM must free a bit of space. How can I fix that?


texts: Term for Methods of Representing Repeated Text in Classifier

texts:
A colleague told me that there are terms for two different methods of representing repeated text in the training set for a classifier, but he could not recall them. What are the terms for the options described below?

Suppose I want to train a classifier, such as the scikit-learn Naive Bayes classifier on sentences extracted from text documents. Any sentence may be appear any number of times in the documents. I have determined the number of instances of each distinct sentence and assigned labels to the ones I will use for training.

Option 1: One Instance per Distinct Sentence
I could include in my training set one instance of each distinct sentence. This method gives equal weight to different sentences, regardless of their frequency.

Option 2: Multiple Instances by Relative Frequency
I could include in my training set multiple instances of each distinct sentence based on the number of times the sentence appears in the corpus. If the corpus contains 100 instances of Sentence 1 and 25 instances of Sentence 2, then my training set would include four instances of Sentence 1 and one instance of Sentence 2.

This method gives greater weight to sentences that appear more frequently in the corpus.

Other Options: Are there related strategies that also have standard names?


texts: GausianNB: Could not convert string to float: 'Thu Apr 16 23:58:58 2015'

texts:
I'm beginner in python so please bare with me. I'm trying to solve one machine learning problem using GaussianNB. I've certain fields which are not in proper date format, so I converted it into UNIX format. For example column state_changed_at has value in csv as 1449619185. I'm converting it into proper date format.

Now the problem is, when I'm selecting those date features to train my model, it gives me an error:

  Could not convert string to float: 'Thu Apr 16 23:58:58 2015'

import pandas as pd
import numpy as np
from sklearn import metrics
from sklearn.naive_bayes import BernoulliNB
from sklearn.naive_bayes import MultinomialNB
import time
from sklearn.naive_bayes import GaussianNB

train = pd.read_csv("datasets/train2.csv")
test = pd.read_csv("datasets/test.csv")
train.head()

import time

#   state_changed_at,deadline,created_at,launched_at are date time fields
# and I'm converting it into unix format
unix_cols = ['deadline','state_changed_at','launched_at','created_at']

for x in unix_cols:
    train[x] = train[x].apply(lambda k: time.ctime(k))
    test[x] = test[x].apply(lambda k: time.ctime(k))

#   state_changed_at,deadline,created_at,launched_at are date time fields.
cols_to_use = ['keywords_len' ,'keywords_count','state_changed_at','deadline','created_at','launched_at']

target = train['final_status']

# data for modeling
k_train = train[cols_to_use]
k_test = test[cols_to_use]

gnb = GaussianNB()

model = MultinomialNB()
model.fit(k_train, target)  # this lines gives me error saying: could not convert string to float: 'Thu Apr 16 23:58:58 2015'

expected = target
predicted = model.predict(k_test)
print(model.score(k_test, predicted, sample_weight=None))

Any help would be really appreciated.
Thank you


texts:
I have 60000 images divided into two classes. I have tried to build transfer learning with pretrained ResNet50 but my new GTX 1080 ti returns -1 after couples of epochs. My guess is that it runs out of memory. Which is weird because with the same script and the same data I could successfully ran it on GTX 1050 ti. Everything is in tensorflow-gpu backend

I have installed CUDA 9.0 with cudnn 7.1.4 on ubuntu for GTX 1080 ti.
Here is nvidia-smi

And I see lack of ram:


texts: Compare probability estimates of two classifiers

texts:
Consider two binary classifiers A and B. Suppose that both A and B are predicting the same target, but that A is trained on a subset of the data for which a different set of features is available than for B. Suppose further that A and B output probability esitmates and that for a set of mixed input types (some with features that A can accept, others for B) I need to rank them by probability of being positive.

Perhaps (and this is completely fabricated) I'm trying to predict early mortality of lions and tigers in a zoo. Different features are available for early lion mortality than for early tiger mortality. However, I ultimately will need to order all of the big cats, (among both lions AND tigers) by likelihood of early mortality.

My ideas so far:

Train a meta-estimator that accepts the input type (belonging to A or B) and the scores from A and B. I'm thinking xgboost or another tree-based regressor.
Attempt to directly and monotonically transform the probability estimates from A and B so as to 'bring them into alignment' in such a way as to maximize the cross-validated AUC. This seems to be similar to the first option.

Other information:
 In my actual application there is class imbalance, and the class imbalance varies between A type datapoints and B type datapoints. Precision TPR/(TPR+FPR) is most important in a business context.


texts:
I have a requirement where I need build a solution to extract information from multiple pdfs.
The information is stored in tabular format in pdf files. The tables are not very well structured and each pdf document has different format of tables.
The pdfs consist of a lot of pages with text and tables.

I tried Tabula, but it didn't produce a satisfactory result. Thought about using pdfminer as well, but it looks very cumbersome to me.

Can you suggest more ideas and tools to solve this problem?


texts: Clustering/ Classifying users based on sequence of action and time

texts:
I have some user data  where each user has certain pattern of being at different places for some time. I want to create a model which will cluster/classify these users based on these patterns and time spent at each place. So suppose user patterns are like-

Place_1(60 min)- Place_2(30 min)- Place_5(45 min)- user 1 -label(1)
Place_1(60 min)- Place_2(60 min)- Place_5(45 min)- user 2 -label(2)
Place_1(60 min)- Place_2(60 min)- Place_5(40 min)- user 3 -label(2)
Place_2(60 min)- Place_1(60 min)- Place_5(45 min)- user 4 -label(3)
Place_2(60 min)- Place_1(60 min)- Place_5(45 min)- user 5 -label(3)

So they should be clustered/Classified as as-

1- User 1
2- User 2, User 3
3- User 4, User 5

The time duration is continuous, also I already have labels for these patterns so I can do classification as well as clustering. I initially thought of doing kmeans clustering on these patterns but by introducing the duration of stay at each place is messing the clustering up. I am currently using Random forest classifier but results are not as promising. Any help will be highly appreciated.


texts: Visualizing mutual information of each convolution layer for image classification problem

texts: I recently came across this paper where the author has proposed a compression based theory on understanding the layers of a DNN. In order to visualize what was going on the authors showed Figure 2 of this paper which is also shown as a video here. For my image classification problem I want to visualize the mutual information exactly in this format. Can someone kindly explain to me how to calculate this numerically for images passing through conv layers in a convolutional neural network. Do I need to fit a kernel density estimator for a vectorized feature maps at each layer and numerically calculate entropy or is there a simpler way to do this? Thanks in advance


texts:
In Distilling a Neural Network Into a Soft Decision Tree the authors mention

  exponentially decaying running average of the actual probabilities with a time window

but provide no formula for it.

What does it mean speaking formal?


texts: In Bag of Tricks for Efficient Text Classification paper which is popular right now, he calculates prec@1 for the datasets in the experimentation segment. What does that mean?


texts: t-SNE Python implementation: Kullback-Leibler divergence

texts:
t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met.
The creators of t-SNE suggests to use KL divergence as a performance criterion for the visualizations:

you can compare the Kullback-Leibler divergences that t-SNE reports. It is perfectly fine to run t-SNE ten times, and select the solution with the lowest KL divergence [2]

I tried two implementations of t-SNE:

python: sklearn.manifold.TSNE().
R: tsne, from library(tsne).

Both these implementations, when verbosity is set, print the error (Kullback-Leibler divergence) for each iteration. However, they don't allow the user to get this information, which looks a bit strange to me.
For example, the code:
import numpy as np
from sklearn.manifold import TSNE
X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
model = TSNE(n_components=2, verbose=2, n_iter=200)
t = model.fit_transform(X)

produces:
[t-SNE] Computing pairwise distances...
[t-SNE] Computed conditional probabilities for sample 4 / 4
[t-SNE] Mean sigma: 1125899906842624.000000
[t-SNE] Iteration 10: error = 6.7213750, gradient norm = 0.0012028
[t-SNE] Iteration 20: error = 6.7192064, gradient norm = 0.0012062
[t-SNE] Iteration 30: error = 6.7178683, gradient norm = 0.0012114
...
[t-SNE] Error after 200 iterations: 0.270186

Now, as far as I understand, 0.270186 should be the KL divergence. However I cannot get this information, neither from model nor from t (which is a simple numpy.ndarray).
To solve this problem I could:

Calculate KL divergence by my self,
Do something nasty in python for capturing and parsing TSNE() function's output [3].

However:

would be quite stupid to re-calculate KL divergence, when TSNE() has already computed it,
would be a bit unusual in terms of code.

Do you have any other suggestion? Is there a standard way to get this information using this library?
I mentioned I tried R's tsne library, but I'd prefer the answers to focus on the python sklearn implementation.

References
[1] http://nbviewer.ipython.org/urls/gist.githubusercontent.com/AlexanderFabisch/1a0c648de22eff4a2a3e/raw/59d5bc5ed8f8bfd9ff1f7faa749d1b095aa97d5a/t-SNE.ipynb
[2] http://homepage.tudelft.nl/19j49/t-SNE.html
[3] https://stackoverflow.com/questions/16571150/how-to-capture-stdout-output-from-a-python-function-call


texts: Normalize / Standardize in a Random Forest?

texts:
If I have a matrix of co-occurring words in conversations of different lengths, is it appropriate to standardize / normalize the data prior to training?

My matrix is set up as follows: one row per two-person conversation, and columns are the words that co-occur between speakers. I cannot help but think that, as a longer conversation will likely comprise more shared words than shorter ones, I should factor this in somehow.


texts: Orange 3 "Find Informative Projections" and target variables?

texts:
Having issues with the "find informative projections" feature in Orange 3. In order to be able to use this feature in the scatterplots, you have to select a target variable in the "select columns". So I choose the value that I'm most interested in finding correlations to. However, whatever I choose as a target variable doesn't show up in the informative projections list as being correlated to other features in my data set. Why not?

I have a lot of features in my data and was under the impression that the "find informative projections" option finds pairs of features that seem to have a linear correlation and sorts them by which pairs are most correlated.

If that's the case, why do you need a target variable? How do I choose an appropriate target variable?

Am I misunderstanding the purpose of the "find informative projections" feature?

I'm attaching a pic of my work flow.


texts: Transfer learning training accuracy starts from a fixed value

texts:
I am training a transfer learning model using tensorflow.keras library for image classification. I am dealing with 7 classes. I have used several pre-trained models trained on 'imagenet'. For example, I am using Xception and unfreezing last 30 layers. When I replace the last layer with softmax layer, the training looks reasonable. The model summary looks like:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
xception (Model)             (None, 5, 5, 2048)        20861480
_________________________________________________________________
global_average_pooling2d (Gl (None, 2048)              0
_________________________________________________________________
dense (Dense)                (None, 7)                 14343
=================================================================
Total params: 20,875,823
Trainable params: 8,990,679
Non-trainable params: 11,885,144
_________________________________________________________________
Number of layers in the base model:  132
Number of trainable layers in the full model:  3

The training for the first two epochs looks like (The training accuracy starts from a low accuracy (10% or so)):

Epoch 1/12
525/525 [==============================] - 190s 362ms/step - loss: 0.7438 - accuracy: 0.7314 - val_loss: 0.3813 - val_accuracy: 0.8648
Epoch 2/12
257/525 [=============>................] - ETA: 1:28 - loss: 0.4986 - accuracy: 0.8182

Problem

The problem starts, when I add one or more new layers before the final softmax layer. The training accuracy starts from exactly 85.71. It does not matter how many layers I add or whatever base model I use. The training accuracy starts exactly from 85.71, and after the first epoch, I get a very high accuracy (almost 94-95) for validation set. Within a few epoch, the validation accuracy get over 98%. But when I test it on a separate validation set, the performance is actually worse compared to model described earlier.

For example, the model summary is:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
xception (Model)             (None, 5, 5, 2048)        20861480
_________________________________________________________________
global_average_pooling2d (Gl (None, 2048)              0
_________________________________________________________________
dense (Dense)                (None, 1024)              2098176
_________________________________________________________________
dropout (Dropout)            (None, 1024)              0
_________________________________________________________________
dense_1 (Dense)              (None, 7)                 7175
=================================================================
Total params: 22,966,831
Trainable params: 11,081,687
Non-trainable params: 11,885,144
_________________________________________________________________
Number of layers in the base model:  132
Number of trainable layers in the full model:  5

The training epochs looks like:

Epoch 1/12
525/525 [==============================] - 191s 363ms/step - loss: 0.1902 - accuracy: 0.9225 - val_loss: 0.0971 - val_accuracy: 0.9640
Epoch 2/12
 58/525 [==>...........................] - ETA: 2:38 - loss: 0.1351 - accuracy: 0.9470

I change the base model with different models, add/remove layers before the softmax, but the training accuracy always starts from 85.71.


texts: Batch normalization for image CNN - Why not use the mean of the entire batch?

texts:
Question
For CNN to recognize images, why not use the entire batch data, instead of per feature, to calculate the mean in the Batch Normalization?
When each feature is independent, need to use per feature. However the features (pixels) of images having RGB channels with 8 bit color for CNN are related. If there are 256 pixels in R channel in an image, 255 for pixel i and 255 for pixel j are both white meaning the same intensity(?) in R color.
Then why not use the mean of the entire data in a batch? If the pixel channel i happens to have the values between (0, 127) and channel j has (128, 255), the meaning that (0, 127) is within [0, 255] and the relational meaning between i and j, which is, pixel i intensity is lower than that of j) gets lost.


texts: Is there any difference between 'classification' and 'categorization' based on machine learning terminology?

texts: When I was learning about classification models, it came to my mind that if there is any difference between "categories" and "classes" on the basis of machine learning terminology? If there is no difference then why do we say classification models, why not categorization models?


texts: R recommendations while applying ggplot2 and plotly together to plot a graph

texts:
I have data frame having 3 columns namely index, Actual & Predicted (all column are numeric).
I am using 2 packages plotly & ggplot2 for plotting index on x-axis & Actual and Predicted on y-axis.
 The command for plotting the graphs goes as follows-

plot<- ggplotly(ggplot(dataframe, aes(x = index))+ geom_line(aes(y=Predicted), colour= "blue")+ geom_line(aes(y=Actual), colour="red"))

plot

The R was giving the plot smoothly without any recommendations, but suddenly it started giving recommendations like,

We recommend that you use the dev version of ggplot2 with ggplotly()
Install it with: devtools::install_github('hadley/ggplot2')

So I tried installing ggplot2 with devtools as given in the recommendations.

I am able to plot the graph but still I am getting the above recommendation.

What does that recommendation mean and what to do if i am getting the recommendation even after installing it with devtools ?


texts: I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also suited for smaller data sets?


texts: What's the best strategy to train a CNN with images that only have labels for positive characteristics?

texts:
I have a large database of images that are only partially labeled for multiple, non-exclusive characteristics or objects present on them. For instance, an underwater scene might feature the labels water, swimsuit and fish on it.

Problem is it's only partially labeled, meaning that the fact the label cat is absent doesn't mean there's no cat on the image.

What'd be the best strategy to train a CNN able to tell the missing labels on the dataset?

The image base has 230 thousand images on it, but given that multiple labels are possible, it's hard to tell the ratio of images that are fully labeled. I'm able to code python and managed to use keras to train on mnist dataset on gpu.


texts:
I am working on collaborative filtering using matrix factorization in MATLAB. I am using Gradient Descent for parameter learning. The cost function to optimize is :

$ J = {\left\| I \odot (R - U V') \right\|}_{F}^{2} + \lambda_1 {\left\| U \right\|}_{F}^{2} + \lambda_2 {\left\| V \right\|}_{F}^{2} $

I have the following equations for updates :

$ \nabla J_{u(i)} = -\sum_j I_{ij} (r_{ij} - u_i v_j') v_j $

$ \nabla J_{v(i)} = -\sum_j I_{ij} (r_{ij} - u_i v_j') u_i $

The dimensions of the matrices are :

$I  -> n * n$

$R -> n*n$

$U -> n*k$

$V -> n*k$

I am not able to come up with vectorized form of the gradient, so I have to loop over i's and j's explicitly, which is slowing down the code a lot.


texts:
According to "Linear interpolation in latent space" in https://hackernoon.com/latent-space-visualization-deep-learning-bits-2-bd09a46920df and others, the latent space representation of an autoencoder is linear in that if I average out two representations and propogate back to input space, I will get a real looking object with the features of the two.

What proof is there to show the space is indeed linear? What is the intution behind in?


texts: Predict the outcome of a new time series (mailing)

texts:
We survey our members on a regular basis on various topics, but with always the same question (basically, do you like it? yes or no), and we classify them based on what is replied, and how many participated in the survey.

When we send the email, we would like to be able to know at what point in time we can estimate that the final result (how many people will fill, what % of yes) is likely to be the one we have now.

We did process the previous surveys, and we can see a pattern where after 6 or 12 hours, the result is similar to the final one (ie when we wait more than a week after the mailing). Obviously, a survey sent friday at 8pm takes longer to reach a conclusive stage than the ones sent on tue at 9am. They are other factors too, for instance if our mail provider experienced delay delivering, then of course we need to wait longer.

Is there a way to know how long to wait (3, 6, 12... hours?) before we get a solid prediction of the result?

Is there an algorithm that can predict the result at any point in time and give an estimate on how likely the estimate is correct (p value like)?


texts: How to run inference of 2 deep learning models simultaneously on video?

texts:
I want to get inference of 2 models.
First model(Runs at 20fps, Pytorch), Second one is a heavier model(Inference time 1 sec, Tensorflow) on webcam feed.
The first model would be running on every frame, The other model is not required on every frame, Something like 1 in every 50 frames.
I tried to use multiprocessing, But I am stuck on how to return outputs of function. The input to both the models is the same. First model processes the frame and returns the processed frame, The second model processes and returns the string. String needs to be displayed along with the processed frame, And it would be updated after every 50 frames.
I have written a pseudo code below, .start() function does not return the processed output, Need to replace that.
def first_model(frame):
    #Process frame here
    return processed_frame

def second_model(frame):
    #Process frame here
    return string_output

cap = cv2.VideoCapture(0)
i = 0
second_output = "Random text" #Output of second model is a string
while(True):
   _,frame = cv2.read()
   p = multiprocessing.pool(args=(frame,),target=first_model)
   first_output = p.start() #This is not correct
   if(i%50 == 0):
         q = multiprocessing.pool(args=(frame,),target=second_model)
         second_output = q.start() #Again, This is not allowed
   cv2.putText(first_output,second_output,region) #Put second output on every frame, on some predefined region
   cv2.imshow(first_output)
   i = i + 1

```


texts:
I have 800 responses to an open-ended survey question. Each response is categorized into 3 categories based on a list of 70 categories. These categories are things like "stronger leadership", "better customer service", "programs", and etc...

My question is, can I use this as a training data set in order to develop a model that I can use in the future as we get more survey responses? We would like to be able to tag, label, or classify each survey response into (up to) 3 of the 70 categories.

Is this even possible? Or do I have to use a NB with simple words? Can you please guide me to tutorials, examples, etc.?

Using R in this exercise.


texts: What is the difference between affinity matrix eigenvectors and graph Laplacian eigenvectors in the context of spectral clustering?

texts:
In spectral clustering, it's standard practice to solve the eigenvector problem

$$L v = \lambda v$$

where $L$ is the graph Laplacian, $v$ is the eigenvector related to eigenvalue $\lambda$.

My question: why bother taking the graph Laplacian? Couldn't I just solve the eigenvector problem for the graph (affinity matrix) itself, like the guy did in this video?

PS: I made this same question in CrossValidated but I think this is a more appropriate channel. Forgive me if I'm wrong.


texts: Problem with the hub.load function in tensorflow ( TypeError: Autotrack object is not callable)

texts:
I have a problem with loading pretrained module for an NLP task and the problem is because of the tf migration I suppose. Tensorflow website says that the problem might be sorted if the signature variable is given correctly. Can you help me correct this code ?

TypeError: 'AutoTrackable' object is not callable

[code]

import tensorflow_hub as hub
# enabling the pretrained model for trainig our custom model using tensorflow hub
module_url = "https://tfhub.dev/google/universal-sentence-encoder-large/3"
embed = hub.load(module_url)

# creating a method for embedding and will using method for every input layer
def UniversalEmbedding(x):
    return embed(tf.squeeze(tf.cast(x, tf.string)), signature='default', as_dict=True)["default"]


texts: Python - Logistic (Logit) Regression - why am I getting an Endog error?

texts:
I'm running the following code:

X = dataset[['X1 transaction date', 'X2 house age', 'X3 distance to the nearest MRT station', 'X4 number of convenience stores', 'X5 latitude', 'X6 longitude', 'X7 distance to Xindian Ditsrict Office', 'X8 distance to Cardinal Tien Hospital', 'X9 distance to Shih Hsin University']]

y = dataset['Y house price of unit area']

model = sm.Logit(y, X).fit()
print(model.summary())

I'm using a CSV dataframe with information about 414 different residential properties in the Xindian District of Taiwan. My goal is to use a logistic regression to determine which attributes have the greatest effect on the properties' prices.

When running my Logistic Regression, I get an Endog Error, as follows:

  ValueError Traceback (most recent call last)
   in () 2 y = dataset['Y house
  price of unit area'] 3
  ----> 4 model = sm.Logit(y, X).fit() 5 print(model.summary())

  //anaconda/lib/python3.6/site-packages/statsmodels/discrete/discrete_model.py
  in init(self, endog, exog, **kwargs) 402 if
  (self.class.name != 'MNLogit' and 403 not np.all((self.endog>= 0) & (self.endog <= 1))):
  --> 404 raise ValueError("endog must be in the unit interval.") 405 406

  ValueError: endog must be in the unit interval.

Does anyone know why I am getting an endog error, and how to correct this issue? My outcome variable is a continuous integer.

Thank you.


texts: When should we choose agglomerative clustering over K-means clustering?

texts:
I was working on a clustering based model and I read about hierarchical clustering and K-Means clustering.
Under what conditions should I choose agglomerative over K-means clustering?


texts:
I am very very new to tensorflow. I am trying to understand its basics. Based on some examples I created simple script:

import tensorflow as tf
import numpy as np

test_data_size = 2000
iterations = 50000
learn_rate = 0.005

def generate_test_values():
    train_x = []
    train_y = []

    for _ in xrange(test_data_size):
        x = np.random.rand()
        y_f = 8 / x
        train_y.append(y_f)
        train_x.append(x)

    return np.transpose([train_x]), np.transpose([train_y])

train_dataset,train_values = generate_test_values()

a = tf.Variable(tf.random_normal([1]), name="a")
x = tf.placeholder(tf.float32, [None, 1], name="idf")
y = tf.placeholder(tf.float32, [None, 1])
model = tf.divide(a, x)

cost = tf.reduce_mean(tf.square(y - model))
train = tf.train.GradientDescentOptimizer(learn_rate).minimize(cost)

init = tf.global_variables_initializer()

with tf.Session() as session:
    session.run(init)

    for _ in xrange(iterations):
        session.run(train, feed_dict={
            x: train_dataset,
            y: train_values
        })

    print "cost = {}".format(session.run(cost, feed_dict={
        x: train_dataset,
        y: train_values
    }))

    print "a = {}".format(session.run(a))

it delivers simple data to predict y=1/x function. Unfortunately it does not work all a get is:

cost = nan a = [nan]

Though it is working with linear model y=ax.

Can anyone explain me why it is not working.


texts:
I have multilabel problem.

I was using onevsrestclassifier and now i want to use onevsoneclassifier.
from sklearn.svm import SVC
classifire = SVC(gamma='auto')
accuracy = []

skf = StratifiedKFold(n_splits=10,random_state=None)
kf = KFold(n_splits=5)

skf.get_n_splits(X_train,y_train)

for train_index,test_index in kf.split(X_train,y_train):
    X1_train,X1_test = X_train.iloc[train_index],X_train.iloc[test_index]
    y1_train,y1_test = y_train.iloc[train_index],y_train.iloc[test_index]
    clf = OneVsOneClassifier(classifire)
#     clf = OneVsRestClassifier(classifire)
    clf.fit(X1_train,y1_train)
    prediction = clf.predict(X1_test)

    score = accuracy_score(prediction,y1_test)
    print(score)

But i am getting error only in OneVsOneClassifier
<ipython-input-47-91b351c58c24> in <module>
     11     clf = OneVsOneClassifier(classifire)
     12 #     clf = OneVsRestClassifier(classifire)
---> 13     clf.fit(X1_train,y1_train)
     14     prediction = clf.predict(X1_test)
     15
..
ValueError: bad input shape (19051, 206)

I am new here. Can you help me ?

print(X1_train.shape)
print(y1_train.shape)
print(X1_test.shape)
print(y1_test.shape)

o/p
(19051, 875)
(19051, 206)
(4763, 206)
(4763, 875)

UPDATE
accuracy = []

skf = StratifiedKFold(n_splits=10,random_state=None)
kf = KFold(n_splits=5)

skf.get_n_splits(X_train,y_train)

for train_index,test_index in kf.split(X_train,y_train):
    X1_train,X1_test = X_train.iloc[train_index],X_train.iloc[test_index]
    y1_train,y1_test = y_train.iloc[train_index],y_train.iloc[test_index]
    clf = OneVsOneClassifier(classifire)
#     clf = OneVsRestClassifier(classifire)
    print("X1_train",X1_train.shape)
    print("y1_train",y1_train.shape)
    print('X1_test',X1_test.shape)
    print('y1_test',y1_test.shape)
    clf.fit(X1_train,y1_train)
    prediction = clf.predict(X1_test)

    score = accuracy_score(prediction,y1_test)
    print(score)

And the output is
X1_train (19051, 875)
y1_train (19051, 206)
X1_test (4763, 875)
y1_test (4763, 206)

So X1_train and test have same columns. But the error is same


texts:
I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number of clusters I expect.

I read the sklearn documentation of DBSCAN and Affinity Propagation, where both of them requires a distance matrix (not cosine similarity matrix).

Really, I'm just looking for any algorithm that doesn't require a) a distance metric and b) a pre-specified number of clusters.

Does anyone know of an algorithm that would do that?


texts: Neural-Networks - preferred method for training, classification v.s. regression

texts:
As a conclusion of their paper "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) (§10 Discussion and Conclusion), LeCun and others conlude that the preferred method for training [a] network should be picked as follow :

If the training set is large and redundant and if the task is classification, use stochastic gradient or the stochastic Levenberg Marquardt method.
If the training set is not too large, or if the task is regression, use conjugate gradient.

I was quite surprised about this conclusion because, from §6. to §9. they describe those algorithms, and discuss their performance, mostly trough the lens of eigenvalues of the Hessian. There is no real discussion about regression v.s. classification, except for the end of §6.2 on conjugate gradient: "Clearly on large and redundant (classification) problems stochastic backpropagation is faster".

Can someone point how that is "clear" for classification (but not regression) ? Doesn't that rely on some important assumptions (balanced classes, classes separability, de-correlation of variables) ?


texts: Can you approximate the empirical probability of a data-point from a trained Variational Autoencoder?

texts: I'm interested in if it's possible to use a trained VAE that when you pass a specific data-point through it, the elbo can help approximate to the data-points empirical probability. If this is incorrect is there a different way of obtaining such an approximation from a VAE?


texts: Q-value estimate in neural episodic control

texts:
Disclaimer: I'm not that familiar with reinforcement learning, so I might lack some basic knowledge on that topic.

I was reading the neural episodic control (NEC) paper by Pritzel et al., 2017 and I'm a bit unclear on the Q-value estimates. Eq. 3 says:

$Q^{(N)} = \sum_{j=0}^{N - 1} \gamma^jr_{t+j} + \gamma^N\text{max}\underset{a'}Q(s_{t+N}, a').$

The $\text{max}\underset{a'}Q(s_{t+N}, a')$ term is found by querying all the memories in the differentiable neural dictionary (DND) for each action and taking the max. This estimate is then added to the replay buffer:

$(s_t, a_t, R_t),$

where $R_t = Q^{(N)}(s_t, a)$

If I understand the paper correctly, once the Q-value estimate is added to the replay buffer the only way to remove it is by overloading the buffer limit, meaning that outdated estimates are continuously being used in the training process. The question is: isn't it better to re-evaluate the Q-value estimate at each iteration? If, instead of storing the Q-value in the buffer we estimated it on the fly from the current state of DND it would perhaps help the model converge even faster?


texts: Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

texts:
Hopefully, you don't mind me posting a question here instead of the regular stack exchange forum. I'm learning ML basics and practising pandas.

assuming the data frame is called df, column name = column name

Suppose I want to replace some 'dirty' values in the column 'column name'. There are "not known" values in this column that mean nothing so i would like to replace them with the mode.

df['columnname'].mode()

returns

0   dog
dtype: object

this code below replaces the "not known" values as NaN rather than the mode.

df.loc[df.index[df['columnname'] == "not known"].tolist(),'columnname'] = df['columnname'].mode()

what am i missing? or how should i fix the code?


texts: Recommender system based on purchase history, not ratings

texts:
I'm exploring options for recommender systems optimized for the insurance industry, which would take into account

i) product holdings

ii) user characteristics (segment, age, affluence, etc.).

I want to stress that

a) there are no product ratings available, thus collaborative filtering is not an option

b) recommended products don't have to be similar to products that have already been purchased, thus item-to-item recommendations are most probably not relevant.

Keep in mind that in insurance you rarely want to recommend similar products to those already purchased ones, as someone with the Car insurance is unlikely to want to buy another Motor product, rather Home or maybe Travel, etc.

That's why I want to develop recommendations on similarities between the users based on their purchase history and/or demographics

Ideally, I'd like to be able to implement it in R, if not possible, then in Python. Thanks for help and suggestions!


texts: R: Rates of change from an initial value

texts:
I have a collection of csvs and must produce yearly rates of change per group within each csv, as well as a rate of change compared to the initial value.

I am using the function below to calculate yearly rates of change, and it works fine through my loop.

func <- function(x, n=1) {
  c(rep(NA, n), diff(x, n) / head(x, -1*n)*100)
}
 df$RateOfChange <- ave(df$Data, factor(df$Group), FUN=func)

Year    Group   Data    RateOfChange
2010    1a      3.5     NA
2011    1a      4       14.29
2012    1a      3.8     -5.00
2013    1a      4.1     7.89
2014    1a      4       -2.44
2010    1b      2       NA
2011    1b      2.3     15.00
2012    1b      2.4     4.35
2013    1b      3       25.00
2014    1b      1.8     -40.00
2009    2       10      NA
2010    2       12      20.00
2011    2       13      8.33
2012    2       15      15.38
2013    2       14      -6.67
2014    2       11      -21.43

However, I also would like to calculate the percentage change in each year compared to the initial value, like so:

Year    Group   Data    BaselineChange
2010    1a      3.5     NA
2011    1a      4       14.29
2012    1a      3.8     8.57
2013    1a      4.1     17.14
2014    1a      4       14.29
2010    1b      2       NA
2011    1b      2.3     15.00
2012    1b      2.4     20.00
2013    1b      3       50.00
2014    1b      1.8     -10.00
2009    2       10      NA
2010    2       12      20.00
2011    2       13      30.00
2012    2       15      50.00
2013    2       14      40.00
2014    2       11      10.00

Surely there is a way to do this. It'd be easy enough in Excel for a single csv, but I have several hundred, each with multiple groups. I have tried dplyr::diff and dplyr::lag, but I can't get the pipeline to abide by the group variables in the way that other questions posted on this site do.


texts: How to visualize mass spectrometer data with plotly?

texts:
Mass spectrometers output analyzation data usually in form of a mass chromatogram. I need a visualization of the output data. The x-axis represents the retention time, the y-axis represents the signal intensity and the z-axis represents the mass frequency. I need the signal intensity vs. retention time (y vs. x) and the signal intensity vs. mass frequency (y vs. z). Consider this short introductional video about how the visualization should look like.

The data may be represented as profile or centroid data. But w.r.t. data representation this does not make any difference because the measurement samples are still discrete w.r.t. z-axis (mass frequency).

I'd like to use plotly Dash to visualize the output data in a dashboard. However I could not find a suitable 3D chart example. How do I get the visualization I am aiming for?


texts: Understanding why in deep reinforcement learning correlations in the data reduce the effectiveness

texts:
From the paper

  Human-level control through deep reinforcement
  learning, Mnih et al. Nature 2015

It says

Reinforcement learning is known to be unstable or even to diverge
when a nonlinear function approximator such as a neural network is
used to represent the action-value (also known as Q) function 20 .
This instability has several causes: the correlations present in the
sequence of observations

I am unsure how to understand this and cannot create any hypothetical examples where this can happen. What are some hypothetical scenarios, or real examples where the correlations present in the sequence disrupt the use of a 'deep learning' approximator is used?


texts: Which is a better method Lasso regression or sequential Foward selection?

texts:
I have a huge data of nearly 200 features and several of them might be correlated? In order to select best features which is preferred Lasso regression or sequential forward selection considering the correlation between features.(I can not run pca on this because feature transformation would transform features which cannot be interpreted).

Few things I understood were: Lasso is really quick and sequential
forward selection is very slow. But I am not sure how would lasso
perform under correlated features i.e. I feel that the weights may be
distributed among features(assume necessary features) which are
correlated and because of low weights due to sharing Lasso might
suggest they are irrelevant features.


texts: Use cross entropy to create decision tree classifier

texts:
Are entropy and cross-entropy the same thing as per basic definition?

If there is a difference:

Decision tree splits take on entropy or Gini index, can we use cross-entropy to split decision trees?
OR should I use it as an evaluation metric after running the decision tree algorithm?

Also, does the decision tree algorithm assumes any distribution?

If yes,

then how can we use the KL Divergence metric?

I am just trying to link a few concepts in a broader view.

These are my concerns with respect to the multiclass decision tree.

Thanks,


texts: Are Undergraduate Statistics Concepts Used in Practice?

texts:
I'm curious for more experienced Data Scientist, have you ever used t - test, ANOVA, Wilcoxon, etc?

Basically my question is, do you perform inference task, or purely prediction tasks? (Machine Learning)


texts: How to calculate the mini-batch memory impact when training deep learning models?

texts:
I'm trying to calculate the amount of memory needed by a GPU to train my model based on this notes from Andrej Karphaty.
My network has 532,752 activations and 19,072,984 parameters (weights and biases). These are all 32 bit floats values, so each takes 4 bytes in memory.
My input image is 180x50x1 (width x height x depth) = 9,000 float 32 values. I don't use image augmentation, so I think the miscellaneous memory would be only related to the mini-batch size. I'm using a mini-batch size of 128 images.
Based on Andrej's recommendation, I get the following memory sizes:
Activations: 532,752*4/(1024^2) = 2.03 MB
Parameters: 19,072,984*4/(1024^2) * 3 = 218.27 MB
Miscellaneous: 1289,0004/(1024^2) = 4.39 MB
So the total memory to train this network would be 224,69 MB.
I'm using TensorFlow and I think I'm missing something. I haven't run the training yet, but I'm pretty sure (based on past experiences) that the memory in use will be much higher than what I've calculated.
If for each image in the mini-batch, TensorFlow keeps their gradients so it can normalize them later for a single weights/biases updates step, then I think the memory should take into account another 532,752 * 128 values (gradients for each image in the mini-batch). If that is the case, then I'd need more 260.13 MB to train this model with 128 images/mini-batch.
Can you help me understand the memory considerations for training my deep learning model? Are the above considerations right?


texts: Unsupervised Learning with audio recordings

texts:
I had a (probably crazy) idea for a project and I was wondering if you all think it would be in any way possible. I'm interested in analyzing sounds made by different types of animals (for example bird songs or the croaking sounds made by different species of frog) and looking for relationships between the sound and other factors like habitat or taxonomic classification. If I could obtain digital recordings, would it be possible to feed the data into a clustering algorithm in such a way that it could identify and compare important characteristics? My fear is that most algorithms would prioritize extraneous details related to the quality of the recording or the type of compression over potentially meaningful factors like pitch, timbre, and pattern.
A Google search uncovered a few possibly related articles:
This one proposes a k-medioids approach but seems to focus on computer engineered sound waves rather than real life recordings.
This one uses a hierarchical algorithm and has a lot of good discussion on data cleansing and extracting "Low-level descriptors" to use as potential model features, however the focus is on classifying music which is obviously a much richer feature space than what I'm considering.
I have very limited knowledge in this area and would have to do a lot more research on data prep, feature creation, model selection, and so on but just wanted to ask if this is completely insane before I go too far down this rabbit hole.


texts: How do we predict what is in an image using unsupervised deep neural networks?

texts:
From my understanding of unsupervised DNNs for image classification:

The input layer is a 4,096 dimension vector (for 64 x 64 images)
The hidden layers represent much lower "features" as identified by the back propagation
As the model is generative, the output layer is also a 64 x 64 image

Therefore, how do can we make a prediction that a new unseen image contains a specific image class (e.g. cat) if we lack labelled data?


texts: A good way to use facial landmark as model input

texts:
We are planning to use facial landmark information as input to the model. Since there are more than 60 points, it doesn't look good to use 60 channels as inputs after one-hot encoding. I found a few papers with similar ideas, but I didn't like them.
What is the exact name of this methodology, and is there a well-known good way?


texts: Gini impurity in decision tree (reasons to use it)

texts:
In a decision tree, Gini Impurity[1] is a metric to estimate how much a node contains different classes. It measures the probability of the tree to be wrong by sampling a class randomly using a distribution from this node:
$$
  I_g(p) = 1 - \sum_{i=1}^J p_i^2
$$
If we have 80% of class C1 and 20% of class C2, labelling randomly will then yields 1 - 0.8 x 0.8 - 0.2 x 0.2 = 0.32 Gini impurity value.
However, assigning randomly a class using the distribution seems like a bad strategy compared with simply assigning the most represented class in this node (in above example, you would just label C1 all the time and get only 20% of error instead of 32%).
In that case, I would be tempted to simply use this as a metric, since it is also the probability of mislabeling :
$$
  I_m(p) = 1 - \max_i [ p_i]
$$
Is there a deeper reason to use Gini and/or a good reason not to use this approach instead ? (In other words, Gini seems to over-estimate the mislabellings that will happen, isn't it ?)
[1] https://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity

EDIT: Motivation
Suppose you have two classes $C_1$ and $C_2$, with probabilities $p_1$ and $p_2$ ($1 \ge p_1 \ge 0.5 \ge p_2 \ge 0$, $p_1 + p_2 = 1$).
You want to compare strategy "always label $C_1$" with strategy "label $C_1$ with $p_1$ probability, and $C_2$ with $p_2$ probability", thus the probability of success are respectively $p_1$ and $p_1^2 + p_2^2$.
We can rewrite this second one to:
$$
p_1^2 + p_2^2 = p_1^2 + 2p_1p_2 - 2p_1p_2 + p_2^2 = (p_1 + p_2)^2 - 2p_1p_2 = 1 - 2p_1p_2
$$
Thus, if we substract it to $p_1$:
$$
p_1 - 1 + 2p_1p_2 = 2p_1p_2 - p_2 = p_2 ( 2p_1 - 1)
$$
Since $p_1 \ge 0.5$, then $p_2 ( 2p_1 - 1) \ge 0$, and thus:
$$
p_1 \ge p_1^2 + p_2^2
$$
So choosing the class with highest priority is always a better choice.

EDIT: Choosing an attribute
Suppose now we have $n_1$ items in $C_1$ and $n_2$ items in $C_2$. We have to choose which attribute $a \in A$ is the best to split the node. If we use superscript $n^v$ for number of items that have a value $v$ for a given attribute (and $n^v_1$ items of $C_1$ that have value $v$), I propose we use the score:
$$
\sum_v \frac{n^v}{n_1 + n_2} \frac{max(n^v_1, n^v_2)}{n^v_1 + n^v_2}
$$
As a criterion instead of Gini.
Note, since $n^v = n^v_1 + n^v_2$ and $n_1 + n_2$ doesn't depend on the choosen attribute, this can be rewritten:
$$
\sum_v max(n^v_1, n^v_2)
$$
And simply interpreted as the number of items in the dataset that will be properly classified.