texts:
Just out of curiousity, is it generally a good idea to reduce the dimension of training set before using it to train SVM classifier?

I have a collection of documents, each of them is represented by a vector with tf-idf weight calculated by scikit-learn's tfidf_transformer. The number of terms (feature?) is close to 60k, and with my training set that consist of about 2.5mil of documents, it makes the training process to go on forever.

Besides taking forever to train, the classification also wasn't accurate most probably due to the wrong model. Just to get an idea of what I am dealing with, I tried finding a way to visualize the data somehow. And I decomposed the document matrix into a (m, 2) matrix using SVD with scikit-learn (wanted to try other methods, but they all crashed halfway).

So this is what the visualization looks like

So is it generally a good practice to reduce the dimension, and then only proceed with SVM? Also in this case what can I do to improve the accuracy of the classifier? I am trying to use sklearn.svm.SVC and kernel='poly', and degree=3 and it is taking a very long time to complete.


texts: How can I estimate user-item purchase probabilities of a e-commerce website?

texts:
I am writing my Master thesis, where the goal is to estimate user-item purchase probabilities. In other words, for a given user, what is the probability he/she will buy a certain item. I have session logs (click sessions on items of users) and buys from an e-commerce website.

I found the evaluation to be very hard. Therefore, my approach was the following:

1 Predict the probability that a user will buy something in the next session, given the current session (and earlier sessions).
Using this, I can filter out people that have a reasonable probability to buy something at all.

2 I want to take the top 50 items (recommended/ranked by the recommender system currently in place) and from these items, I want to estimate the probability. (As estimating this from all items would be impossible to evaluate at all).

Something else I could do is only looking at the items that a user have clicked on, and estimate those probabilities.

The biggest problem is how to evaluate these probabilities?

Can someone help me define the problem, or give me some hints/tips on how to proceed? If this would not be possible, I have to find another (related) topic to my thesis.


texts:
I want to make a predictor using Keras LSTM model. I have a sequence of places visited. The task is to predict the last destination.
I went through different examples but it seems I am not able to shape the input properly.
I am stuck on how to prepare in my program my data to give them to the LSTM model.
Here is a minimum code related to my problem.
input_csv ='input.csv'
max_features = 6
df = pd.read_csv(input_csv)

df.head()
#Cafe =0.1, Park =0.2, Shop =0.3, Home=0.4, Movie = 0.5, School=0.6

For example, Person A  visited
cafe (0.1) ->   park(0.2) ->    cafe (0.1) ->   park(0.2)   and finally ended at school(0.6). The desired task is to predict y (end) based on input X (place1,place2,place3,place4).
X = df.iloc[:,1:5].to_numpy()
X_train = X.reshape(6, 1, 4) # X.reshape(samples, timesteps, features)
X_train.shape
#(6, 1, 4)

y = df.iloc[:,-1].to_numpy()
y.shape
#(6,)

#Building model
model = Sequential()
model.add(LSTM(6, dropout=0.2, recurrent_dropout=0.2, input_shape=(None, 1)))
model.add(Dense(max_features, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Train
model.fit(X_train,y)
pred = model.predict(X_train)
predict_classes = np.argmax(pred,axis=1)

ValueError: Error when checking input: expected lstm_4_input to have shape (None, 1) but got array with shape (1, 4)
I would appreciate if someone could help me in clearing my confusion or pointing to some explanations. Thank You.


texts: Need help understanding Hard SVM quadratic program equation

texts:
This is from the textbook "Understanding Machine Learning" by Shalev-Schwarz p. 169.

Can anyone help me understand why the solutions to this optimization problem need to be divided by the norm of w?
(Other resources online mention this translation to quadratic programming, but omit the division part.)
Thank you very much!!


texts: Deep Learning ROC and Average Precision Curve Results

texts:
I used Vgg16 to create a deep learning model and the dataset is imbalanced so, I used class_weight argument in fit_generator method.

The model result as the following:
accuracy= 98.9% and loss= 0.1731

And I got the following ROC and Average Precision curves:

Why I got these results?
Is there any problem in ROC and average precision?


texts: How many video streams can single GPU handle for object detection

texts:
I need to detect objects from multiple video streams at realtime (or close to it, like 10 FPS).
How many GPUs do I need to detect objects using YOLOv3 or MobileNet for, say, 10 video streams?
Is it possible to use CPU or something else?
I don't need an exact number. I just need to understand scalability perspective and costs per single stream.


texts: Churn Prediction Model based on Customer Activities

texts:
I am new to data science, so forgive me if i have not done my research well.

I want to build a system that calculates the churn scores for each customer and hence try to prevent it. I just want to test the concept, so result is not really important.

The business was about sending newsletter to customer and receive ad revenue for it.

The data
We have customer activity like open emails, read emails, click emails, and that is about it.

The idea is to look into the customer activity and see if we can identify if the customer is about to cancel the subscription.

I have look into several classification models and wonder if there is a model that can look into the activities history, not just the current events.

Any help will be appreciated!


texts:
I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding.

I have this question because I’m tackling a NLP task using attention layer. I believe it should be very easy to learn (the most important part is to learn alignments). However, my neural networks only achieve 50% test set accuracy. And the attention matrix is weird.
I don’t know how to improve my networks.

To give a example:
English: Who are you?
Chinese: 你是誰？

The alignments are
‘Who’ to ‘誰’
‘are’ to ‘是’
‘you’ to ‘你’

How does attention learn that?

Thank you!


texts: ValueError: Shapes are incompatible when fitting using ImageDataGenerator

texts:
I got this error ValueError: Shapes (None, 1) and (None, 3) are incompatible when training my Sequential model. I could not figure out which shapes are actually incompatible. This is the first time I do image classification. Here are my codes:

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3)))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(3, activation='softmax'))

model.summary()

model.compile(optimizer='rmsprop',
  loss='categorical_crossentropy',
  metrics=['accuracy'])

# this throws value error
model.fit(
  train_generator,
  steps_per_epoch=25,
  epochs=20,
  validation_data=valid_generator,
  validation_steps=5,
  verbose=2
)

I want the last Dense layer to return an array of 3 probability scores (large, medium, small).

Here is how I created the train_generator and valid_generator:

from tensorflow.keras.preprocessing.image import ImageDataGenerator
image_datagen = ImageDataGenerator(
  rescale=1./255,
  rotation_range=20,
  horizontal_flip=True,
  shear_range = 0.2,
  fill_mode = 'nearest')

index = len(df)//10
df_train = df[index:]
df_valid = df[:index]

train_generator = image_datagen.flow_from_dataframe(
  df_train,
  x_col = 'png_image',
  y_col = 'target',
  target_size=(150, 150),
  batch_size=4,
  color_mode = 'rgb',
  class_mode='sparse',
)

valid_generator = image_datagen.flow_from_dataframe(
  df_valid,
  x_col = 'png_image',
  y_col = 'target',
  target_size=(150, 150),
  batch_size=4,
  color_mode = 'rgb',
  class_mode='sparse',
)

The dataframes look like this:


texts: How to train with cross validation? and which f1 score to choose?

texts:
I got similar results in 2 models which consists of similar algorithms.
Model 1 with cv=10 has a f1'micro' of 0.941. See code below.
Model 2 only train test split (no cv) has f1'micro' 0.953.
Now here is my understanding problem. Before I did a Grid-Search to find best hyperparameters. Now I would like to do just a cross validation to train the dataset. Like the red marked in the picture. In the code there is still the Grid Search inside.
Question 1:
Is this code doing that what I want? (is that a cross validation to train the dataset?)
Question 2:
when I have 2 models like in the picture, model 1 with cross validation (red marked) and model 2 with train validation test data -> what are the reasons to choose model 1 with cross validation and why?

X = df.drop('columnA', axis=1)
y = df.columnA

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.20, random_state=42)

xgb_params = {
    'max_depth': [6],
    'learning_rate': [0.3]}

xgb_clf = GridSearchCV(XGBClassifier(random_state=42),
                      xgb_params,
                      cv=10,
                      n_jobs=-1,
                      verbose=True)

xgb_clf.fit(X_train, y_train)

xgb_pred = xgb_clf.predict(X_test)
print(accuracy_score(xgb_pred, y_test))
print(f1_score(xgb_pred, y_test, average='micro'))

Im sorry if my point of view is strange, but I have a lack of knowledge and Im confused about Cross Validation and Kfold and how to use it.


texts: How is the "base value" of SHAP values calculated?

texts:
I'm trying to understand how the base value is calculated. So I used an example from SHAP's github notebook, Census income classification with LightGBM.
Right after I trained the lightgbm model, I applied explainer.shap_values() on each row of the test set individually. By using force_plot(), it yields the base value, model output value, and the contributions of features, as shown below:

My understanding is that the base value is derived when the model has no features. But how is it actually calculated in SHAP?


texts: 2D Z-score/Mahalanobis distance that includes a penalty for uncertainty

texts:
I have some 2D points and I want to assess their performance against the target point.
When I was doing this in 1D, I took the Z-score Z = (x- mu)/sigma, but that didn't really penalise clouds of points that were wide but centred around the correct answer, so I started using the 'log posterior', which is the derivative of the Gaussian distribution,
log_posterior = -(((data - np.mean(samples))/np.std(samples))**2)/2 + np.log(1/(np.sqrt(2*np.pi)*np.std(samples)))

but now I'm working in 2D, and the parameters are correlated, so I've replaced the Z-score part with the Mahalanobis distance, and at the moment, I've put the log of the determinant of the covariance matrix as the second term - but it's this second term I am unsure of, especially as I have no 2 pi etc. What should I really be using here? I have:
new_log_posterior = distance.mahalanobis( np.array([data[0],data[1]]) , np.mean(data_points,axis=0), scipy.linalg.inv(np.cov(data_points.T)))  + np.log(np.linalg.det(scipy.linalg.inv(np.cov(data_points.T))))


texts:
I am very new to machine learning, and I am trying to build a model to classify this data set (UCI heart disease). I have built a simple model using random forest and got an 80% accuracy. The size of the set is about 900. To get the best value possible, should I continue with random forest or go with deep learning (such as TensorFlow)?

Thanks!


texts: Does Gradient Boosting perform n-ary splits where n > 2?

texts: I wonder whether algorithms such as GBM, XGBoost, CatBoost, and LightGBM perform more than two splits at a node in the decision trees? Can a node be split into 3 or more branches instead of merely binary splits? Can more than one feature be used in deciding how to split a node? Can a feature be re-used in splitting a descendant node?


texts: customer segmentation with categorical variables

texts:
I was adviced to write in this group regarding my question about modeling categorical database.
I have a customer dataset, which is a survey result. I have 1595 obs. and about 200 columns(200 because most of the cases the questions were multiple choice and we had to split it into columns). Majority of variables are categorical or binary. I do not have continous variables at all. My task is to do customer segmentation, clustering. There is no initial assumptions although as I have also the questionnaire so can logically seperate the important questions.

I face several issues regarding the modeling

I need to validate the choice of variables i use
I am trying to find associations, pairwise associations and trends, as I do not have initial assumtions who can be my segments
Clustering models are not working good for categorical variables and the ones I tried for example kmods, ignore the associations, correlations and return me not clear picture.

Can you please suggest how to approach, or from where to start. I am new in data analytics and I need some hints to go on with the analysis and I will be grateful to have some guidance at least high level what can be done.

Thanks in advance!


texts: Strategies for automatically tuning the hyper-parameters of deep learning models

texts:
I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and suggestions/critiques on my proposed strategy below. I realise that experimentation may be the best way forward but I don't want to unnecessarily duplicate somebody else's work.

I know that a popular approach is to assume that the function of model error wrt. HPs is a bayesian process, the parameters of which are updated as new points are evaluated (ie. an expensive full training of the underlying deep learning model) so that a more rapid convergence on the optimal HPs can be achieved by quickly predicting better candidates for the next HPs to be evaluated than a simple grid search would be able to. Ilija Ilievski et al report more efficient convergence by using an RBF surrogate model (with pySOT) instead of bayesian https://github.com/ilija139/HORD.

Given that changes in some HPs (let's call them cheap) eg. ADAM initial learning rate and learning rate decay won't affect training time wheras doubling some HPs (let's call them expensive) eg. SGD batch size or total number of steps will double training time, does it make sense to approach this problem by trying to quickly refine cheap HPs whilst the object function is cheap(er) to evaluate and then allowing the object function to become more expensive gradually as the expensive HPs inflate?

One way of implementing this startegy might be to hold expensive HPs at relatively low values whilst refining cheap HPs and then allowing expensive HPs to grow and then iterating this process until some precondition is met, ie. model-error no longer goes down or processor-time budget is exceeded.

An alternative and possibly more 'organic' approach that I imagine would effectively achieve something similar would be to give each evaluation of the object function a certain processor time budget that would gradually increase. So when the object function over-ran its budget, it would be (gracefully) stopped and whatever error result that model gave, would be taken alongside whatever HPs were used (with number of steps being the actual number of steps taken, rather than what was requested). The idea being that this would pressure the HP surrogate to rapidly refine the cheap HPs initially whilst allowing the expensive HPs to be refined over time.

Does this make sense?


texts: Machine Learning and stability of correlations

texts:
I need some help improving my understanding. As far as I know many (if not all) ML algorithms assume time stationarity, that means that the sampled distribution should not change over time.
On a bigger dataset that includes multiple days I see the correlatios to change significantly. For example during first hour Feature A and B have a correlation of 0.8. On second hour  their correlations drops to 0.2 or also goes to negative values.
Is not this also a problem? Is not assumed that the correlation of features should not change (at least not drastically) as more data comes into play?
If I am correct on that can you please provide me some more references to polish a bit my understanding but also go much deeper?
Thanks a lot your great help.
Regards
Alex


texts: Theoretical question around multiclass classification

texts:
Assume there's a customer dataset with monthly installments repaying goods purchased in advance. The objective is to build a model predicting who is going to be bad debtors and who can afford to buy more products.
One approach would be to build multiple binary models: one for bad debt, one for upselling X, one for upselling Y, etc. Then run the bad debt first, and then, on those predicted as good debtors, run all the upsells models and make a final decision with the aggregate of the probability outcomes.
Another approach could be to train one model with multiple classes, bad debt, can afford up to X, can afford up to Y, etc.
In my research found unclear and conflicting results. Some researchers suggest multiclass classification works best but other researchers and practitioners suggest that breaking the final goal into simpler models is best.
What are your thoughts? Has any worked on or encountered a similar problem?


texts: Can I specify the root node splitting feature in XGBoost?

texts: Just what the title says. Suppose I know the feature that I want to be used for splitting for the root node in a tree model in XGBoost; is there a way for me to tell XGBoost to split on this feature at the root node?


texts: Is there any model agnostic way to calculate the weight importance for neural networks given a set of inputs?

texts:
I was curious if it's possible to calculate what weights are important and what weights are redundant (or have high redundancy) for separate tasks in neural networks?
And if this is doable in a model agnostic way (maybe also differentiable..)?


texts: Regression - How Random Forest and Gradient Boosting really works?

texts:
I read a lot about random forest and gradient boosting, but I do not know how these two algorithms really work.

For example, see the simple picture about basketball (picture 1) from this link:

How does Random Forest and how does Gradient Boosting work?
Has each tree in the random forest different trainings data AND different features?

Sorry about m question but I don‘t find an easy non-mathematical answer.


texts: Should I scale/normalize the data before training a feedforward neural network using only lagged values?

texts:
So I am trying to forecast the price of a certain commodity that is quite volatile. Now I wanted to train a regression feedforward neural network to predict the price of next week.

Just to be clear, I only use lagged values of this commodity's price (including seasonality lags etc.).

Now I read a lot of papers and posts that data scaling/normalizing is very important when you have different features measured in different units (quite logical). However, in my situation all the prices are in euros and there are no external features. I already looked at the first differences of the prices to have values between -1 and 1, apart from computation time the results weren't that different.

Is it still needed to scale/normalize the data?


texts: Use Tfrecord to feed multi-input neural network

texts:
I have a dataset on Tfrecord that includes images and tabular data, I want to feed those data into a mixed neural network data. I saw examples of how to use multi-input neural network by using numpy arrays
model.fit([X_images,X_tabular],[Y_images,Y_tabular])

but transforming Tfrecord dataset into a numpy array by iterating slows down the process.
Is there a way to train it directly on the Tfrecord data. So far I have a model that train only an images and this is the code that I am using
tfrecords_schema = {
    'id': tf.io.FixedLenFeature([], tf.string),
    'y_label': tf.io.FixedLenFeature([], tf.int64),
    'image': tf.io.FixedLenFeature([], tf.string),
    'x_1': tf.io.FixedLenFeature([], tf.float32),
    'x_2': tf.io.FixedLenFeature([], tf.float32),
    'x_3': tf.io.FixedLenFeature([], tf.int64),
    'x_4': tf.io.FixedLenFeature([], tf.int64),
}
def _parse_function(example_proto):
    parsed_example = tf.io.parse_single_example(example_proto, tfrecords_schema)
    return parsed_example['image'], parsed_example['y_label']

def _parse_error(image, label):
    return image != b''

def _preprocess_function(image, label):
    image = tf.image.decode_jpeg(image)
    image = tf.image.resize(image, (299, 299))
    label = label_decoder(label)
    return image, label

train_data = (
    train_data
    .map(_parse_function)
    .filter(_parse_error)
    .map(_preprocess_function)
    .apply(tf.data.experimental.ignore_errors())
    .batch(BATCH_SIZE)
)
validation_data = (
    validation_data
    .map(_parse_function)
    .filter(_parse_error)
    .map(_preprocess_function)
    .apply(tf.data.experimental.ignore_errors())
    .batch(BATCH_SIZE)
)
AUTOTUNE = tf.data.experimental.AUTOTUNE

train_data = train_data.prefetch(buffer_size=AUTOTUNE)
validation_data = validation_data.prefetch(buffer_size=AUTOTUNE)
def make_model(image_shape, num_classes):

    inputs = tf.keras.Input(shape=image_shape)
    x = tf.keras.applications.inception_v3.preprocess_input(inputs)

    base_model = tf.keras.applications.InceptionV3(input_shape=IMAGE_SHAPE, include_top=False, weights='imagenet')
    base_model.trainable = False
    x = base_model(x, training=False)

    global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
    x = global_average_layer(x)
    x = tf.keras.layers.Dropout(0.3)(x)

    prediction_layer = tf.keras.layers.Dense(num_classes)
    outputs = prediction_layer(x)

    return tf.keras.Model(inputs, outputs)
model = make_model(IMAGE_SHAPE, NUM_CLASSES)

learning_rate = 0.0001

model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=learning_rate),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"]
)

callbacks = [
    tf.keras.callbacks.ModelCheckpoint(filepath="tf/pvc_data/tmp/weights.hdf5", verbose=1, save_best_only=True)
]
history = model.fit(
    train_data, epochs=1, callbacks=callbacks, validation_data=validation_data
)


texts: Decision tree regression: Polynomials unnecessary?

texts:
I am testing out different models for a regression task.
When using OLS, Ridge and Lasso, I use different polynomial degrees of the explanatory variables.
Example: For two variables x and y, degree 2 would give the explanatory variables x, x^2, xy, y, y^2.

When using decision tree, however, I am not sure whether it makes sense to use any any higher degrees than 1 as explanatory variables. Example: Does it make sense to test for x^2, xy and y^2 when applying a decision tree regressor?

The reason I ask, is that the decision tree regressor is a non-linear regressor. On the one hand this could perhaps be an arguement it not making sense to include higher order polynomials, as the decisoin tree already can deal with non-linearity.


texts: How to see actual numbers in a hist on R?

texts:
Just as an example I'm using iris data. I have a histogram, which is nice, but the distribution is weird, and I'd like to see what the bars represent:

Here's a histogram:

And I'd like to see the high/low that each bar represents. For instance, the first bar has a frequency of 34, but a width of something in the range of the lowest third of 0.0-0.5.

Obviously there won't be a width of 0.0, so what is the width range? How can I show it on the histogram?


texts: Common Techniques to Generate from a Regression Neural Network Model

texts:
I am used to train neural networks that are designed for generation, such as GANs or VAEs.

I am wondering what are the common techniques to generate data that would minimize the target/energy learned by a regression model, following the idea of Deep Dream.

I can think of two ways :

1) Use the trained regression neural network as a loss function (with its gradients) for another neural network that is trained to produce structures that produce a given energy / target, as given by the first neural network.

2) Use an standard optimization algorithm (not a neural network) to find which inputs minimize the output of the regression model.

Are there any other common methods to do this ? What are the most know / effective methods ?

Any idea / reference would be great !


texts: How do I identify the postulated 600 object classes in OpenImages v. 6?

texts:
I have downloaded the OpenImages version 6 image sets (train, validation, test) from CVD Foundation as per instructions from the OpenImages homepage. The OpenImages homepage states that there are 600 boxable object classes in the dataset.
When I sift through the entire list of training set bounding box annotations from https://storage.googleapis.com/openimages/v6/oidv6-train-annotations-bbox.csv, it turns out 599 unique label names. So, that is one short of the supposed 600 classes.
If I download the boxable class names from https://storage.googleapis.com/openimages/v5/class-descriptions-boxable.csv (this is linked from the OpenImages version 6 page despite its apparent version 5 origin), this list contains 601 classes.
It seems neither of the above approaches identify the correct 600 classes, so how do I correctly identify the "official" 600 classes?


texts:
I'm using a python script to log IO of a grid job.
the log is formatted like this:

timestamp;fullpath;event;size
1526994189.49;/tmp/folder/;IN_ISDIR;6
1526994189.49;/tmp/folder2/File;IN_ACCESS;36

Those files are millions of line long. I'm using Spark to generate graphs and detect anomaly in job IO.
But before doing that I need to insert the job ID and the jobname to the column making the file :

timestamp;fullpath;event;size;jobid;jobname
1526994189.49;/tmp/folder/;IN_ISDIR;6;123456;afakejobname
1526994189.49;/tmp/folder2/File;IN_ACCESS;36;123456;afakejobname

The thing is I'm new to big Data technologies and I would like to know if using parquet format is it better to put both jobname and jobid
or knowing that I have only 15 different jobname and jobid in the same log is it better to convert it on the fly using SparkSQL and make a join to a very small table with just jobname;jobid and put only the jobid in my log.


texts: How to find a confidence level given the z value

texts:
I have a task from which I would like to find the confidence level given the z value. I have a sample population. From that population, given its distribution, I would like to find the confidence level of a given value of that population. In other words, given a a value of the population, I would like to know if it is within 95% (confidence level) of the whole population or 68% or 50% and so on. Usually, we can find the z value and confidence interval and given confidence level as explained here How to find the confidence Interval. But I would like to find the confidence level given the z value (which in this case is a given value from the population).

How can I tackle this? If possible it should be in python or in R


texts: I was recently trying to train a convolutional neural network to classify people as Hispanic or white (for learning purposes). I couldn't find a good dataset of just those two races, so I had to manually scrape images from the web. I ended up with 48 total images, 24 of them Hispanic and 24 white. When I trained my network, I tried many different architectures and hyperparameters, but the accuracy stayed at exactly 50%. I learned this was because the network was outputting 1 every time, classifying every image as Hispanic because it found a local optima for the cost function. I believe this is because of my 50/50 data split, although I may be wrong. If if is because of the split, how does this not happen in the real world? For example, if I had 10 million images, half of them white and the other half Hispanic, how does the network avoid falling prey to the same trap?


texts: What ML algorithm can I use for building a "recommended" list for players?

texts:
We have a large touchscreen kiosk in local malls where people can go up to it and play a game under different categories. We want to implement ML to build a recommended list of games for each player. For now, let's assume there are 5 different games under each of the 4 categories and the games can be either available or unavailable at any given time. Also, there is a table containing a log of each play and a table of all the games.

Ideally, we would want to pass 2 different data sets; example: a list of all the plays in the last 30 days with any necessary data, and a list of all the currently available games to play with the necessary "relatable" data. Out of this, we would then end up with a list of the top 10 recommended games for each player based on all that data. There will be more data than just the game's category to tie it together to the player/person, but I want to get a simple demo working first.

I am guessing I need to look into some type of recommendation system but I'm not sure which would best to try out. For now, we are looking at using Python; AWS's ML tool will be too costly. I am open to recommendations on other languages/tools.


texts: What is the difference between topic modeling and clustering?

texts: I know that topic modeling and clustering are related, but not similar techniques. Can anyone suggest what are the main differences?


texts: Best python library for training using Hidden Marov model with Gaussian Mixture

texts:
I would like to train my data using HMM- GMM (Baum Welch approach with gaussian Mixture) to find the best parameters suited for my data.

Note : My data is continuous and not discrete.

I tried with hmmlearn from scikit learn, but i believe it is not supporting continuous HMM-GMM model, but i tried with discrete data, it woks fine.

I tried to use pomegranate, but i cannot able to understand the document, also i am not sure whether it uses GMM to fit my data.

I try to install hmms libraray, but i receive an error while installing,

  Command "python setup.py egg_info" failed with error code 1 in
  C:\Users\eponr\AppData\Local\Temp\pycharm-packaging\hmms\

Collecting hmms
  Using cached https://files.pythonhosted.org/packages/e9/e4/c070c44ec8a391f6d5501316d1ed7615058f1fd365ff4ed65c9636d0bf62/hmms-0.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\eponr\AppData\Local\Temp\pycharm-packaging\hmms\setup.py", line 22, in <module>
        ext_modules=cythonize(glob.glob('hmms/*.pyx'), language_level=3, include_dirs=[numpy.get_include()]),
      File "C:\Users\eponr\PycharmProjects\untitled1\venv\lib\site-packages\Cython\Build\Dependencies.py", line 955, in cythonize
        c_options = CompilationOptions(**options)
      File "C:\Users\eponr\PycharmProjects\untitled1\venv\lib\site-packages\Cython\Compiler\Main.py", line 563, in __init__
        raise ValueError(message)
    ValueError: got unknown compilation option, please remove: include_dirs

Kindly suggest some libraries in python, which describes my problem ?


texts: Scikit learn: which regressors natively support multi-target regression?

texts:
The docs on sklearn.multioutput.MultiOutputRegressor state that it implements a strategy for extending regressors that do not natively support multi-target regression.

I'm interested to know: which ones do natively support multi-target regression ? (and/or how can I find out?)

based on this article, I assume:

ExtraTreesRegressor
KNeighborsRegressor
LinearRegression
RidgeCV

All support this? (unless I'm getting confused and a 'multi-output estimator'  is not the same thing as 'a regressor that natively supports multi-target regression'; I'm new to this!)


texts: Inverse Autoregressive Spline Flow Implementation

texts:
I have to implement an algorithm for a university project, however I can not seem to wrap my head around it. The algorithm should be an inverse autoregressive normalizing flow using splines. It should work for 1D data (time series) and not for 2D images. It is inspired by this paper https://arxiv.org/pdf/1606.04934.pdf. In particular what is important for me is this algorithm:
I dropped the part with the EncoderNN and for the AutoregressiveNN i am using a pixelCNN (implementation follows). In addition I have a function that can calculate z and the log probability given the W, H and D. Also I am sampling the initial z in a differently, however, that is not what's important.
class PixelCNN(nn.Module):
    def __init__(self, c_in, c_out ,n_layers, K=3, B=8 ,kernel_size=7,n_classes=3):
        super(PixelCNN,self).__init__()

        ''' PixelCNN part '''
        self.init_layer = MaskedConv1d(c_in,c_out, kernel_size,mask_type='A')
        self.layers = nn.ModuleList([])
        for _ in range(n_layers):
            self.layers += nn.ModuleList([MaskedConv1d(c_out,c_out, kernel_size//2,mask_type='B')])
            self.layers += nn.ModuleList([nn.Tanh()])

        ''' Splines part '''
        self.K = K
        self.B = B
        self.n_transforms = n_transforms
        self.network = GatedResNet(c_in,c_out,3 * K - 1)

        ## Make rational quadratic splines
    def forward(self,x):
        x = self.init_layer(x)
        for layer in self.layers:
            x = layer(x)
        return self.create_splines(x)

    def create_splines(self,x):
        out = self.network(x)
        W,H,D = torch.split(out,self.K,dim=-1)
        W = torch.softmax(W, dim=-1)
        H = torch.softmax(H, dim=-1)

        W = W * 2 * self.B
        H = H * 2 * self.B

        D = F.softplus(D)
        return W,H,D

The AutoregressiveNN in the algorithm above has to output the parameters m and s which are used to improve z. In my case if I am not mistaken those should be the splines, so the forward method in the pixelCNN returns the splines only (I am not sure if this is correct). Also I do not know what to do with the loop for for t <- 1 to T do. Since each loop outputs m,s then should I also output a set of splines in this manner? I've seen some implementation of IAF using MAF without splines and sometimes they just make a function forward and backward and then they swap their places in training and sampling. Is that also possible here and is it a better option?
And finally, how should I update z with the splines? More clarification might be needed for people to answer. I can provide that if I missed something.


texts: How do I get started with machine learning and image recognition?

texts:
I'd like to get started with machine learning, specifically image recognition. I know that Python is the most popular language for ML since it's easy to pick up and there's tons of libraries for it. I have a few questions:

What are the advantages of using a pre-trained model vs training my own?
Where would I find pre-trained models? At what point would it be worth it to train my own models, and how can I do so?
Which library should I be using as a beginner? TensorFlow, scikit-learn, etc.


texts: How to create new feature based on clustering result

texts:
I'm still a beginner in machine learning and I want to know how to code this situation based on python and machine learning (clustering).

I have data like:

id      Column1    duration(seconde)    column3
1       aaa        20                   bbb
2       ccc        01                   ddd
3       eee        150                  fff
4       ggg        25                   hhh

I want to group my data according to the duration column value and create new column containing a category name based on duration cluster.  I want to get this result:

id      Column1    duration(seconde)    column3      NewColCategorie
1       aaa        20                   bbb          Cat2
2       ccc        01                   ddd          Cat1
3       eee        150                  fff          Cat3
4       ggg        25                   hhh          Cat2
5       iii        175                  jjj          Cat3


texts: Change values of consecutive repeating letters in pandas column

texts:
I've got one Dataframe like:

id value block
1 a 1
2 a 1
3 a 2
4 a 2
5 b 3
6 c 4

And I want to change the value column to the next value based where
the series changes. Like below. The change must be defined by the
block column.

id value block
1 a 1
2 a 1
3 b 2
4 b 2
5 c 3
6 None 4

I thought about using shift but there are different lengths of continuity in the value column.

I think this should work somehow by using cumsum to detect the change.


texts: Which output of a BiLSTM layer should be used for classification

texts:
I am trying to implement a BiLSTM layer for a text classification problem and using PyTorch for this.
self.bilstm = nn.LSTM(embedding_dim, lstm_hidden_dim//2, batch_first=True, bidirectional=True)
lstm_out, (ht, ct) = self.bilstm(embeddings)

Now, in some examples (I came across in internet) people are passing ht through Linear layer and generating output. Some people are using lstm_out for the same. Now I have two questions -

For a text classification which one should be used?
If I use ht, for bidirectional=True it is of the shape (2, m, n) - but we need (1, m, n) - what is the best way to convert it?


texts: Regression yields much smaller standard deviation and the mean is off, what could be wrong?

texts:
I'm modeling a regression problem. An initial attempt yields the following:

labels.mean(): 0.00018132978443886167
labels.std(): 0.013450786078937208

predictions.mean(): 0.0005549060297198594
predictions.std(): 0.00430255476385355

As you can see, the mean is off, and the standard deviation is totally different. I wonder what does it indicate?

My guess: does it mean that my features are not discriminative enough, so that the model see examples w/ positive and negative labels alike, hence the small variance in the output?

I'm running the regression using XGBRegressor, with early-stopping. I have 1M training examples, 100K validation examples (for early-stopping), and another 100K for testing purpose (for which the mean and the standard deviation are shown above).

I also checked that the label distribution of the three sets are mostly basically the same.


texts: Data augmentation / feature extraction on pre-trained convnets

texts:
I'm reading 'Deep Learning with Python' by François Chollet, which is an excellent book. He talks about using pre-trained convnets (in his example, VGG16) and then running smaller datasets to tweak them (eg the dogs vs cats kaggle dataset).

My understanding is that there are two options:

1) Lop off the dense layers from (eg) VGG16. Run the small dataset (just cats and dogs) through the remaining convolutional layers. Take the output and then train a second, dense, convnet, as a separate, second step, on that.

2) Lop off the dense layers from VGG16. Replace them with new, randomly initalised, dense layers that take the input from the original convnet. Freeze the pre-existing convolutional part of the network, and then train the whole convnet with the smaller dataset.

According to the book, approach (2) allows you to use data augmentation but approach (1) doesn't.

I don't understand why that is the case, or indeed why the two approaches are different, since both are in effect taking the output from fixed, convolutional layers and then training a smaller, dense network on it. What have I misunderstood?


texts: How do I setup a server in the cloud for machine learning?

texts:
As datasets and the number of parameters get larger, it becomes increasingly difficult to run validation locally because of limited disk size and computing power.  As a result, people might use an old laptop or a server in the cloud.  In particular, I'm interested in learning more about the second option.

For the setup, it sounds like I will need a storage system and a computing system.  Is this S3 and EC2?  Redshift and Amazon ML?  Combinations from some other vendor?  At the moment, I'm asking as an amateur participant for Kaggle, but I'd also be curious to know what the pros use.

On the machine itself, is there any way to interact with a GUI rather than a command line to set this up? Would I have to install Anaconda or some other Python distribution before getting started?  Is there a specific file structure to use? What are the main pitfalls to watch out for?

Ultimately, I am looking for the practical advice to set everything up.  If you're able to just offer links to documentation, that would be immensely helpful as well.  Thanks!


texts: How to preprocess raw data in the form RDF triples to perform Information Gain and Principal Component Analysis?

texts:
I have my dataset in the form RDF triples in various domains such as Movie, music etc.

Data is in the form of RDF triples (Subject, Property,Object (all uris). A sample input in the movie domain is displayed below.

As far as my understanding, We need data in the form of a matrix for PCA and we need to classify data into different labels for Information gain. I need to automatically fetch relevant features(properties) from all the properties available.say 50 out  150 features.  It would be really great if someone could help me with more insight into pca and information gain and how to perform these machine learning algorithms based my raw rdf triples dataset.


texts: Predict Best Branch Locations For a Company

texts: I have a dataset of existing 156 branches of my company with the longitude and latitude of each branch. Now we want to open 10 more branches. How can we predict the best locations for opening the new branches using machine learning? As much as I have searched on google, I have gone through by using ARCGIS geo-spatial data or using openstreetmap but couldn't find all the steps properly. Therefore, Kindly guide me as I am a junior data scientist in my company and its my task to deliver the best locations.


texts: Basic method of optimizing hyperparameters

texts:
I recently read the LIPO blog post on the dlib blog:
http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html

It mentions that it can be used for optimizing hyperparameters of eg metaheuristic algorithmsike simulated annealing or genetic algorithms.

I looked for info on how optimizing hyperparameters work in general and the Wikipedia page is the most informative I found but it doesn't answer my basic questions:
https://en.m.wikipedia.org/wiki/Hyperparameter_optimization

My question is just: what is the basic idea for optimizing hyperparameters?

If I have some problem I'm trying to solve with simulated annealing, I know that the starting temperature and the cooldown rate are important in determining how well the algorithm does at finding a solution.

I know that I could completely run the algorithm with one set of parameters, modify one of the parameters, completely run it again, then reset the parameters and modify the other parameter and run it again. This could give me a numerical gradient that I could use to modify the parameters via gradient descent.

However... At this point I had to run the whole algorithm 3 times just to get a single modification of the hyperparameters.

I feel like I must be missing something obvious because optimizing the hyperparameters would take many many hundreds or thousands of times or more the cost of running the whole thing once, which doesn't seem useful at all. Can someone clue me in?


texts: Understanding the multidimensional-nature of the data being fed to a RNN and its output

texts:
Assuming we have a time-series dataset whose window_size = 30 and the batch_size = 4, which makes the overall input = 4*30 (2D). But as RNN expects 3D input, tf.expand_dims is used to make it a 3D input (as per the lecture, new inut becomes 4*30*1, where the last dimension is 1 as the example deals with a univariate time-series).
What I don't get is that what does adding a dimension mean? Eg. what will be the element [0,0,0] of the input?

Also in keras, the typical format for fitting is

model.fit(input, output, epochs=400)

But in an RNN sample code for time-series data, I found

model.fit(dataset, epochs=400)

where  dataset is a tf object containing the time-series data.
Why is the input and output not given explicit  for the model to train in case of the first code? The timestamp is already included in the input in a way(in the 4*30*1 input, the 2nd dimension is supposed to be time-stamps), but how does the keras know against what output labels the input has to be trained?


texts: Why `max_features=n_features` does not make the Random Forest independent of number of trees?

texts:
Consider the following simple classification problem (Python, scikit-learn)

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

def get_product_data(size):
    '''
    Given a size(int), sets `log10(size)` features to be uniform
    random variables `Xi` in [-1,1] and an target `y` given by 1 if
    their product `P` is larger than 0.0 and zero otherwise.
    Returns a pandas DataFrame.
    '''
    n_features = int(max(2, np.log10(size)))
    features = dict(('x%d' % i, 2*np.random.rand(size) - 1) for i in range(n_features))
    y = np.prod(list(features.values()), axis=0)
    y = y > 0.0
    features.update({'y': y.astype(int)})
    return pd.DataFrame(features)

# create random data
df = get_product_data(1000)

X = np.array(df.drop(df.columns[-1], axis=1))
y = df['y']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33,
                                                        random_state=1)

def predict(clf):
    '''
    Splits train/test with a fixed seed, fits, and returns the accuracy
    '''
    clf.fit(X_train, y_train)
    return accuracy_score(y_test, clf.predict(X_test))

and the following classifiers:

foo10 = RandomForestClassifier(10, max_features=None, bootstrap=False)
foo100 = RandomForestClassifier(100, max_features=None, bootstrap=False)
foo200 = RandomForestClassifier(200, max_features=None, bootstrap=False)

Why does

predict(foo10)  # 0.906060606061
predict(foo100)  # 0.933333333333
predict(foo200)  # 0.915151515152

give different scores?

Specifically, with

max_features=None, all features are selected for each tree
bootstrap=False, there is no bootstrap of samples
max_depth=None (default), all trees reach the maximum depth

I would expect each tree to be exactly the same. Thus, regardless of how many trees the forest has, the predictions should be equal. Where is the tree's variability coming from in this example?

What further parameters would I have to introduce in the RandomForestClassifier.__init__ in such a way that foo* have all the same score?


texts: Google Sheets: how to find max value in column B, corresponding values in column A, and max among these

texts:
I have two numeric columns, A and B. I want to find the max value in column B, which will return multiple rows. Then I want to find the max value in column A from among these rows.

A   B
 5  315
 7  315
10  275

The function should return:

 7  315

It would be fine if this were a combination of two functions - one that finds the max value in B (315), and another that finds the corresponding max value in A (7). That's how I'm trying it right now using:

=index(A2:B10,match(max(B2:B10),B2:B10,0))

But this solution only finds the latest row that has column B's max value:

 5  315

Someone suggested I use the query function, but I couldn't get that to work either.


texts: Whats an explanation of PCA that is intuitive for someone in senior leadership who doesn't have a technical background?

texts:
I've been pulled onto my first Data Science project at work. Classic problem of predicting sales based on web traffic data, etc. While I don't know about the specific techniques I will be using in my model, I think that PCA will probably be tried at some point, and if I use it successfully as a preprocessing step before I chug the data into some estimator, I'm thinking about how I might go about defending/explaining my work.

My question is, what is an intuitive explanation for PCA that doesn't rely heavily on the linear algebra that makes up the process? Perhaps an analogy that draws a parallel to something that any layman can readily understand?


texts:
-Hi Experts-

I just read about factorise() function in Pandas. Using this I'm able to encode (enumerate) my string values into numbers. But, now I'm not able to understand what  numbers corresponds to what string.

Ex.

df['product_name'] # Ex. A, B, C

df['product_name'] = df['product_name'].factorize()[0]
df['product_name'] # Ex. 0, 1, 2

Just illustration, not actual o/p -

A - 0
B - 1
C - 2

How can i get this? Please advise.

-Curious newbie :)


texts: How do I use a custom stopwords filter in the Java Weka API?

texts: I am using the Java Weka API to build a classification model. I can use the builtin stopwords filter. However, I need to use a custom filter for my problem. I do not know how to use a custom stopwords filter in the Java Weka API.


texts: Find starting point and stopping point of noisy pulse

texts:
I have noised pulse signal like these. I need to identify starting point and stopping point of each peak. First and third pictures have 2 pulses with noise. Second picture has single pulse with noise.
Suppose I have about 100 graphs of them. Is it possible to use machine learning to identify starting point and stopping point?


texts: What do you pass for the cv parameter in the sklearn method cross_val_score

texts:
In sklearn, there is a method for cross validation called cross_val_score. One of the parameters of this method is 'cv'.

I understand in cross validation, there is no splitting the data into training and validation (70-30 split). Instead, you split the data into 'k' subsamples, then train it on the K-1 subsamples and validate using the kth sample.
And repeat it for each of the 'k' subsamples.

So is this cv = k, i.e the number of subsamples in which you split the training data?


texts: CUDA_ERROR_OUT_OF_MEMORY: out of memory. How to increase batch size?

texts:
I have one GPU: GTX 1050 with ~4GB memory.
I try Mask RCNN with 192x192pix and batch=7. I got an error:  CUDA_ERROR_OUT_OF_MEMORY: out of memory
I found this
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

  File "<ipython-input-2-0806c9f06bd0>", line 3
    session = tf.Session(config=config, ...)
                                       ^
SyntaxError: positional argument follows keyword argument


texts: Using TF Dataset API to process sequences for stateful RNN

texts:
I am trying to use the TensorFlow (v1.13) Dataset API to save and load long sequences for a stateful RNN.

Basically, lets say I have n_seq sequences, each fixed to a length of 120.

I want to load batches to train my stateful RNN according to this post, basically:

           batch 1         batch 2         batch 3         batch 4

element0    s21             s22              s23             s24   ...
element1    s11             s12              s13             s14

where sij denotes j-th window of the i-th sequence. This is done based on how a stateful RNN works in TF, as when stateful=True, "the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch." (source)

Initial Solution

I am saving each sequence to its own *.tfrecord file. Then I am loading them using:

tf_dataset = tf.data.Dataset.list_files(path, shuffle=False, seed=1)
tf_dataset = tf_dataset.interleave(
    lambda filename: tf.data.TFRecordDataset(filename).map(parse_func),
    cycle_length=BATCH_SIZE,
    block_length=1
)
tf_dataset = tf_dataset.batch(batchsize)

This actually mostly works EXCEPT for the final files. Namely if n_files % BATCH_SIZE != 0, the finale files will be together in the same batch.

Is there a better way to handle this?

NOTE: This question was previously asked on stackoverflow.coom, but deleted due to lack of response. I think this will be a better home for it.


texts: How important is the channel order in deep-learning computer vision tasks?

texts:
I stumbled across this question while working with OpenCV, which stores color images in BGR order in memory, while most other libraries I know of use RGB order.

How important is this difference? Would I be able to use BGR images for an RGB-trained network? Obviously, a fire truck is red in most regions, and not blue. But don't CNNs look for texture rather than simple colors?

Even assuming there is a difference - could it still make sense to apply channel permutations as a kind of data augmentation during training? This would spread the spectrum of textures from each original channel to both others, again, the assumption being that colors are not all that important.

I would be curious to know if research like this already been done - I would be surprised if it had not.


texts: Best or recommended R package for logit and probit regression

texts:
Could somebody please recommend a good R package for doing logit and probit regression? I have tried to find an answer by searching on Google but all the links I find go into lengthy explanations about what logit regression is, which I already know, but nobody seems to recommend an R package.

Thanks in advance.

Jerome Smith


texts: IDE alternatives for R programming (RStudio, IntelliJ IDEA, Eclipse, Visual Studio)

texts:
I use RStudio for R programming. I remember about solid IDE-s from other technology stacks, like Visual Studio or Eclipse.

I have two questions:

What other IDE-s than RStudio are used (please consider providing some brief description on them).
Does any of them have noticeable advantages over RStudio?

I mostly mean debug/build/deploy features, besides coding itself (so text editors are probably not a solution).


texts: How to incrementally train xgb with additional features?

texts:
I found this thread similar to my question:
How to reach continue training in xgboost
However, I'm looking for a way to incrementally train my xgb model with updated features (col_num increased with old features still there but new features appended). I cannot choose other models so xgb is the only choice.


texts: Neural Networks: Predicting probabilities of the possible values of y, instead of just predicting y

texts:
I have a true value y that I'd like to predict with a regression, but I'm interested in the probabilities that y will be different values. Y is theoretically continuous but in the dataset it is rounded to integers. Let's say y could be 0-9. I want 10 probabilities, one for each possible value. I tried doing this categorically, with a neural network having 10 output nodes, this hurts the predictions since we lose the relationships between categories, 1 is closer to 2 than it is to 10.

Example toy problem:

y is the weight of an object in pounds. The dataset has Y values rounded s.t. y can be from 0 to 9 pounds. Predict the probabilities that y will be 0-9 pounds based on the features X.

Example output: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1,] (ten # summing to 1)

I'd like to be able to accomplish this with Keras.


texts: How to create a big data frame in Python

texts:
I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error.
I tried
pd.DataFrame(X.toarray(), columns=tokens)
and
pd.read_csv(X.toarray().astype("float32"), columns=tokens, chunksize=...).
And it seems that when I convert $X$ to a numpy array using  X.toarray(), I get an error.
Can someone tell me what is an easy solution for this? Is there anyway I can create a sparse dataframe from $X$ without memory error?
I have been running my codes on Google Colab Pro and I think it provides me less than 100 GB Ram.


texts: Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture?

texts:
I am doing a few experiments on medical data. I am about to transfer learn the pretrained networks for my problem.

Firstly, I have to pick a network architecture.
Secondly, I would like to optimize it's parameters/parameters of optimizer, to get better performance.

I would like to pick the network architecture based on 10-fold cross validation of several architectures. I will perform cross validation in a way that I have data split to train:test in a 80:20 manner, then train is split into 10 splits. Test set shouldn't ever change. Based on the cross validation, I would like to pick a model I would optimize further on validation set.

Is it okay to test the best architecture on test data too, to see whatever I am moving in a right direction, before optimizing the parameters or am I cheating?

The dataset is imbalanced with relatively high class variance, so I am not even sure whatever the 2 folds for testing will trully represent the dataset.

Thanks


texts: Is DQN limited to working with only image frames?

texts:
I have few questions about Deep Q Network.

Does DQN only accept image frames as input? I have never hear (read) a paperwork where it doesn't use image frames.
If the first is a No, then does image frames as input for training faster than any other options? For example, in any atari game, is frames input faster to train while having the same perfomance quality?

I know an image frames is but numbers too, like other datas, but from what I know even a low quality frame is considered huge.


texts: Scikit-learn OneHotEncoder effect on feature selection

texts: If I need to run feature selection on my dataset isn't it problematic to use OneHotEncoder? Couldn't it then decide to remove a one of the encoding columns? How should I deal with this? Thank you.


texts: I am currently using an one class classification svm and I am trying to boost the classification results by employing more than one svm-occ with varying gamma parameters and combine these decisions with majority voting. But the results are not that satisfactory , since with a careful choice of gamma parameter I can get better results with just one-class svm. So , I was just wondering if I can employ also another one-class classifier e.g. SVDD and then combine its decisons with svm-occ . But the problem is my project is being developed in python enviroment and I can't find an implementation of SVDD with sklearn. So is there any suggestions which classifiers to use and if an according python implementation exists or how to combine them to obtain in overall better classification results ? (**I am keeping the nu parameter on svm-occ fixed)


texts: How can I change the loss function when the shape of my data changes?

texts:
Since my data is too large, I use pd.read_csv('',chunksize=). I am using categorical_crossentropy as my loss function, however, on the last chunk, I have just one target. So I get the error:

  You are passing a target array of shape (2110, 1) while using as loss
  categorical_crossentropy.

Now I know I can use binary_crossentropy.So this is what I did:

X_train, X_test, y_train, y_test = train_test_split(train_data, train_labels, shuffle=True, test_size=0.3)
if y_train.shape[1] == 1:
   loss = 'binary_crossentropy'
else:
   loss = 'categorical_crossentropy'

When I do this, I get the error:

  IndexError: index 1 is out of bounds for axis 0 with size 1

My data is one-hot encoded. How can I resolve this error? Thanks


texts: Why is my time series model predicting strange results?

texts:
I am trying to predict some time-series data. The output data predicts two numbers (one that's usually greater than 1 and another that is usually less than 1). I've plotted about 800 samples where the first number is on the x-axis and the second number on the y-axis. This is just to see the diversity of the data.
As seen below the actual output data is pretty scattered albeit with more data closer to 1 for both numbers. However, the predicted output shows an interesting pattern, where samples that have the first number to be much higher than 1 also predicts the second number to be much lower than 1. I'm not sure if this is under or overfitting.
If underfitting I would expect the predictions to be clustered in a blob close to the mean or median output (around 1.2,0.9) but instead there's a pattern to the predictions. I have no idea what's going on and any insight would be greatly appreciated.
Actual Output

Predicted Output

If interested, here's my keras model with Adadelta optimizer and 'mean_absolute_error':
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_1 (Conv1D)            (None, 124, 100)          32800
_________________________________________________________________
batch_normalization_1 (Batch (None, 124, 100)          400
_________________________________________________________________
p_re_lu_1 (PReLU)            (None, 124, 100)          12400
_________________________________________________________________
dropout_1 (Dropout)          (None, 124, 100)          0
_________________________________________________________________
cu_dnnlstm_1 (CuDNNLSTM)     (None, 124, 80)           58240
_________________________________________________________________
dropout_2 (Dropout)          (None, 124, 80)           0
_________________________________________________________________
batch_normalization_2 (Batch (None, 124, 80)           320
_________________________________________________________________
cu_dnnlstm_2 (CuDNNLSTM)     (None, 30)                13440
_________________________________________________________________
dropout_3 (Dropout)          (None, 30)                0
_________________________________________________________________
batch_normalization_3 (Batch (None, 30)                120
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 62
=================================================================


texts: Detect blur image using ssdmobilenet and tensorflowlite

texts:
I have clear images of cards vs blurry images of card. My task is to capture photo when the image is not blurry, as you can see from the description I need this code to run in real time on android device.
I have done some background reserarch on this topic 'Identify blurry image'. And found out few interesting solutions.

Apply opencv transforms such as laplace or sobel filter. The blurry image will have less edges. And then using techniques such as SVM to find out which image has less edges
Use other opencv transforms similar to sobel to get edges of images and then find image with less edges.

Although these transforms produce good output. They are badly slow. I need something which has speed similar to tflite object detection using android. Taking this logic in my mind my obvious step was to annotate images 1500(blurred cards) vs 1500(non blurry cards) and retrain ssdmobilenet model using tensorflow object detection api.
My dataset
(clear)

(blur)

However when I exported the trained model to android I completely messy output, it appears as though the model has not learned anything from the data. My question is Is this problem solvable using object detection api as I mentioned above ? if yes where am I going wrong ? If no what are the fastest alternatives to detect blur in real time


texts:
I'm learning R and trying to do a comparison of MLB player salaries vs. WAR

One dataset contains all the salaries and the other contains WAR information, and they have their variables labelled differently. For example, each of the team names in one set is an abbreviation (ATL, NYY, HOU vs. Braves, Yankees, Astros). With that example, I'm sure I could find a way to make ATL == Braves, but with all the players that doesn't seem so easy.

The players are labelled in one dataset as the first 5 letters of the last name, first two letters of the first name, and then two numbers to differentiate matches. The other set simply has the names. An example would be Bryce Harper vs. harpebr03.

There's no apparent way to sort via alphabetical order and matching up each player's variable ID would take way too much time. Is there an easier way to link these differently named variables in R?


texts: Neural Network - distinguishing between several normalized values is impossible?

texts:
It's a common practice to normalize inputs to the neural Network.

Let's assume we have a vector of activations.

One of techniques, the Layer Normalization simply looks at the vector's components,  re-centers this activated vector from $\mu$ to zero, then divides by the standard deviation $\sigma$

How is it then possible to distinguish activations [1,2,3,4] from [4,5,6,7]  if both will be re-centered to the same vector [-1.5, -0.5,  0.5, 1.5f]  and then divided by std deviation? Also, I can see such a problem when merely normalizing the input-state-vectors for any Neural Net.

Edit:

there seems to be a hint in first half of page 4 in the paper, however due to my weakness in Maths I can't comprehend it :(

Edit after accepted the answer:

guys, don't forget that Layer Norm (and Batch norm) both have Gain and Bias terms. If the network performs poorly, the gain is tweaked to undo division-by-std-deviation, and tweaking the bias is used to undo the shift (the re-centering). This allows some neurons to indeed pay attention to scaling and shifting, when really required.


texts:
What are recommended books or online courses that are mostly helpful to someone wanting to delve into machine learning specifically in audio?

Or is the learning path mostly the same as for any other machine learning field?


texts: Searching for movie dataset containing movie synopses?

texts:
To build a hybrid recommendation system, I used the movielens 1M dataset, for the collaborative filtering part. Now, I'm looking for a database/dataset that contains descriptions/summaries/details/synopses/plots of movies for the content-based recommendation.
Is there someone who could help me and tell me where I can find a such dataset?
thank you in advance.


texts: Is the adjusted R square good for the interpretation prediction accuracy of machine learning regression model and how to interpret it?

texts:
Is the adjusted R square good for the interpretation of machine learning regression (SVM, Random Forest, Elastic Net, etc) model prediction accuracy?
How to interpret machine learning regression prediction accuracy model with adjusted R square?


texts: Understanding the mathematics of AdaGrad and AdaDelta

texts:
I have been building some models for a project, but I can't wrap my head around the math of Adagrad and Adadelta algorithms.

I do understand how vanilla gradient descent works and I have written code for making it work successfully.

I will be grateful if anyone explain these two things to me or provide some resource to understand them.


texts: Make a random forest estimator the exact same of a decision tree

texts:
The idea is to make one of the trees of a Random Forest, to be built exactly equal to a Decision Tree.

First, we load all libraries, fit a decision tree and plot it.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
import random
from pprint import pprint
import pdb
random.seed(0)
np.random.seed(0)
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

data = load_iris()

dtc = DecisionTreeClassifier(random_state=0)
dtc.fit(data['data'].squeeze(),data.target)

tree.plot_tree(dtc)

We then do the same thing with the random forest

rf  = RandomForestClassifier(n_estimators=1,max_features=None,random_state=0)
rf.fit(data['data'].squeeze(),data.target)
tree.plot_tree(rf.estimators_[0])

My question:

Is it possible to make the exact same the first tree of the random forest and a decision tree?


texts: Why is mutual information symmetric, meaning why does I(A,B) = I(B,A)? Isnt the definition of mutual information, I(A,B), something like "the reduction of entropy in A when given B"? P(A|B) doesnt equal P(B|A) right?


texts:
I found a question (Question 7) here:

Question: For k cross-validation, larger k value implies more bias
 Options:  True or False

My answer is: True.
Reason:
Larger K means more folds means smaller test set which means larger training set. As you increase training data you bring down variance which means increase bias.
So as K increases --> Training data size increases --> Variance reduces --> Bias increases
Hence answer is True
But the website says answer is False.
Can someone explain if my logic is wrong and why their answer is right?


texts: ML Models: How to handle categorical feature with over 1000 unique values

texts:
I am trying to build an ML Classification model on a data set that contains quite a few categorical columns. However, few of them have over 1000 unique values. I am concerned that if I run one-hot encoding or pandas get dummies on them, it will simply result in too many features to work with.

So, I tried to find the top N unique values that account for, say, 90% of the underlying data and group the rest of them under say, 'Other' or 'miscellaneous'. But that's making the 'Other' or 'miscellaneous' value as the most prominent one. I am concerned that this might skew the model/results. Any pointers as to how I should handle such a scenario?


texts: ValueError: array length 13996092 does not match index length 214200 for Kaggle submission

texts:
My model is based on the model linked here on Kaggle.
Specifically, the following lines of code to generate an uploadable table to make my submission:
submission_pfs = my_model.predict(X_test)
# we will keep every value between 0 and 20
submission_pfs = submission_pfs.clip(0,20)
# creating dataframe with required columns
submission = pd.DataFrame({'ID':test_data['ID'],'item_cnt_month':submission_pfs.ravel()})
# creating csv file from dataframe
submission.to_csv('sub_pfs.csv',index = False)

But I get the following error and I am not sure why:
ValueError: array length 13996092 does not match index length 214200

X_test is all my test data (a time series of values). The dimensions of numpy being (424124, 33, 1)
test_data is the test dataset, (just 3 columns with all the ids (product ID, shop ID and entry ID) which has a shape of 214200 rows × 3 columns
The shape of submission_pfs is (424124, 33, 1) as well
The 214200 comes from test_data. While 13996092 comes from multiplication 424124*33
I tried:
submission = pd.DataFrame({'ID':X_test['ID'],'item_cnt_month':submission_pfs.ravel()})

But that gives:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

While:
submission = pd.DataFrame({'ID':X_test,'item_cnt_month':submission_pfs.ravel()})

gives:
Exception: Data must be 1-dimensional

I am not sure what I am doing wrong and how I can fix this


texts:
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis works from my school, but unfortunately all of this data is in 3 different PDFs.

Is there any way I could (ideally not too slowly, since I would like to use a large data set) gather this information from PDF? Or perhaps look for this data elsewhere?

On the website the metadata is only basic, such as name of the advisor, student and the title.

I've edited my question to clarify that not all of them are in the same place. I guess I would have to use some sort of web crawler?

EDIT 2: I have looked into PyPDF as recommended and the problem that I see is that the data from the PDF that I want is for example the grade which is just some number somewhere in the PDF, or the professor title which will be different for every file.


texts: using regression method for payroll monthly controls

texts:
I serve as internal auditor in few clients ,one of my client has thousands of employees in different location, most of them in the head office, the client looks for corporate control for the salary monitoring
is it make sense to use the regression method in order to find outliers, potential parameter can be -years of experience, gender, level/rank etc I planned to go over all the monthly payroll and look for significant outliers ,because of the differences between the global location, it might be a good idea to focus only in the head office the idea is to train the model for previous months average and test it for the current month what do you think is too much effort or theoretical ? or can have a good chance to bring value ?
thank you


texts: Why dimensions of matrix in neural network not matching

texts:
My goal is to implement a feedforward neural network using Andrew Ng's course on youtube. Here, there is no separate bias matrix used. I am not able to get the delta_weight matrix match dimension with weight matrix, which I can subsequently subtract to reduce error.
If anyone could help spot the potential errors in the logic shown below, that would be amazing.

Thanks in advance!


texts: I have few basic questions on R Stem and Leaf Plot


texts:
I am struggling to figure a way to determine if a person is 'no good' or 'good'.

A little bit about what I am trying to accomplish, I have a data set of a payment for a violation with a check or a bounced check on a given day, if there is a bounced check it means there is no such account.

If they have a pattern of payments and bounces in chronological order then they are doing something fishy such as gaming the system. But, if they have payments, bounces, payments, and little to no bounces after that then the initial bounces could have been human entry errors.

I want to use the data set to determine who are the 'no good' people that have a pattern, ex. paying, bouncing, paying, bouncing.

Here is an example of my data set:

person     state date                           status  bounce_code
A           NY  23-DEC-15 06.27.08.000000000 PM Paid
A           NY  23-DEC-15 06.12.58.000000000 PM Paid
A           NY  18-DEC-15 10.14.39.000000000 AM Return  R03
A           NY  15-DEC-15 04.16.58.000000000 PM Return  R03

The original data set contains thousands of people and their history of payments and bounces.

One idea is to analyze each person to see if they fit a "mold" (payment, bounce, payment, bounce) and if they do i will label them as a 'no good' person.

Please let me know if this is confusing and I am open to suggestions on how to approach this problem.


texts: How to evaluate clusters base on an attribute of the dataset?

texts:
I have a data set of persons with attribute job that have 300 different nominal value.
attribute job have a lot of noise. I decide to cluster my data base on other attribute (other feature of person) that related to their job. I hope I reach to clusters with high correlation to job.

Now, I want to know, how much I were successful. In my opinion, more correlation between job and the clusters means more success. I found chi-squared as a method to check correlation between two nominal attributes.
But the result is very low p-value for all clustering.

Is there any method to evaluate a clustering with a predefined class?


texts: Explanation on some steps of AdaBoost.R2

texts:
I am trying to understand AdaBoost.R2 in order to implement it and apply it to a regression problem. In this circumstances I need to understand it perfectly, however there's some step i don't really get.
The paper is available here, and Adaboost.R2 is presented in section 3:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.314&rep=rep1&type=pdf

In step 4, $\operatorname{sup}|.|$ is used; I've never seen that notation, what does it mean exactly?
In step 7, "** means exponentation", in that case that would mean $w_i\beta  *\operatorname{exp}([1-L_i])$, right?


texts:
I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?

Adding more details from comments below.

I'm new to this so guidance is appreciated.

Memory: GPU is K80
Framework: Cuda and cuDNN
Data size per workloads: 20G
Computing nodes to consume: one per job, although would like to consider a scale option
Cost: I can afford a GPU option if the reasons make sense
Deployment: Running on own hosted bare metal servers, not in the cloud.

Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU.


texts: An exhaustive, representative test database in phrase search algorithm

texts:
For a phrase searching algorithm, imagine the goal is to search for a name phrase and return matched results based on a pre-defined threshold. For example, searching for "Jon Smith" could return "Jon Smith", "Jonathan Smith", "Jonathan David Smith", "Jonathan Smith-Mikel", "Jonathan 'Smith' Mikel" etc.

The plan is to manually choose N test cases and put them in a benchmark database. I have concern about this plan because the test cases is likely to be not exhaustive. I know there are pretty mature search engines there, so is there a test database which covers all possible cases, such as different name combinations, punctuations, symbols, etc. such that we can use this as our benchmarks instead of guessing?

For example, this test database should contain all cases for "Jon Smith", as well as connectors such as hyphen, apostrophe, and so on.


texts: Machine learning in electrical circuit simulation

texts:
In the field of VLSI design, and specifically in digital circuits it is common to use electrical simulation software to find signal integrity issues.
I would like to use machine learning instead of running a true electrical simulation to do the same, in order to boost run time (simulations can be very heavy). For example, a NAND logic gate with 2 inputs and 1 output, and on each input there is a noise glitch, and I want to predict the glitch which may occur on the output. A glitch is a waveform represented as a series of voltage and time points. That is, for a given electrical circuit I would like to build a model which can be used to predict an output waveform given input waveforms. Instead of running a (spice) simulation code which calculates results analytically, the output waveform will be predicted or estimated by the model.
Training data can be generated plentifully by running real simulations as many as needed.
I need a direction for types of machine learning algorithms and architectures.
From my research I think using a graph-neural-network (GNN) could be a good direction since an electrical circuit is actually a graph, and it is important to capture the relationships between its elements and other graph properties.

Thanks!


texts: Input for LSTM for financial time series directional prediction

texts:
I'm working on using an LSTM to predict the direction of the market for the next day.

My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$ represents a vector of features for day $i$, i.e $x_i \in \mathbb{R}^D$. The target variable $y_i$ is the direction of the market for the next day $$ y_i = I_{P_{i+2} - P_{i+1} > 0} $$ where $P_i$ is the opening price of the asset for day $i$.

I am wondering now how to configure the input to fit in an LSTM framework. The LSTM requires a sequence of length $T$ and uses this together with target $y_T$. One approach is a rolling window. Take input as input the sequences $I_1 = x_1, \ldots, x_T$, $I_2 = x_2, \ldots, x_{T+1} \ldots $ and use $y_T, y_{T+1} \ldots$ as the target variables. The problem with this is that $I_1, I_2$ are very similar, they differ only on two points yet the target variable may be $1$ for the first series and $0$ for the other, making it impossible for the LSTM to learn in my opinion.

I am wondering if anybody has any idea of how to approach this problem. The rolling window approach above is considering the inputs $I_k$ as independent, similar to if we would have input $I_k$ to be a sentence which should be classified as french or english. I want the LSTM to take into consideration that the sequences its fed are all parts of the same long sequence if that makes sense.

A lot of papers use recurrent neural networks for this problem, but never really specify on how they structured the input.


texts:
I am trying to fine tune t5 to train the mode over a huge data set
my problem is very close to this  issue
except I am out of RAM not gpu memory
as you can see from the screen shot the number of processes related to train_t5 is always increasing until OOM accrues.
training step returns {"loss": loss, " self.train_loss": self.train_loss.avg} ; loss is the loss from t5 and it is torch tensor number, self.train_loss is the accumulated loss acorss all training steps across all  and it is float number.
validation step returns  {"val_loss: loss, "batch_length": len(batch['field1']), "pred": pred, "field1": batch['field1'],"field2": batch['field2']}; loss is a torch tensor, batch_length is just int number, while pred, fiel1, field2 are texts which each of themm added to list at the end of validation to caculate other metrics, then I use .clear to empty the lists at the end of validation, so I do not have any thing stored while training or evaluating.
why my machine is OOM? what is the reason behind that?


texts:
Why don't all feature selection methods in sklearn allow specifying desired variance explained?

sklearn.decomposition.PCA does allow inputting a percentage of variance that one wants to be explained in place of n_components. However other methods such as SparsePCA expect the user to "know" how many n_components he/she wants to take.


texts: How can I specify I want output of which units in LSTM?

texts:
I see this code concept(with Keras library) in most code examples of LSTM:

model.add(LSTM(X))
model.add(Dense(Y))

But I don't really know if I have 10 time-steps in input side and need the last time-step in output is the following code true?

model.add(LSTM(10))
model.add(Dense(1))

Or if I have 10 input time-steps and need 10 output time-steps(I mean I need all outputs from all units) is the following code true?

model.add(LSTM(10))
model.add(Dense(10))

How should I specify if I have 1 layer LSTM with 10 units, I need all of 10 outputs? Or I need 1 last output from unit 10? Or the last 5 outputs from unit 5 to 10?


texts: RandomForestClassifier : binary classification scores

texts:
I am using sklearn's RandomForestClassifier to build a binary prediction model. As expected, I am getting an array of predictions, consisting of 0's and 1's. However I was wondering if it is possible for me to get a value between 0 and 1 along with the prediction array and set a threshold to tune my model.
Many thanks in advance


texts: Do you know a dataset for regression where deep learning outperforms svm and random forests?

texts:
I am currently trying to learn about deep learning. I asked myself where deep learning outperforms SVM and Random Forest on regression?

Do you have any dataset for regression where deep learning outperforms SVM and Random Forest?

My main objective for asking this question, is to see on which kind of datasets deep learning outperforms SVM and Random Forests. Since I do not know what kind of datasets these are, I do not want to be too restrictive when asking about these datasets.

Update:
From the answer given by tom, it gets clear to me, what kind of dataset I am looking for. So to be more specific: If you know a dataset with spatial/temporal structure at different levels of granularity which is known that for example convolutional deep networks outperform SVM and Random Forest on this dataset, then please share this dataset or a link to it.

Don't know why the question is still marked as on hold for being to broad, as I updated the question.

Thank you for your help.


texts: Analysing wrongly predicted labels and pick most significant

texts:
I try to do behavior analysis of network systems based on network flow. I have a model that predicts if a certain flow pattern is a Windows Server, LX Server or Windows Client. (Highly simplified) I am interested in the flows that were incorrectly predicted.
Since I am predicting based on flows the resulting table has many entries per host. The predicted label for the same host might be different depending on the flow. I have attached a screenshot of how it could look like. System A is there twice for 2 different flows which has been "falsly" predicted.
I am trying to find a good way to compress the table so that there is 1 entry per host with maybe the most significant false predicted label in order to trigger some alert. I probably will require some sort of threshold to minimise the false positives accepting the risk that I might miss some true positives.


texts: Encoding multiple observations from the same feature space

texts:
My data contains multiple observations of categorical feature.The feature space is medical symptoms, so the data for this feature is like : ['fever','pain','yellow skin' .... ] .The amount of symptoms observations per sample is not fixed and i have around 50 different symptoms

how can i encode this feature into something that ML model can deal with ?  the order of the symptoms in the array is not important.

i tried one hot encoding but projecting feature space of 50 category levels into 50 indicators means losing information (having a sparse matrix )

any ideas?


texts: Why does my KNeighborsClassifier graph look like this?

texts:
I'm new to data science/ml and working on using the sklearn libraries to classify data. I'm currently using the KNeighborsClassifier with 5 fold cross validation whilst tweaking the k value but its producing a graph that looks quite strange.

I have my training data and test data in 2 different CSV files and load them in like this:

trainData = pd.read_csv('train.csv',header='infer')
testData = pd.read_csv('test.csv',header='infer')

I then separate the classifiers (Y is name of the column in my dataset that's the classification):

trainY = trainData['Y']
trainX = trainData.drop(['Y'],axis=1)

testY = testData['Y']
testX = testData.drop(['Y'],axis=1)

I use sklearn KNeighborsClassifier with 5 fold cross validation whilst tweaking the k value from 2 to 20:

trainAcc = []
testAcc = []

for i in range(2,20):
    clf = KNeighborsClassifier(n_neighbors=i, metric='minkowski', p=2)
    trainScores = cross_val_score(estimator=clf, X=trainX, y=trainY, cv=5, n_jobs=4)
    testScores= cross_val_score(estimator=clf, X=testX, y=testY, cv=5, n_jobs=4)
    trainAcc.append((i, trainScores.mean()))
    testAcc.append((i, testScores.mean()))

I then print the graph:

plt.plot([x[0] for x in trainAcc],[x[1] for x in trainAcc], 'ro-', [x[0] for x in testAcc],[x[1] for x in testAcc], 'bv--')

But I get something weird like this:

Can anyone explain where I went wrong and why my graph looks the way it does.

Thanks.

EDIT: It is indeed weird because when I run it without doing the cross-validation, I get a more normal graph like this:

clf.fit(X=trainX, y=trainY)
predTrainY = clf.predict(trainX)
predTestY = clf.predict(testX)
trainAcc.append(accuracy_score(trainY, predTrainY))
testAcc.append(accuracy_score(testY, predTestY))


texts:
I want to make an mobile app where you scan using the camera and it gives you the card you just scanned. (from a board game)
I got a PNG of each and every single existing cards, but I don't really know what to use.

For now, I've been using the text in the card using OCR to find the card, but it's not 100% accurate as the typography is pretty special.

I've been thinking about using image classification, but I'm not sure it's the right way to do it, as I would be training for hundreds of classes. (got over 400+ different cards)

Do you have any idea what I should be using ? (I'm sorry if this seems like a basic question, but I just started learning about CV and ANN)
Thanks for your help !


texts: Giving Emails as Input to Machine Learning Algorithms

texts:
I want to classify emails as Spam and Non-Spam.

I have a labelled dataset of 20,000 emails in TXT format. The emails are in individual files and also in one combined file.

An example email looks like this:

From: "Sender Display Name" <sender@abc.com>

To: systudent <systudent@abc.com>, tystudent <tystudent@abc.com>,
btechstudent <btechstudent@abc.com>, mtech16 <mtech16@abc.com>, mtech17 <mtech17@abc.com>

Subject: Register to the event

Date: Tue, 21 Nov 2017 14:16:17 +0000

X-Originating-IP: [13.90.24.116]

Body:

Some spam text

<https://somelink.com/abc>

EOM
Label: Spam

The features that I want to use are: Sender Display Name, Sender Email address, Receivers, Subject, Date, IP, URL.

How do I convert these to input feature vectors or how can I give these as input which are currently in TXT format, to Machine Learning Algorithms like Random Forest, Naive Bayes, etc. ?


texts: Trying to understand the Fowlkes-Mallows Score

texts:
I recently bought Chris Albon's ML flashcards and I'm working my way through them. But this one on the Fowlkes-Mallows score has me stumped, as his definitions of false negatives and false positives seem reversed:

If y_hat are my predictions, and I've said that a pair of observations are in the same cluster when they're not, how is that a false negative and not a false positive?

Of course I read the Wikipedia article and did some Googling, but haven't turned up a clear answer yet.

Am I misunderstanding? Did he get this wrong?


texts: Getting a neural network to approximate x^2

texts:
I don't get why it is so hard to get my neural network to learn such a simple function. I've tried all sorts of combinations of layer numbers, number of neurons but it doesn't seem to want to learn. Any suggestions would be greatly appreciated!
Here is my code:
import numpy as np
import numpy
from tensorflow import keras
from tensorflow.keras import layers
import random

inputs = keras.Input(shape=(1,))
dense = layers.Dense(50, activation="relu")
x = dense(inputs)
x = layers.Dense(50, activation="relu")(x)
x = layers.Dense(50, activation="relu")(x)
x = layers.Dense(50, activation="relu")(x)
x = layers.Dense(50, activation="relu")(x)
outputs = layers.Dense(1)(x)

model = keras.Model(inputs=inputs, outputs=outputs, name="kinetic_energy")
adam = keras.optimizers.Adam(lr=0.0002,beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False,)
model.compile(
    loss='mean_squared_error',
    optimizer='adam',
    metrics=["mean_squared_error"],
)

for j in range(10):
    x = np.array([])
    y = np.array([])
    for i in range(1000):
        randint = random.randint(0, 10000)
        square = randint * randint
        x = numpy.append(x, randint)
        y = numpy.append(y, square)
    model.fit(x, y, epochs = 100)

x = np.array([20])
z = model.predict(x)
print (z)
```


texts: Install Spark and Hadoop in the same machine

texts: I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it worked perfectly. But, when I started installing Hadoop and set the JAVA_HOME environment variable HDFS worked, but spark break down and shows: Files was unexpected at this time. when I launched it. When I deleted JAVA_HOME, Spark worked again but it's not the case for HDFS. What should I do in this case?


texts:
I am noob to Machine Learning. I am running a DQN code, I understand the concept of Replay Memory but I don't understand how often shoud it be run. I saw one code calling it once in every 'episode' and in another code it was called every iteration in each episode.
Obviously, calling it more often consumes lots of time, and not calling it more often would probably affect the performance of learning agent. So, my question is should I call it once evry episode or should I call it 't' times in each episode?
Many thanks!


texts: Function of functions and theirs arguments

texts:
I would like to build a pandas random dataframe. To fulfill that purpose I need a Python function taking as arguments :

numpy distributions
their arguments.

For example :

distribution 1 : normal | arguments : means = 0 , standard dev = 1 , size = 100

distribution 2 : uniform | arguments : low = 0 , high = 1 , size = 100

etc...

I do not know in advance what will be the different distributions and their arguments.

The main function will then generate random samples of the distributions using each corresponding arguments.

I have tried something like :

import numpy as np

def myfun( **kwargs ) :
    for k , v in kwargs.items() :
        print( k )
        print( v )

When I call that function with these arguments :

myfun( fun_1 = 'np.random.normal' , arg_1 = { 'loc' : 0 , 'scale' : 1 , 'size' : 7 } ,
       fun_2 = 'np.random.uniform' , arg_2 = { 'low' : 0 , 'high' : 1 , 'size' : 7 } )

The output is :

fun_1
np.random.normal
arg_1
{'loc': 0, 'scale': 1, 'size': 7}
fun_2
np.random.uniform
arg_2
{'low': 0, 'high': 1, 'size': 7}

But my purpose is not to print the desired distributions and their associated parameters but to generate a sample for each distributions.


texts: How are neural nets combined to form intelligent systems?

texts:
Self-driving cars, arms that can learn ow to pick up objects, machines that can have conversations, etc.  I understand how neural nets work.  But I really don't know much about how neural nets are combined to form intelligent systems.  Is it all rule-based above a few base nets for tasks like e.g. image recognition?  Or are there meta-nets that do deep learning on the outputs from base nets?

Things to read/references would be super-helpful!


texts: How to replace the missing values in Age column from Titanic/kaggle project

texts:
The code below doesn't give any errors but it doesn't iterate after the first observation, so it's useless. I tried using sapply without any improvement and I don't know if it's a smart approach to use it in conjunctions with the other functions...
Any idea how to improve my code is much appreciated.

for (i in 1:nrow(train)) {
  if (is.na(train[i,5])) {
    if (grepl("Master", train$Name, fixed=TRUE) == TRUE) {
          train$Age[i] = round(mean(train$Age[train$Name == "Master"], na.rm = TRUE), digits = 2)
        } else if (grepl("Miss",train$Name, fixed=TRUE) == TRUE) {
      train$Age[i] = round(mean(train$Age[train$Name == "Miss"], na.rm = TRUE), digits = 2)
        } else if (grepl("Mrs",train$Name, fixed=TRUE) == TRUE) {
      train$Age[i] = round(mean(train$Age[train$Name == "Mrs"], na.rm = TRUE), digits = 2)
        } else if (grepl("Mr",train$Name, fixed=TRUE) == TRUE) {
      train$Age[i] = round(mean(train$Age[train$Name == "Mr"], na.rm = TRUE), digits = 2)
        } else if (grepl("Dr",train$Name, fixed=TRUE) == TRUE) {
      train$Age[i] = round(mean(train$Age[train$Name == "Dr"], na.rm = TRUE), digits = 2)
    }
  }
}


texts: How does combining neurons create non-linear boundaries?

texts:
I have been working with NNs for a while, but haven't dug too deep into this unfortunately.

By looking at the three neurons below, in each of their boxes we can see that they are really just making linear separations in the x1, x2 plane (of course not taking into account the third and upward Y dimension that make the sigmoid) and combining these into the decision boundary we see at the right.

I understand we need non-linear activation functions, but why?

How can combining non-linear perceptrons can make this three walled decision boundary?
Mathematically, can it be illustrated how we are allowed to go from a simple linear function, to a more complex non-linear function using multiple neurons?

I have looked around and articles just says that they can, but not how?


texts: Is there a way to get an explanation of the model prediction for a specific example?


texts: recommender system: how to compare different scores when calculated individually?

texts:
I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for each product allowing me to take the other targeted products as features (e.g, if a customer has product X, that information could help predict detention of product Y ; conversely, knowing that a customer has product Y could help predicting detention of product X).

So I end up with ~10 models (say model X, model Y, etc.) with almost the same dataset (only the target changes, and the feature corresponding to that target is removed). However, as each model is different, I am not sure how to compare the scores I obtain for all my products and how to make the best recommendation.

For example, suppose that for a given customer:

score for product X is 0.7 and score for product Y is 0.8 but the precision of model Y is not as good as model X. Should I really recommend product Y?

I was suggested to standardize the scores for each product across customers to end up on the same scale. However, if a product ends up with very higher scores than others, this step would in effect penalize this product.

Any ideas?

PS : the detention of each product is pretty low (between 0.5 % to 2 %). I thought a first step would be to re-balance the dataset for each model (say with under of over sampling) in order to have the same class imbalance for each model as I observed that re-balancing biases the scores upwards (unfortunately without really improving the performance of my models).