texts: Multilabel multiclass classifier returns same probabilities for any input

texts:
I am new to deep learning. I have a collection of website texts and the activity class each company belongs to. A company can list multiple activity classes (NACE-codes)

The goal is to build a recommender that would recommend the five best activity classes based on a website the user provides.

The model is trained on the top 10K most frequent words that occur in the websites. The words are represented as scores ranging from 0 to 1. Each text is a line in the matrix, with 3000 columns (the first 3K words per website are taken into account).

The labels are k-hot encoded (activity class labels are columns and have 1 if the company lists the activity class; otherwise 0)

There is a significant imbalance between the labels, so each label is weighted by dividing the total number of classes by the time each class occurs in the dataset.

When I train the model, the loss goes down and my custom 'precision' score (correct activity class is among the top-5 predictions) goes up somewhat during training. However, when I use the model to predict, it will always return the same scores between 0 and 1 (what you would expect, given the sigmoid layer) for the classes, no matter what the input is. So, it seems that all the model has learned is that some classes are more likely to occur than others.

Any suggestions or guidelines would be appreciated. I have read Chollet's "Deep learning with python", but it is not exactly a great source of advice on this particular topic.

My model looks like this:

 library(keras)
 metric_precision <- function(y_true, y_pred) {
   metric_top_k_categorical_accuracy(y_true, y_pred, k = 5)
 }

    model <- keras_model_sequential()
    model %>%
    layer_embedding(input_dim=10000, output_dim=512,
     input_length=3000, mask_zero = FALSE, trainable=TRUE ) %>%
    layer_dropout(0.3) %>%
    layer_conv_1d(filters,kernel_size, activation = "relu") %>%
    layer_max_pooling_1d(5) %>%
    layer_dropout(0.3) %>%
    layer_conv_1d(filters,kernel_size, activation = "relu") %>%
    layer_global_max_pooling_1d() %>%
    layer_dense(ncol(train_y),"sigmoid")

  model %>% compile(
    loss = 'binary_crossentropy',
    optimizer = 'rmsprop',
    metrics = c(precision=metric_precision)
  )

  model %>% fit(
    train_x, train_y,
    batch_size = batch_size,
    epochs = epochs,
    validation_data = list(test_x, test_y),
    class_weight = classweights
  )


texts: Using Images uploaded in google drive in Colab

texts:
Does anyone know how to use already uploaded images in Google Drive to colaboratory for creating a trainloader?
I'm creating the trainloader and model in Pytorch


texts: I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both the columns you find the correlation coefficient to be high. Now for the values that aren't there, can you use linear regression to predict or find them out, by using the values you know as training data?


texts: Python sklearn model.predict() gives me different results depending on the amount of data

texts:
I train my XGBoostClassifier().
If my testing set has:
0: 100
1: 884

It attempts to predict 210 1's. Around 147 are wrong (False positives) and 63 1's correctly predicted (True positives).
Then I increase my testing sample:
0: 15,000
1: 884

It attempts to predict 56 1's. Around 40 are wrong (False positives) and 16 1's correctly predicted (True positives).
Am I missing something? some theory? some indication on how to use model.predict(X_test)?
Does it say somewhere - if you try to predict 10 items is gonna try harder than if you try to predict 10000 items? In what situation model.predict(X_test) would give me a different result for Joe Smith if his prediction is accompanied by 8000 more rows?
The code I use is the following:
from xgboost import XGBClassifier
xgb = XGBClassifier(subsample=0.75,scale_post_weight=30,min_child_weight=1,max_depth=3,gamma=5,colsample_bytree=0.75)
model = xgb.fit(X_train,y_train)
y_pred_output = model.predict(X_test)
cm = confusion_matrix(y_test, y_pred_output)

y_pred_output2 = model.predict(X_test2) #contains the same 884 1's plus 14500 more rows with 0's as the target value
cm = confusion_matrix(y_test2, y_pred_output2)

it produces two different matrices:
#Confusion matrix for y_test with 15000 0's and 884 1's
[[14864   136]
 [  837    47]]

#Confusion matrix for y_test with 500 0's and 884 1's
[[459  41]
 [681 203]]

Notice that the same 884 positive class items are being used across both attempts. Why would the true positives go down to 47 just because we now have more Negatives on the X_test?


texts: Temporal Difference Learning Getting Stuck

texts:
I'm trying to implement a temporal difference algorithm that learns the maximum revenue over a period of time using prices as the actions, inventory as the state, and revenues realized as the reward. The problem I'm having is that I can't seem to get it to converge on an optimal policy. It seems like it gets stuck at a revenue somewhere around 60% of optimization and then won't budge anymore. Are there some common pitfalls that might be causing this? I've tried playing with the rate of exploration a little bit, but that hasn't seemed to help it much.

EDIT:
Ok, so I went back through everything, and it seems like the problem is that, when it stops exploring, it just keeps increasing the Q-values of states that it's already deemed the best in the past. So for example, it visits the price 5 at a certain state and time and gets a reward. Then, in the next several episodes, it continues to visit 5, and continues getting a reward, adding that reward to the Q-value until it's pretty high. At that point, even if it were to explore at the same state and time, the reward it gets isn't enough to overcome the inflated Q-value, so it just goes right back to 5 in the next episode. Here are the steps I'm trying to follow.


texts: Using LeakyRelu as activation function in CNN and best alpha for it

texts:
Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write:

model.add(Conv2D(32, kernel_size=(3, 3),
           input_shape=(380,380,1))
model.add(LeakyReLU(alpha=0.01))

I mean now by the written lines, the activation function for Conv2D layer is set as LeakyRelu or not?

Further, I want to know what is the best alpha? I couldn't find any resources analyzing it.


texts: neural network to find a very simple linear model (scikit-learn)

texts:
I'm trying to test different machine learning algorithm to try to find correlation between various data on MRI scans.
Since I'm dealing with medical data, I don't have access to many events, but still I'm trying to see what a simple fully connected NN while provides me.
By debugging, it looks like even if I enter the output almost as is in the input, a simple NN using the default scikit-learn NN regressor is unable to find a satisfactory model.
I tried to reproduce the effect and I'd like to share with you the following code. I guess I'm just doing something wrong, because, basically, I'd like the NN to produce a simple model where label=X2-X1 and it doesn't work to me which is really surprising me.

here is the code:

import sklearn.neural_network as NN
import matplotlib.pyplot as plt
import numpy as np
N_events=100000 # number of rows of dataset
N_features = 2 # number of features
X = np.random.rand(N_events,N_features) # chose the data randomly
labels = X[:,1]-X[:,0] # label = X2-X1

C = NN.MLPRegressor(hidden_layer_sizes=(2,2),max_iter=500000,random_state=1) # very simple scikit-learn NN regressor
n = N_events // 2 # half data for training, half for test

print(X[0:10,1]-X[0:10,0]) # check that indeed label=X2-X1
print(labels[0:10])
C.fit( X[0:n,:] , labels[0:n]) # train the model

# plot y_predicted vs y_true for the test set...
plt.figure()
y_real = labels[n:] #
y_pred = C.predict(X[n:,:])
plt.plot(y_real,y_pred,'.')

# plot y_predicted vs y_true for the training set... even this doesn't work
plt.figure()
y_real = labels[0:n] #
y_pred = C.predict(X[0:n,:])
plt.plot(y_real,y_pred,'.')

I was expecting the NN with 2x2 = 4 degrees of freedom to easily find such a simple model (note that I even don't add unused features which could disturb the fit, since N_features is 2 here), so I guess I'm just not using the code correctly.
Can anyone helps me ?

Many thanks


texts: Matplotlib -- TypeError: 'module' object is not callable

texts:
I am new into ML. As per video tutorials when i try to execute the following lines i get an error.
However the instructor get successful execution.

Error is :

i have google about it but didnt get any clue. May be there is change in library ?
How can i solve this Error.


texts:
I have problem with loading data into R:

fileUrl <- "http://jadi.net/files/iran_it_status_1394_detail_data_jadi_net.tsv"

download.file(fileUrl , destfile="iran_it_status_1394_detail_data_jadi_net.tsv")

dev <- read.delim("iran_it_status_1394_detail_data_jadi_net.tsv",
              header=TRUE,sep="\t",blank.lines.skip = TRUE,
              na.strings="",fileEncoding="UTF-8",
              stringsAsFactors=FALSE,skipNul = TRUE)

I receive the following error:

Error in read.table(file = file, header = header, sep = sep, quote = quote,  :
  no lines available in input
  In addition: Warning message:
  In read.table(file = file, header = header, sep = sep, quote = quote,  :
  invalid input found on input connection    'iran_it_status_1394_detail_data_jadi_net.tsv'

Edit: The dataset has 1217 rows and 33 variables.

names(data) <- c("timestamp","age","sex","birth_province","work_province","experience","education",
            "certificate","learn","project","book","language","wish_language","db","desktop_os",
            "wish_os","mobile","env","theme","src_ctrl","tab_space","drink","items","device","title",
            "org_type","org_emp","income","perk","job_contract","job_type","hour_wage","happy")

for language variable I expect this output:

data[1:3,"language"]

C#, Javascript, R, SQL

Java, C#, Javascript, Objective C, Swift, SQL

C#, SQL

Python Solutions are also welcome


texts: Training loss decreasing but validation loss not decreasing

texts:
I found out many questions on this but none solved my problem. I have a problem in which the training loss is decreasing but validation loss is not decreasing. When looking for an answer to this problem, I found a similar question, which had an answer that said, for half of the questions, label a wrong answer as correct.  If you re-train your RNN on this fake dataset and achieve similar performance as on the real dataset, then we can say that your RNN is memorizing.
So I tried it, and these were the results:

These are the loss and accuracy plots of the real dataset, that is, one with all correct labels.

And these are the loss and accuracy plots of the fake dataset, that is, the dataset which has half randomly assigned labels.
Both the loss and accuracy plots looks similar, so does this mean that my model is memorizing both the data (real and fake). If yes, how can I fix this?
Thank you in advance!


texts:
Which of the following is best (or widely used) for calculating item-item similarity measure in mahout and why ?

Pearson Correlation
Spearman Correlation
Euclidean Distance
Tanimoto Coefficient
LogLikelihood Similarity

Is there any thumb-rule to chose from these set of algorithm also how to differentiate each of them ?


texts:
I am starting to play around in datamining / machine learning and I am stuck on a problem that's probably easy.

So I have a report that lists the url and the number of visits a person did. So a combination of ip and url result in an amount of visits.

Now I want to run the k-means clustering algorithm on this so I thought I could approach it like this:

This is my data:

url      ip    visits

abc.be   123   5
abc.be/a 123   2
abc.be/b 123   2
abc.be/b 321   4

And I would turn in into a feature vector/matrix like so:

abc.be  abc.be/a   abc.be/b   impressions
   1       0          0          5
   0       1          0          2
   0       0          1          2
   0       0          1          4

But I am stuck on how to transform my data set to a feature matrix. Any help would be appreciated.


texts: Random Forest variable Importance Z Score

texts:
I am really worried How does RandomForest Algorithm Works internally,

Out of bag error
Variable Importance
Z Scores

Let me know all these three ? How to calculate it? What it controls?


texts: Confidence Intervals for Multi-Categorical Votes

texts:
I have an ngram-based language model that produces a long tag list for a given sentence. For example, the just-previous sentence, broken into bigrams, and run through the model might produce something like:

{I have}=>C1 {have an}=>C2 {an ngram}=>C1 {ngram based}=>C3, etc.

resulting in counts: C1=2, C2=1, C3=1 (for the shown segment above).

Easy enough to pick the winner by sorting either the counts, or after turning them into percentages, which would control for sentence length. But I want a CI on that winner -- that is, I want to know when it's a statistical tie between the top N categories (either by count or percent).

I'm sure that there's an obvious way to do this....

...Pointers appreciated!


texts: What is the quantity sold for a specific fruit & country combination?

texts:
What is the algorithm that generates these potential quantities that meet the given criteria?

Essentially - there are number of quantities for a fruit and country combination. E.g:

    Country+Fruit   Potential Quantity
1   India+Apple     25
2   India+Apple     27
3   India+Banana    35
4   India+Banana    37
5   France+Apple    130
6   France+Apple    132
7   France+Banana   11
8   France+Banana   13
9   France+Banana   15
10  France+Cherry   88

For complete dataset click here.

Each Country has to be as close as possible to the following values for all fruits sold:

   Country                 Total Fruits Sold
1. Total India Fruits        1403
2. Total China Fruits        1370
3. Total England Fruits      1115
4. Total France Fruits       1169
5. Total Germany Fruits      1470

And total fruit quantity across all countries has to be as close as possible to the following values:

   Fruits           Total Fruits Sold Across All Countries:
1. Total Apples        508
2. Total Bananas       253
3. Total Cherries      982
4. Total Guavas        389
5. Total Kiwis         681
6. Total Oranges       489
7. Total Mangos        608
8. Total Strawberries   1060

An example combination that comes close to meeting the above criteria (having 5 countries and 8 fruits) is:

    Country Fruits      Matched Combination
1.  India Mango         120
2.  India Apple         40
3.  Germany Apple       20
4.  Germany Mango       80
.
.
40. France Mango        186

What is the algorithm that generates these potential quantities that meet this criterion?

Problem 1: do we have to do a brute force method of generating all possible combinations or is there a more efficient way?

Problem 2: how do we define "closeness" - the exact match is the best - what if there is no exact match - then what is the next best option?


texts: LSTM low training/validation error but really bad predictions

texts:
I'm building a LSTM model to create an automatic drums composer.
I'm following this post: LSTM Metallica
I've built my model and done all the enconding, I was able to emulate the behavior of the networks proposed in the post.
In fact, if I use the dataset provided by the blog user (Metallica dataset) my model actually learns repeating patterns of musical bars.
Here's an example of the input encoding:
0b010000000 0b010000000 0b000000000 0b010000000 0b010000000 0b000001000 0b000000000 0b000001000 0b010000000 0b010000000 0b000000000 0b010000000 0b010000000 0b000001000 0b000000000 0b000001000 BAR

0b010000000 0b010000000 0b000000000 0b010000000 0b010000000 0b000001000 0b000000000 0b000001000 0b010000000 0b000000000 0b000000000 0b000001000 0b000000000 0b000001000 0b000001000 0b000000000 BAR

0b100000001 0b000000000 0b000000000 0b000000000 0b010000001 0b000000000 0b000000000 0b000000000 0b100000001 0b000000000 0b000000000 0b000000000 0b010000001 0b000000000 0b000000000 0b000000000 BAR

0b100000001 0b000000000 0b000000000 0b000000000 0b010000001 0b000000000 0b000000000 0b000000000 0b100000001 0b000000000 0b000000000 0b000000000 0b010000001 0b000000000 0b000000000 0b000000000 BAR

0b100000001 0b000000000 0b000000000 0b000000000 0b010000001 0b000000000 0b000000000 0b000000000 0b100000001 0b000000000 0b000000000 0b000000000 0b010000001 0b000000000 0b000000000 0b000000000 BAR

Where 0b100000001 represents the note played on the drums, while BAR represents the ending of a musical Bar. As you might notice, we have 16 events/notes per Bar.
If i train the network over the dataset provided, I get good results, my model learns the pattern/structure inserting a BAR every 16 notes predicted.
The loss of my model is the following:

Which is the typical behavior of a network that starts overfitting at a certain point.
I tried the same model over my own dataset of songs, which I already know it's not that good (notes pattern do not make so much sense some time, we generated the drums artificially) but we still have a BAR every 16 notes.
I expected my network to poorly learn the drums generation but at least to learn the repeating pattern of BAR. So for example it generates totally random notes, but every 16 notes it generates a BAR
This unfortunately doesn't happen, the network has a totally random behavior, I tried to tune many parameters (input length, LSTM size. layers ecc...) without any success.
Also my network training has a strange behavior:

These are the loss of the model over my dataset (using different parameters), and I can clearly notice that there's something wrong, the behavior of the model over the original dataset (Metallica, the first plot) had much more sense in my opinion.
Also  thought of the underfitting/overfitting problem, but given the losses, both of them are really really low, and I can't understand why, given such low losses, my model hasn't learn anything at all from the data.
The model is the following:
model = Sequential()
model.add(LSTM(512, return_sequences=True, input_shape=(maxlen, num_chars)))
model.add(Dropout(0.2))
model.add(LSTM(512, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(512, return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(num_chars))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam')

Which could be the problem? I started to think that it's the dataset, but I expected my network to (at least) learn the repeating pattern of BAReven if the notes do not make much sense from the musical point of view.


texts: I have hourly data of a metric for 15 days, Can i predict the outcome values for same metric for the next 15 days?

texts:
I have tried a linear regression model for the same data, Since the regression line is continuous i'm not sure if it works to predict the outcome values for next 15 days, or for a given period of time!

Are there any models to be considered?


texts: Assumptions/Limitations of Random Forest Models

texts:
What are the general assumptions of a Random Forest Model? I could not find by searching online. For example, in a linear regression model, limitations/assumptions are:

It may not work well when there are non-linear relationship between dependent and independent variables.
It may not work if the dependent variables considered in the model are linearly related. Therefore one has to remove correlated variable by some other technique.
It assumes that model errors are uncorrelated and uniform (No hetroscedasticity).

Are there any assumptions/limitations on similar lines.


texts: Name of this algorithm for supervised cluster assignment

texts:
Is there the name for an algorithm of cluster assignment that is based uniquely on the distance between the data point to classify and the center of the cluster?

Let me be more clear:

Let's say that I have two clusters $A,B$ made by $N$ points $x(i)_A$ and $x(j)_B$. I have to assign a new point $x_{new}$ to it's cluster.

The point is assigned to the cluster that minimize the distance between the point and the center of the cluster:

$$ min_{A,B} ( d_A(x_{new}, \frac{1}{N}\sum_{i \in A}x_i), d_B(x_{new}, \frac{1}{N}\sum_{j \in B}x_j) ) $$

Does this procedure has a name? Is there an algorithm similar to this? The closest algorithm that I can think of is k-Nearest Neighbour, but it's easy to think of cases where kNN performs properly and this algorithm performs very poorly. For instance, this algorithm as such will perform badly on data shaped non-uniformly / non-gaussian.

This question arise from a paper that offer a procedure for doing supervised quantum machine learning, and claims that:

  Consider the task of assigning a post-processed vector $u \in R^n$ to
  one of two sets $V, W$, given $M$ representative samples $v_j ∈ V$ and
  $M$ samples $w_k ∈ W$. A common method for such an assignment is to
  evaluate the distance $|u−\frac{1}{M}\sum_{j} v_j |$ between $u$ and
  the mean of the vectors in $V$ , and to assign $u$ to $V$ if this
  distance is smaller than the distance between $u$ and the mean of $W$.


texts: Impact of sparse features on tree-based models

texts:
Say you have a highly imbalanced binary classification problem.
Some of the features are binary features, where they're false most of the time, but when they're true they tend to be highly predictive (of the minority class).
What impact, if any, would the sparseness of the feature have on the likelihood of the feature being picked as a split?


texts: Generating synonyms or similar words from multiples word embeddings

texts:
I am looking for a way to generate synonyms, using word embeddings. From one word, and from multiple words. Such as the two example below:

"word" -> Word embedding -> generate synonym of "word"

"word", "synonym of word .. "->  -> Word embeddings -> generate a synonym of both word

I am very new at this. What do you think I should use ?

I also want to use a tools that, in further work, would take into account context for word embedding generation, such as:

"sentence in including a word" -> Word embedding of word -> generate synonym of "word" in that context

I think I will start to do it with BERT... How should I start ? or which alternative should I use ?

Thanks for your help !


texts: Sentiment analysis using sources other than the IMDB data

texts:
I'm curious as to whether the training data for a sentiment analysis tool needs to be specifically geared toward the domain it's being using in. For example, the IMDB movie review data makes sense if I want to gauge sentiment on movie reviews, but would it still be okay to use if I want to gauge sentiment on what the general public thinks about a company's stock?

If I want to gauge sentiment on a company's stock, should I create my own training set that uses data from financial news/social media reaction to companies?


texts: How can I fit categorical data types for random forest classification?

texts:
I need to find the accuracy of a training dataset by applying Random Forest Algorithm. But my the type of my data set are both categorical and numeric. When I tried to fit those data, I get an error.

  'Input contains NaN, infinity or a value too large for dtype('float32')'.

May be the problem is for object data types. How can I fit categorical data without transforming for applying RF?

Here's my code.


texts: When should I update weights and biases in Neural Network?

texts:
So, I am building a Neural Network from scratch for (typically) classifying MNIST digits.
Everything is going fine, I can get up to 85% accuracy accross all testing data with stochastic gradient descent. However, I would like to make it better. So, naturally, I am implementing training in mini batches.
My questions is this: when batchsize=1 (aka Stochastic) when should I adjust the WaB of the NeuralNetwork?
Should I compute the WaBDeltas for each example, then average them out, and only when I finish the epoch adjust WaB on the Neural Network?
Or should I adjust the WaBDeltas of the Neural Network after each training example, and proceed training with the adjusted WaB?
Simply put, when am I to adjust the WaB deltas to the neural net?
In terms of code (sorry, not very clean):
// should it be
let nn = new NeuralNetwork(...)
for(trainingExample in examples){
  let WaBDeltas = calculateWaBD(nn,trainingExample);
  nn.adjustWaB(WaBDeltas)
}

// or
let nn = new NeuralNetwork(...);
let WaBSum;
for(t in trainingExamples){
   let WaBDeltas = calculateWaBD(nn,t);
   WaBSum.add(WaBDeltas);
}

WaBSum.divide(trainingExamples.length);
nn.adjustWeightsAndBiases(WaBSum);

Thank you


texts: Difference between shap values and feature contributions

texts:
I always found both concepts a bit confusing since they are quite similar. Would someone provide clear example where to apply each?

Shap values ref: https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d


texts: Classification/Regression Problem where Response Variable is Ordinal

texts:
Data Science novice here!

I'm trying to work on the white/red wine quality data set, where I 'm trying to predict the quality of the wine. All the features are numerical.

The response variable however, is ordinal with a quality score of integers 1 through 10 . I have seen tutorials try to group the scores ie: (0-4: Bad, 5-7: Good, 8-10: Great), but what if I wanted to predict the score as it is?

Should I use a regression approach where I try and minimize the error of my predicted scores versus actual scores?

Or should I use a classification model anyways and instead of calculating a F-score to evaluate the model, find the model that minimizes a cost function?

Or perhaps there is another approach that works best?


texts: CNN Back Propagation without Sigmoid Derivative

texts:
I'm new to CNN and trying to study some MATLAB sample codes (cause I need to know the internal calculation). I recently realized that the sample code I'm using doesn't multiply error by sigmoid's derivative in back propagation. The feed forward process has sigmoid as last layer's activation function so from my understanding, back propagation error = (outputs - target) * sigmoid's derivative(outputs). However, the author intentionally disabled this multiplication with the following code:
if cnn.loss_func == 'cros'
    if cnn.layers{cnn.no_of_layers}.act_func == 'soft'
        cnn.CalcLastLayerActDerivative = 0;
    elseif cnn.layers{cnn.no_of_layers}.act_func == 'sigm'
        cnn.CalcLastLayerActDerivative = 0;
    end
end

My reference code.
When cnn.CalcLastLayerActDerivative = 0, error is defined just as (outputs - target). I tried to initialize cnn.CalcLastLayerActDerivative = 1 so that sigmoid's derivative is considered in back propagation but then I got worse error rate. I'm not sure whether it's just because sigmoid's derivative is in the range [0,0.25] or I'm not understanding back propagation correctly. Does anyone know why this is happening and whether I should add sigmoid's derivative in my calculation?
Thanks!


texts: Location Invariance not achieved in CNN in spite of 99% test accuracy

texts:
I have been trying to decode a captcha using CNN. The number of training samples is 1,82,000. So far, I could achieve 99% accuracy on training as well as test set.
The issue however is that LOCATION INVARIANCE is not achieved while training. The captcha text if slightly shifted horizontally gives bad prediction.Below image is one of the samples on which network was trained and gives good prediction accuracy for similar images (Accuracy~99%) .

But if following image is given , it gives some random prediction.

The only difference between above two images is location of captcha text (apart from the actual captcha value.)
As far as I know, the usp of CNN is the fact that location of a pattern doesnt matter. So what could be the possible reasons for this ?


texts: How to train f(x)=x*x using Artificial neural network?

texts:
let's take some training data of size 100

x_input = [1,2,3,4,.....,100]
y_label = [1,4,9,16,....,10000]

Now, let's consider that we don't know the function f where f(x_input) = x_input2
How should we train it?


texts: Model-agnostic variable importance metric

texts:
I use genetic / evolutionary algorithms in python's TPOT package to find the overall best model (GBM, RF, SVM, elastic net, etc) and its tuning parameters. Now I need a way to measure each variable's contribution to the chosen model's predictive performance. How can I do this in a model-agonistic way?

My current approach is to retrain the best model architecture after holding out each of the variables. For example, if my variables are [a,b,c] I'll retrain on [a,b], [a,c],  and [b,c]. I define the removed variable associated with the worst performing model as the most important variable and I define the variable's predictive contribution as the decrease in predictive performance. I measure all variable's predictive performance this way. Is there anything obviously wrong with this approach? Is there a better approach? I'm familiar with variable importance in decision trees, and p-values in linear models, but I need a model agnostic approach.


texts:
I want to train a model (either classification or regression, doesn't matter) with features/inputs of different sizes, but I am not sure how to do it.

For example, for each data-point, feature 1 and 2 are just real numbers and feature 3 is a vector of length 100, so if I have $n$ data-points, then I have a $n \times 1$ vector for feature 1, another $n \times 1$ vector for feature 2, and a $n \times 100$ matrix for feature 3.

The first question is what kind of model I should use or try? Gradient boosting, random forest, or neural network?

The second question is how to feed the features of different sizes to the model? Just concatenate them to form a $102 \times 1$ vector for each data-point?

Thanks.


texts: Sklearn - multiclass text classification

texts:
I took a challenge to classify bugs (by scanning their logs) into a different groups (classes).
I've already accomplished part of the task by fetching + cleaning the data, but I'm now stuck on getting a very low score of prediction.
I feel like I'm missing the basics and I need some direction.
2 questions I have:

should I balance the dataset I'm using before feeding it to the classifier?
e.g. lets say I've the below mapping of :

{A: 100, B: 20, C: 350, D: 120, E: 220}
Should I leave it as is?, should I balance so each class type will hold equal amount of bugs?
I've seen some demonstrations of classifying Spam/Not spam or Cats/Dogs where it was required, but on the other hand I've read here: https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f
That I should leave it as is.

I've divided the features into ~5 different columns and I'm not sure it's the way to go.
For instance, I've:

errors, python traces, other traces, test traces, test errors ==> label
Is it too much? Does having too many features set damage the prediction?
Now matter what I've done (using balanced/unbalanced dataset) gave me very low score (never passed 40%)
I've been using TFIDF vectoriser and AutoML for testing many classifiers at once.


texts: How to interpret a random variable in the variable importance?

texts:
I have a problem, for simplicity let's say it is a binary classification problem.

I am trying to solve this problem using XGBoost.

A standard output plot for any ML algorithm, is the feature importance, and I want to look at the top n features.

But how to decide on n ?

A method would be to add a random variable (a vector with numbers from uniform distribution) in the XGBoost model, and then see in which place this random variable lands, and only look at the variables above the random variable.

If I do that, and then I re-run the model, including only the variables that score above the random one, then the accuracy score in the test set, decreases. Why ?


texts: open source time series sales data for forecasting

texts:
I'm looking for open source time series sales data (past 2-3 yrs or more) that contains at least the following variables.

1. Date: the date of Sales
2. Product: multiple products, sub-types
3. Store / department: multiple stores
4. Promo (week, year)
5. Promo Interval
6. Competition Open Since(month, year)
7. Competition Distance
8. Level of Assortment
9. Number of Customers

Also, the following external variables (not limited to) that may affect sales:

1. Marketing Cost
2. Advertisement Cost
3. Research Cost
4. Inflation
5. Seasonality
6. Weather
7. Is holiday

The goal is to predict sales for next year, and to analyze the effect of external variables on the predicted sales. I tried searching in datasetsearch, kaggle, data.world etc. but yet to find a comprehensive dataset. I'd appreciate if you could give some pointers.


texts:
If I use stacked CNN layers with 3x3 kernels, zero padding, and with no pooling layers, the output feature map will consist of feature vectors, each vector of which is related directly to the original 3x3 block of the input image, right?
Therefore, for example, I could send the output feature map vectors as timesteps to LSTM and it would supposedly learn dependencies between clusters of the image given the image clusters have a temporal dependency.


texts: Which variables matter most for prediction of another variable?

texts:
I have a dataset and need to predict, out of 9 variables, which ones matter most to predict number 10.

I first tried using the selectKBestmethod from sklearn.feature_selection and it looks like 4,5,6 and 8 are best.

Then I tried to look at linear correlation using pearsonr between each of the variables from 1 to 9 and number 10, and I get 2,4,5,8 are most correlated.

However, when I used PCA from sklearn.decomposition and calculate the pca.explained_variance_ I get these values
[  2.13128046e+01   3.44315766e-01   3.26052258e-01   2.67148345e-01
   1.85871921e-01   1.55241540e-01   1.31994073e-01   9.34982387e-02
   1.03297667e-02]

Isn't it a problem that the first variable is so much higher than all the other ones? What does it mean?


texts: How to feed my JSON dataset in Keras for character level text classification

texts:
I have a JSON dataset, for example:

{"candidate00001": "graceful ones."
 "candidate00002":"One more,Marvelous said, sounding royally bored from his seat."
 "candidate00003":"She tired,Joe said, though not unkindly."
}

in which, candidate00001 is my first class, candidate00002 is the second class and so on.

I am new to python, so i want to implement a character level classification by Keras using this tutorial.

my problem is that I don't now how to convert my JSON file as x_train and y_train.


texts: Is this a data issue, or a model issue? A Keras binary classification model

texts:
I've been trying to create a binary classification model that predicts wether there will be a train delay based on the train and time. Here is a link to the data

The issue I'm having is that my accuracy goes to 94.07 in the first 5 epochs. Meanwhile, my class prediction will always be 0 and never 1.

From what I understand, this is "Accuracy Paradox". A symptom of Class Imbalance. To combat this, I implemented Kfold.

kfold = StratifiedKFold(n_splits=10,shuffle=True)
cvs_scores = []

for train,test in kfold.split(X,Y):

  history = model.fit(X[train],Y[train],epochs=50,batch_size=15, shuffle = False, verbose = 1)
  scores = model.evaluate(X[test],Y[test],verbose=0)

  print("%s: %.2f%%" % (model.metrics_names[1],scores[1]*100))
  cvs_scores.append(scores[1] * 100)

print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvs_scores),numpy.std(cvs_scores)))

No luck. Still had the same issue as before.

Here is how I import my data:

raw_data = pd.read_csv('MTA_DELAY_DATA_DUMP - Sheet1.csv')

X = raw_data.iloc[1:-2,0:2].dropna().values
Y = raw_data.iloc[1:-2,2:3].dropna().astype(int).values

My Model:

model = Sequential()
model.add(Dense(32, kernel_initializer='uniform', activation='relu',input_dim =2))
model.add(Dense(16, kernel_initializer='uniform', activation='relu'))
model.add(Dense(8,  kernel_initializer='uniform', activation='relu'))
model.add(Dense(1,  kernel_initializer='uniform', activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.001),metrics=['accuracy'])
history = model.fit(X,Y,epochs=150,batch_size=15, shuffle = False, verbose = 1)

I tried assigning class weights to balance the data out. Even manually deleting 0's in the data, but nothing seems to result in accurate predictions. Am I doing something wrong in the model, or is this simply data that cannot be utilized by machine learning?


texts: How to get probability of classification

texts:
I have the binary classification, I tried several models KNN, SVM, decision tree, and random forest. I have 50 000 samples, X_train has 50 000 rows and 2300 columns. Everything works well, but I want to build some semi-supervised model because I have some unlabeled samples. In this case, I need to get the probability of classification that I tried, but it doesn't work.
At first, I tried it for KNN
from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 1, metric = 'minkowski', p = 2)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]]. I don't understand why it is 1? (as first I thought it is 100%, but I get it for all test samples)
Then I tried it for the decision tree
classifier = DecisionTreeClassifier(random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]] too. Why it is an integer?


texts:
I have the GPS Coordinates of a person performing an activity. The activity has a start GPS point, start time and end GPS point, end time.
The rule is that the difference between the start GPS point and the end GPS point should not exceed 100 metres. If it exceeds, the activity will be marked invalid.
Using Google play services in an Android app, we are capturing the GPS coordinates in an array with timestamp to calculate the 100 metre rule.
During the course of activity, the app gathers around 35-40 readings.
In the readings gathered, some points show a significant spike eventhough the person performing the activity has not moved much.
This results in the distance between the start and end points to exceed 100 metres and hence the activity becomes invalid.
Since we donot have much control on the data gathered, I would like to know if there is a way we can ignore these outliers and go on to calculate the correct distance
Sample reading
p1 p2 0.10 metres
p1 p3 0.50 metres
p1 p4 200  metres
p1 p4 200.5 metres
p1 p5 201 metres
and so on


texts:
I'm trying to run cross validation with mean squared log error with sklearn and getting the following error message:

ValueError: Mean Squared Logarithmic Error cannot be used when targets contain negative values.

This would suggest that I have negative values in my 1d array y. However, I have tried about 10 different ways of checking, including importing into excel and I can see no negative values in there.

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from sklearn.metrics import mean_squared_log_error
import pandas as pd
import numpy as np

train_csv = 'train.csv'
df_train = pd.read_csv(trainData)

# define variables
target = 'SalePrice'
indep_variable = 'OverallQual'

# scoring
scoring_cross_val = 'neg_mean_squared_log_error'
scoring = mean_squared_log_error

# initate model
lin_reg = LinearRegression()

# example data
X = df_train.drop(target, axis=1)
X = X[indep_variable].to_numpy().reshape(-1, 1)
y = df_train[target].to_numpy().reshape(-1, 1)

# fit model
lin_reg.fit(X, y)

# cross validated model error
cv = cross_val_score(lin_reg, X, y, cv=2, scoring=scoring_cross_val)

I created a version of the code above with some simple inputs to check it isn't a bug in my version of sklearn. The code runs without a problem.

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from sklearn.metrics import mean_squared_log_error
import pandas as pd
import numpy as np

# scoring
scoring_cross_val = 'neg_mean_squared_log_error'
scoring = mean_squared_log_error

# initate model
lin_reg = LinearRegression()

# example data
X = np.array([1.,2.,3.]).reshape(-1, 1)
y = np.array([4.,5.,6.]).reshape(-1, 1)

# fit model
lin_reg.fit(X, y)

# cross validated model error
cv = cross_val_score(lin_reg, X, y, cv=2, scoring=scoring_cross_val)

If anyone gets the chance to help, the csv can be downloaded from Kaggle:
https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data

Iain


texts: When the shallow network has a small number of neurons, how it will need more sum-product?

texts:
I am reading a paper, where I am not able to understand the meaning of the following statement, can someone please help.

a shallow network requires exponentially many more sum-product hidden units
than a deep sum-product network in order to compute certain families of polynomials.

Particularity my question is, when a shallow network has a small number of neurons, how will it need more sum-product?


texts: A neural network that keeps learning as new training data becomes available

texts:
I have a text-classification problem with a lot of training data. Running cross-validation takes a lot of time - several days or even weeks. In order to make the system more responsive, I am thinking of the following scheme:

Train a network on instances 1,...,100.
Test it on instances 101,...200; output the accuracy.
Train the existing network on instances 101,...200;
Test it on instances 201,...300; output the accuracy.
And so on.

Ideally, I would like that:

Training on each additional 100 instances will take a constant time (i.e, I will not have to re-train the network on all previous instances);
The trained network will have the combined "wisdom" of all previous instances (so its accuracy will tend to improve with time).

Is this possible to do with standard deep-learning tools (e.g. dynet)?


texts: What are the Most Dissimilar MNIST Digits?

texts:
Using whatever definition of dissimilarity over sets that you'd like, what are the most dissimilar two digits in MNIST? I was thinking that a reasonable approach to answering the question would be to pass the two sets through some state-of-the-art VAE and then check the distances in the latent space.  The dissimilarity metric could be distance to/from centroids of the clusters (easiest) or every point to every point in every other cluster ($O(n^2)$), or one of the other set measures of difference.
Has there been any research done on this (to the extent you even need research to answer the question)?  Does anyone have an idea about what the answer might be? I've seen TSNE plots of the latent space for some VAEs, but they're not exactly rigorous measures of distance or dissimilarity.


texts: Is it possible to use kNN with streaming data

texts:
I have built a kNN model using scikit learn that is able to predict a binary outcome very well.  The data itself is quite basic, it is simply a 1-D waveform.  When feeding the waveform into the model it predicts. However, I would like the data to be streamed through the model if that makes sense, but I am running into issues how to implement.
My first thought was to use a sliding window, but I don't know if that's the best way to go about things because depending on the speed at which data arrives, what if I have a case where there are two distinct peaks in my waveform that require two separate classifications. In that case, if I make my window small, then there is not enough data to make a proper classification. Any guidance or literature on methods would be great.


texts: Correct calculation of BIC (Bayesian Information Criterion) to determine K for K-Means

texts:
I am trying to calculate BIC in python. In python, there is no inbuilt library for computing BIC. I referenced the following link to compute variance and BIC further:- https://stats.stackexchange.com/questions/90769/using-bic-to-estimate-the-number-of-k-in-kmeans?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
The variance formula given in the highest voted answer is not working for my code design as centers is a two-dimensional array and in my case it's a single dimensional array. So I converted the variance formula given in the link to calculate but I learned it's not the correct way.
Here is my python code:-

def compute_variance_grayscale(self, clusters, centroids, gImage, k):
    # compute variance
    variance = 0

    for i in range(len(clusters)): #iterate over the available clusters
        cluster = clusters[i]
        sum = 0;
        centroid = centroids[i]
        for j in cluster: #for each data point in cluster j compute the sum
            sum += (gImage.getImageValueAt(j[0], j[1]) - centroid) ** 2

        variance += sum / (len(cluster) - k); #add the sum to the given variance

    return variance

        def compute_variance_rgb(self, clusters, centroids, cImage, k):
    # compute variance
    variance = 0
    for i in range(len(clusters)):
        cluster = clusters[i]
        sum = 0;
        centroid = centroids[i]
        for j in cluster:
            b,g,r = cImage.getImageValueAt(j[0], j[1])
            b_e = (b - centroid[0]) ** 2
            g_e = (g - centroid[1]) ** 2
            r_e = (r - centroid[2]) ** 2

            sum += (b_e + g_e + r_e)

        variance += sum / (len(cluster) - k);

    return variance

Can anyone please point to the right way to calculate variance?
Also, it would be great if someone could explain to me the computation of BIC and use of const terms as mentioned in the highest voted answer in the above link:-

np.sum([n[i] * np.log(n[i]) -
               n[i] * np.log(N) -
             ((n[i] * d) / 2) * np.log(2*np.pi*cl_var) -
             ((n[i] - 1) * d/ 2) for i in range(m)]) - const_term


texts:
I'm trying to use the recent COVID-19 data from the site of Italian Civil Protection, but they use a rather complicated time format that I'm finding troublesome as a novice to plot as data in a graph.
This is how the data is presented:

[1] 2020-02-24T18:00:00 2020-02-25T18:00:00 2020-02-26T18:00:00 2020-02-27T18:00:00 2020-02-28T18:00:00 2020-02-29T18:00:00

and I would like to use the format as DD-MM, without the time and the year.
How can I do it?


texts: Handwritting Recognition moving from character level to word level

texts:
Given the experience on MIST, I try this problem as a character level.

I have a handwritten text and I want to "OCR" it. Even though I made progresses with openCV (on the image pre-processing, before a DNN classifier).

I think that the best approach would be to move to word level (into RNN). I am writing this in Python. But I don't know how to change the last layer of the DNN into an input layer for RNN.

Is there any code for dummies to see how is this done (for images as input of the first DNN layer)? Besides I see that you could put a language model on top (this would be a dictionary right?). Advice?


texts: how to iterate over certain rows of a certain column an update their value. by applying condition

texts:
I want to update the value of opening stock and closing stock by adding 20 when dcsDep2 < 3.028512.
 here is the snapshot of dataset:

I tried with this code:

for(index_label,row_series) in dep1.iterrows():
    while row_series["Closing Stock"]<3.028512:
            dep1.at[index_label,"Closing Stock"]=row_series["Closing Stock"]+22

but no output is shown.


texts:
My task consists of two points:

1) Make data clustering;

2) Assign new data to the resulting clusters;

I wanted to highlight the boundaries of clusters as min/max values for each coordinate of an observation belonging to the cluster, then assign observations from the new data to a particular cluster in accordance with its boundaries. However, the problem is that cluster boundaries intersect and each observation can belong to several clusters.
What adequate methods can be used to discriminate data in this case?


texts:
I have two groups of images, each one with 1000 samples.

The speckle pattern, in this context, is the same as a random pattern or "white noise" image. So these images are fundamentally different.

In group one, each figure is generated by considering a random function that returns something similar to a speckle pattern (see fig. 1).
In group two we follow the same procedure as group 1, but we plot a small point above that can be positioned anywhere and with any color (see fig. 2).

I want to classify both groups and I already tried to do it with simple neural networks, but I have been unsuccessful.

What is the best technique for this kind of problems?

Fig. 1:

Fig. 2:


texts:
I created a model for Recommending top 10 items to users similar to the approach used here https://github.com/benfred/implicit/blob/master/examples/lastfm.py
I wanted to evaluate the model using NDCG Metric. But I am not sure how to implement it. Can somebody help with it?


texts: Different accuracy for different rng values

texts:
While tuning the SVM classification model in Matlab, I came across the rng function in matlab in which seed (stabilizes the random shuffling of the data in the algorithm) is changed. When the function called is rng(1) then I am getting one accuracy value (99%). When it is changed to rng(2) then I am getting another value (57%). So there is a huge change in accuracy as visible. What does this mean? Am I training it wrong?

The train and test set correct rate (in %) that I am getting with different runs without changing rng are(train,test)
(96, 82.8)
(94.6, 95.3)
(96, 85.9)
(96, 90)
(95, 95)


texts: What is the best way to store images in python for machine learning

texts:
I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values.

So far I have a pandas dataframe that contains the target value, patient id, image id and the path to the corresponding image.

I can access the image by using

io.imread(df['path'])

So it is possible for me to loop through all the images to access them. The question now is, where do I store the images so that I can apply principle component analysis on them?

If I were to simply store it in a dataframe it would contain 7500 columns; 1 for each pixel value. My dataset contains 280,000 images. That means my my dataframe would need to be 280,000x7500. I Feel that there is a better way to approach this problem.

Your input to this matter would be highly appreciated.


texts: What fits in a Data Description Report/ Data Exploration Report?

texts:
So I am trying to get familiar with Crisp-DM and found the terms "Data Description Report" and "Data Exploration Report", which seem oddly vague in their definition. So far I only found this right here:
https://www.ibm.com/support/knowledgecenter/en/SS3RA7_15.0.0/com.ibm.spss.crispdm.help/crisp_data_description_report.htm

But this seems to be on the shorter end in my opinion. Is there any example of a Data Description Report anywhere? If not, is there any systematic methodology you personally use to record your findings while trying to understand data?


texts: Split text into phrases of a person and an operator

texts:
I have about 5,000 texts without punctuation marks.
Each text is a conversation between an operator and a person.
For example: "Hello hello how can I help you how can I find out how much money I have in my account"

How can one separate phrases between a person and an operator using machine learning methods?


texts: Face matching using VGGFace library keras

texts:
I am working on face matching model (matching between id-card faces and selfies), where I am using the resnet50 pre-trained model from VGGFace library, and then retraining all the layers. The training set accuracy is around 97% the validation set accuracy is around 90%. However when I test the model on completely (same image type i.e. a selfie and id-card face) unseen data it fails and the accuracy there is around 65-70%. I am unable to understand why this huge difference between validation set and test set accuracy.
My model code is as follows:
epoch=50
learning_rate=0.0001 #earlier 0.0001,0.001
decay=learning_rate/epoch
batch_size=16 #32,16 earlier
keep_prob=0.8

def get_base_model(input_shape=(224,224,3):
   base_model=VGGFace(model='resnet50',include_top=False,input_shape=input_shape,pooling=None)
   output=base_model.get_layer('add_12').output
   activation=LeakyReLU(alpha=0.3)(output)
   flat=GlobalAvgPool2D()(activation)
   adam = optimizers.Adam(lr=learning_rate,decay=decay)
   final_model=Model(inputs=base_model.input,outputs=flat)

   for layers in final_model.layers:
      layers.trainable=True

   final_model.compile(loss=binary_crossentropy,optimizer=adam,metrics=['accuracy',keras_metrics.precision(), keras_metrics.recall()])

   return final_model

base_model=get_base_model()
selfie_input=Input(shape=input_shape)
pan_input=Input(shape=input_shape)

encoded_selfie=base_model(selfie_input)
encoded_pan=base_model(pan_input)

adam = optimizers.Adam(lr=learning_rate,decay=decay)

L1_layer=Lambda(lambda tensors : K.square(tensors[0]-tensors[1]))
L1_distance=L1_layer([encoded_selfie,encoded_pan])

drop=Dropout(1-keep_prob)(L1_distance)
output=Dense(output_shape,activation='sigmoid',kernel_initializer='glorot_uniform')(drop)
siamese_net=Model(input=[selfie_input,pan_input],outputs=output)
siamese_net.compile(loss=binary_crossentropy, optimizer=adam,metrics=['accuracy',keras_metrics.precision(), keras_metrics.recall()])

siamese_net.fit_generator(generator=train_generator,validation_data=valid_generator,epochs=epoch,callbacks=callbacks,class_weight={0:1.2,1:1.5})

My training size is around 110K unique faces
My validation size is around 7K unique faces
My test set size is around 7K unique faces
The issue that I am facing is when I save the model using save_model()
and then I predict (by loading the model and then sequentially predicting it...in a different script) the values on completely different faces different from those in train, test, validation set it predicts poorly and the accuracy drops to around 65%.
I do not know why such a huge gap.
I think it has something to do with the BatchNormalization (different performance during training and inference) layer as mentioned here but I am not sure.
Please help me out.
PS : I train my models on google colab GPU and run the evaluation script on an AWS ec-2.
Keras version 2.3.0 and tensorflow version 1.15 (gpu)


texts: Using Low Frequency Labels with High Frequency Features

texts:
I am trying to build a model (most likely a regression or random forest regression) for quarterly financial data. My training data has a daily cadence, but I am not sure how to work with these to predict a quarterly dependent variable. Clearly I could summarize these by quarter (avg, mean, etc.) but that would result in few total observations, and it feels like that is throwing out some valuable variance to train on.
Thanks for the help!


texts: Does NER work on large documents around 1500 - 3000 words or so?

texts:
Let's say I have a resume and I have segmented the work section.
Usually work section of resume contains company name, designation, work period and job description. Now I have 1000's of resumes and I have manually annotated the work section of each resume with those 4 labels.
But the problem here is each work section is quite large around 1400- 3000 words?
Also job-description annotation is not one word like company name etc...  the entire responsibilities of the job is annotated as one entity.
So, will this work? please let me know what are the things I should consider for such NER entity extraction?
Any suggestions would really help. Thank you in advance.


texts: How to decide the number of trees parameter for Random Forest algorithm in PySpark MLlib?

texts:
I am working on Random Forest algorithm in PySpark MLlib and have a doubt regarding the number of trees parameter that we pass to the model. The standard format of Random Forest modeling in PySpark MLlib is:

model = RandomForest.trainRegressor(trainingData, categoricalFeaturesInfo={},
                                    numTrees=3, featureSubsetStrategy="auto",
                                    impurity='variance', maxDepth=4, maxBins=32)

The doubt that I have is how to decide the optimum value of trees to pass to numTrees parameter? I assume the more the number of trees better should be the performance but would it keep on improving with increase in number of trees? Is there a point after which it will start to have negative impact in performance? If yes then how can I find the optimum number of trees for my dataset?


texts: What does high variance mean in a binary classification machine learning model?

texts:
My understanding of high variance is that the targets are spread widely around. The output values are "all over the place".

In a binary classification model, there can only be 2 outcomes. I am at a loss when visualizing what high variance mean in a binary classification model.


texts: Classify driver based on time-series sensor data

texts:
I want to build a model that can detect which driver is driving now the car based on a dataset that contains 20 driver records for 3600s each driver ( the dataset contains all the car sensors values every second ) .

So , now i have that dataset that contain the drivers records .
How can i train a model that can identify the driver based on 60 seconds ( or more for example).

Means , i want to make predictions with a dataframe of rows and not a single row .
because we can't identify a driver with a single row .

:)


texts: Getting uniform distribution over topics from gensim's LDA?

texts:
I am trying to learn topics distribution for each document in a corpus.

I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100)  and when I try to infer vectors for each document I am getting uniform distribution over them. This is highly unlikely since documents are of different topics.

The relevant code snippet is:

#input : scipy sparse term-doc matrix (no_terms * no_docs)

corpus = gensim.matutils.Sparse2Corpus(term_doc)

lda = gensim.models.LdaModel(corpus, 100)

vec_gen = lda[corpus]

vecs = [vec for vec in vec_gen]

Now for each vector in vecs I am getting same probability for each topic.

Can anyone point out where I am going wrong?


texts: Does Box plot with many outliers effects the result?

texts:
I am using Haberman's cancer survival dataset https://www.kaggle.com/gilsousa/habermans-survival-data-set
to draw a box plot. Here Surv_status is the target variable which has two classes and axil_nodes_det is the feature. I am getting the plot as follows with too many outliers from the first class

I want to know whether more outliers will affect the prediction of output


texts: How does Latitude and Longitude be helpful in making the Venues/Places Recommendation system?

texts: I am trying to build a recommendation system which suggest the places on the basis of their ratings , reviews etc . I want to use Latitude and Longitude , but I don't know how it will be helpful in building the venues recommendation system ?


texts:
Hi I am trying to train a CNN to differentiate between pictures of dogs and pictures of cats. It does not seem to learn anything no matter how I change the architecture. I have used the following code to create the datasets:
train_generator = tf.keras.preprocessing.image_dataset_from_directory(
    G:/Dogs_cats/Train',
    labels="inferred",
    label_mode="binary",
    class_names=None,
    color_mode="rgb",
    batch_size=150,
    image_size=(350, 350),
    shuffle=True,
    seed=1)
   # validation_split=0.2,
   # subset='validation',
   # interpolation="bilinear",
  #  follow_links=False,)

validation_generator = tf.keras.preprocessing.image_dataset_from_directory(
   'G:/Dogs_cats/Validation',
    labels="inferred",
    label_mode="binary",
    class_names=None,
    color_mode="rgb",
    batch_size=150,
    image_size=(350, 350),
    shuffle=True,
    seed=1)

I then created my CNN model:
model = tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(60, (3,3), activation='relu', input_shape=(350, 350, 3),padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.MaxPooling2D(2, 2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Conv2D(5, (1,1), activation='relu', padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.Conv2D(60, (3,3), activation='relu',padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.MaxPooling2D(2, 2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Conv2D(60, (3,3), activation='relu',padding='same',kernel_initializer='he_uniform'),
        tf.keras.layers.MaxPooling2D(2, 2),
        tf.keras.layers.Dropout(0.3),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(412, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid')
])

from tensorflow.keras.optimizers import Adam

model.compile(loss='binary_crossentropy',
        optimizer=Adam(),
        metrics=['BinaryAccuracy'])

and trained it:
history = model.fit_generator(train_generator,
                              epochs=450,
                              callbacks=[early_stopping],
                              verbose=1,
                              validation_data=validation_generator)

However the model doesn't improve at all or improve its accuracy on either the training or validation datasets.
Epoch 1/450
134/134 [==============================] - 116s 846ms/step - loss: 3993.2059 - binary_accuracy: 0.5017 - val_loss: 0.7324 - val_binary_accuracy: 0.4792
Epoch 2/450
134/134 [==============================] - 117s 845ms/step - loss: 0.8022 - binary_accuracy: 0.5158 - val_loss: 0.7285 - val_binary_accuracy: 0.4784
Epoch 3/450
134/134 [==============================] - 99s 719ms/step - loss: 0.7572 - binary_accuracy: 0.5128 - val_loss: 0.7252 - val_binary_accuracy: 0.4762
Epoch 4/450
134/134 [==============================] - 92s 666ms/step - loss: 0.7219 - binary_accuracy: 0.5165 - val_loss: 0.7219 - val_binary_accuracy: 0.4750
Epoch 5/450
134/134 [==============================] - 93s 671ms/step - loss: 0.7132 - binary_accuracy: 0.5177 - val_loss: 0.7134 - val_binary_accuracy: 0.4734
Epoch 6/450
134/134 [==============================] - 93s 677ms/step - loss: 0.7080 - binary_accuracy: 0.5160 - val_loss: 0.7009 - val_binary_accuracy: 0.4826
Epoch 7/450
134/134 [==============================] - 93s 675ms/step - loss: 0.6989 - binary_accuracy: 0.5205 - val_loss: 0.7013 - val_binary_accuracy: 0.4856
Epoch 8/450
134/134 [==============================] - 93s 676ms/step - loss: 0.6947 - binary_accuracy: 0.5162 - val_loss: 0.7015 - val_binary_accuracy: 0.4852

What can I do to make the CNN learn? I am not sure if there is a problem with the way I am loading my dataset or with the way I am running the model.


texts: Reshaping big dataset with MinMaxScaler giving error

texts:
My data set is of shape (1249, 228). Most of the entries are zero and other are integers like 1,2,5,10,20 etc. I want to transform this set for the input into LSTM. But when I am applying MinMaxScaler. It is giving the following error:

load the dataset:

dataset1 = pd.read_csv('g:/hello.csv', engine='python')
dataset1= dataset1.drop('packages', axis=1)
dataset1 = dataset1.astype('float32')

normalizing the set

scaler = MinMaxScaler(feature_range=(0, 1))
dataset1 = scaler.fit_transform(dataset1)

  ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

How can I transform this data set according to the input in LSTM.


texts: How to represent audio data in a format that can be used for preprocessing and modelling?

texts:
I have a project that I am working on currently. The project is to classify audio data. The data is in two folders train and test. In the train folder, there are multiple folders like cat, dog, elephant etc and these folders contain multiple audio files. For example, there are multiple audio files inside the folder dog, cat etc. The folders are also the class names representing the class the audio belongs to. The audio is in .WAV format. How to represent this data in a format on which I can use Machine learning models?

To explain my question a bit: let's take a .csv file. We have rows and columns. Rows are data points and columns are features and training machine training models on this data makes sense.

So, How do I represent this audio data that I can work with and fit ml models?


texts: how is correct usage of the validation split in neural networks?

texts:
I have a dataset separated in train, test and validation splits.

After each epoch, I evaluate the loss and accuracy in the validation split.

When the loss in validation split is not better, I stop train and choose that as final model.

But, I should merge train and validation as final model? How can I choose the best model?


texts: Dimensionality Reduction for Function Fitting method using Kernels

texts:
I have a set of continuous noisy measurements $x_i \in R^n$ with $i=1,...,N$ for which I know the value range, i.e. $x_{min} \leq x_i \leq x_{max}$. Corresponding to the measurements, I have a set of outputs $y_i \in R^m$. To give you some numbers: $n=10$, $N=100000$ and $m=2$.
I would like to find a continuous nonlinear function such that $y \approx F(x)$. Before running any interpolation method however, I realized to have a lot of redundant samples (very close to each other). I'm currently applying a binning of the samples and keeping one random sample per bin (after normalizing the data in [0,1]). The number of samples goes from 100000 to 10000 approximatively.
Question: Can someone suggest an alternative approach for removing redundant data maybe based on the distance between samples?


texts: can a perceptron be used for regression?

texts:
In machine learning, the perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to some specific class.

on wiki page, the problems section includes regression without more explanation.

can a perceptron used for regression?


texts:
I've started studying decision trees, and I noticed that the examples online used categorical features to split the data at each node. I'm working on data sets with a binary classification output and a collection of categorical and continuous features. I was wondering: how would one would decide where to split continuous data for a decision tree node?

Since a continuous feature would exist on a single-variable interval, my idea was to just consider those points which existed in the "overlap" region between the two labelled groups. I would start with an initial estimate (the midpoint of the overlap region). Then, I would move my boundary line next to another point in the overlap region. I would calculate the error rate from the confusion matrix that this classification would yield. If the error rate is lower than the one from the current boundary estimate, I replace my estimated boundary with the new boundary that was just proposed. I repeat this process for each point in the overlap region until I've found a partition which minimizes the error rate.

Is this a reasonable way to split continuous data? I've seen people talk about Gini impurity, although I do not understand why one would use that instead of error rate. I've implemented my described algorithm in python and it seems to do as good a job as is possible for reasonable data sets.


texts: Implementation of custom loss function invariant to batch size

texts:
When Implementing custom loss function how to make it invariant to the batch size. For example lets say dice loss is being implemented. The formula for dice loss is:
$$
\sum_{c}{} 1 - {DSC}_{c}\\
c: \text{Classes},\hspace{5mm} {DSC}_{c}: \text{Dice score of class }c
$$
This formula does not clarify how to deal with the batch size. It is worth noting that according to this formula, the value of loss will increase if the batch size is increased. The obvious intuition would be to normalize the loss value using batch size. I have two question in this context.

Is it theoretically valid to normalize the loss using the batch size?
How to normalize?

For example, In case of image the tensor at hand is 4D. The dice score / loss can be calculated for each image and each class resulting into a 2D tensor like below, where $n$ is no. of classes and $m$ is the batch size.

$$
\begin{bmatrix}
l_{1, 1} & \dots & l_{1, n}\\
\vdots & \ddots & \vdots\\
l_{m, 1} & \dots & l_{m, n}
\end{bmatrix}
$$
      Now the aggregated loss can be calculated as:
$$
\sum_{j=1}^{n}\frac{1}{m}\sum_{i=1}^{m} l_{i, j} = \frac{1}{m}\sum_{j=1}^{n}\sum_{i=1}^{m} l_{i, j}
$$

Alternatively, we can ignore the batch size from the beginning and generate per class loss in a 1D tensor:
$$
\begin{bmatrix}
l_{1}^{'} & \dots & l_{n}^{'}
\end{bmatrix}
$$

      Accordingly, the normalized aggregated loss will be:
$$
\frac{1}{m}\sum_{k=1}^{n} l_{k}^{'}
$$
Is there any fundamental difference between these two approaches? Which one is correct (if any)?

Related Discussion on GitHub


texts: Working with few instances of specific target feature over large dataset

texts:
I have data over a single, a machine includes different components, all the parts are interacting, the data are tracked for those parts, it tracks power consumption and many other relevant feature measurements. The longevity of the machine, from the seller, is specified to be around 2000-15000 hours. Due to this large range, it would be ideal to constrain the target for machine failure, with machine learning. Because we have tracked all the data, with the different components in the system. One specific component fails.  Over a period of 9 months, the machine fails 5 times. Monthly average about 3-4 month. It is this, specific target we want to predict when the chances for failure are high.

Thus the specific instances for failure are limited. How do we deal with this, from a Machine Learning perspective, as opposed to giving a monthly average, it is not a safe bet? How do we train on limited instances for failure? This problem might not have one right answer, but many possible advice.


texts: splitting into train test by train_test_split of float values?

texts:
How to split into train test by train_test_split of float values ?
I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ValueError: The least populated class in y has only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2., what is the better solution?

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=seed )

    from imblearn.under_sampling import RandomUnderSampler
    from collections import Counter

    rus = RandomUnderSampler(random_state=seed, return_indices=True) # (50:50)

    from sklearn import preprocessing
    lab_enc = preprocessing.LabelEncoder()
    training_scores_encoded = lab_enc.fit_transform(y_train)
    #print(training_scores_encoded)

    X_res, y_res, idx_res = rus.fit_sample(X_train, training_scores_encoded)

    from sklearn.preprocessing import StandardScaler

    scalerT = StandardScaler().fit(X_train[:,[0]])
    #scalerT = StandardScaler().fit(X[:,[0]])

    X_train[:,[0]] = scalerT.transform(X_train[:,[0]])
    X_test[:,[0]] = scalerT.transform(X_test[:,[0]])
    X_res[:,[0]] = scalerT.transform(X_res[:,[0]])

    scalerA = StandardScaler().fit(X_train[:,[idx_Amount]])
    #scalerA = StandardScaler().fit(X[:,[idx_Amount]])

    X_train[:,[idx_Amount]] = scalerA.transform(X_train[:,[idx_Amount]])
    X_test[:,[idx_Amount]] = scalerA.transform(X_test[:,[idx_Amount]])
    X_res[:,[idx_Amount]] = scalerA.transform(X_res[:,[idx_Amount]])

    from sklearn.linear_model import LogisticRegression

    models = []
    models.append(('LR', LogisticRegression()))

    # Default parameters for each model
    models

    from sklearn.model_selection import cross_val_score
    from sklearn.model_selection import StratifiedShuffleSplit

    XX = X_res
    yy = y_res

    #StratifiedKFold is used by default for cv integer
    #cv=10
    cv = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=seed)
results = []
names = []

for name, model in models:
    cv_results = cross_val_score(model, XX, yy, cv = cv, scoring = 'accuracy')
    results.append(cv_results)
    names.append(name)
    msg = "%s: %.2f (+/- %.2f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)

fig = plt.figure()
fig.suptitle('Accuracy')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()

results = []
names = []

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedShuffleSplit

XX = X_res
yy = y_res

#StratifiedKFold is used by default for cv integer
#cv=10
cv = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=seed)

for name, model in models:
    cv_results = cross_val_score(model, XX, yy, cv = cv, scoring = 'recall')
    results.append(cv_results)
    names.append(name)
    msg = "%s: %.2f (+/- %.2f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)


texts:
I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, whether makes sense to use my trained model to make classification on top of the new dataset. The problem is the new dataset has no assigned target value yet, so I can't calculate accuracy or another metric to check the model's performance. I assume that the model will perform well if the distribution of the new data is similar to the old one. E.g. If I train my model on Wikipedia articles, I shouldn't expect my model to work well on tweets, which are way shorter (max 160 characters) and contaminated by a big number of non-obvious and rare abbreviations.

In the case of tabular data, the task is pretty simple. I can visualize each of the features, perform a z-test, etc.
What about the text data?

My idea is to leverage such quantities as:

Distribution of the nb of tokens;
Distribution of the length of the input strings;
Number of the common words;
Distribution of the percentage of OOV (out of vocabulary words);

Is there any other, more robust method?


texts:
Hi i am getting this error:

  Error in if (x < min) { : missing value where TRUE/FALSE needed
  In addition: Warning message:
  In if (x < min) { :
    the condition has length > 1 and only the first element will be used

I have seen this error in others post but his solutions don't fix it,
I think this may be a different case

My code:

min <- data[1,2]

for(i in data){

  if(data[i,2] < min){

    min <- data[i, 2]

  }

}

The column that I am using(the second) all are numeric values

Thanks!


texts: Bidirectional LSTM usage with sensor data

texts:
I am applying a deep LSTM network in order to classify time-series data from different sensors. In the field (energy) I often see the research using bidirectional LSTMs for forecasting. I don't get it how can we forecast future time steps if we need the "future" for bidirectional flow through the network. I assume creating a control algorithm using bidirectional LSTM doesn't make sense if we are interested in the current situation? For classifying past data I understand the point of bidirectional networks. Or did I get it wrong?
P.S. in my use case I get better results using unidirectional than bidirectional network, even though similar research used bi-network.


texts: How can I apply K means algorithm to detect a pattern?

texts:
I have a dataset:

How can I apply kmeans algorithm to find clusters based on "date" column . So that I can retrieve tweeting/retweeting activity every hour and generate a pattern.
It would be great if someone can help me out.
Thank you.


texts: Normal equation for linear regression is illogical

texts:
Currently I'm taking Andrew Ng's course. He gives a following formula to find solution for linear regression analytically:
$θ = (X^T * X)^{-1} * X^T * у$
He doesn't explain it so I searched for it and found that $(X^T * X)^{-1} * X^T$ is actually a formula of pseudoinverse in a case where our columns are linear independent. And this actually makes a lot of sense. Basically, we want to find such $θ$ that $X * θ = y$, thus $θ = y * X^{-1}$, so if we replace $X^{-1}$ with our pseudoinverse formula we get exactly the $θ = (X^T * X)^{-1} * X^T * y$.
What I don't understand is why nobody mentions that this verbose formula is just $θ = y * X^{-1}$ with pseudoinverse. Okay, Andrew Ng's course is for beginners and he didn't want to throw a bunch of math at students. But octave, where the assignments are done, has function pinv() to find a pseudoinverse. Even more, Andrew Ng actually mentions pseudoinverse in his videos on normal equation, in the context of $(X^T * X)$ being singular so that we can't find its inverse. As I mentioned above,  $(X^T * X)^{-1} * X^T$ is a formula for pseudoinverse only in case of columns being linear independent. If they are dependent (e.g. some features are redundant), there are other formulae to consider, but anyway octave handles all these cases under the hood of pinv() function, which is more than just a macro for  $(X^T * X)^{-1} * X^T$. And Andrew Ng instead of saying to use pinv(X) * y gives this: pinv(X' * X) * X' * y, basically we use a pseudoinverse to find a pseudoinverse. Why?


texts:
From my very basic understanding, it looks like a lot of the machine learning techniques for graphs depend on information at the level of nodes. My problem is something like this:

Suppose I want to generate a graph which is square-free, on $n$ nodes. I am able to generate a lot of smaller graphs which are triangle-free, and maybe a few which are square-free, and want to use this as training data in the hopes that a neural network or something similar is able to learn some inherent "structure" which will allow it to produce an $n$ node square-free graph.

Are there techniques in machine learning which will allow me to do something like this? Could someone point me to specific papers where a similar problem is tackled? My suspicion is that GANs could help with this problem, but I wouldn't know where to start.

Furthermore, the way I see it, determining whether or not a graph is square-free goes beyond the level of nodes alone, and we have to consider subgraphs in this context. Does this make techniques with graph neural networks moot for this problem? I could see how you might be able to encode some information at the node-level (such as the degree of each node), but it seems redundant if you have the full graph structure.


texts: What is bad, good and excellent metric score for time series model?

texts:
I have created a couple of models for my master project and I used several metrics for evaluation. I used MSE, MAE, MAPE, RMSE not because I really learned about them a lot, because I saw in many other projects these metrics being used.
Now I have a problem, I need to interpret results. I search for some articles or some studies that classify metrics performance as good or bad or excellent. The only material I found now is this one.
With this interpretation

MAPE <     10% is Excellent, MAPE <     20% is Good

It would be nice to provide this formal or informal interpretation for each of the metrics MSE, MAE, MAPE, RMSE specifically for time series prediction problems.


texts: A good machine learning approach for distribution of a whole?

texts:
So I had done with different classification, regression and clustering approaches for predictions of values etc. I was wondering if there is a machine learning approach for distribution of a whole based on on some features (I do not know if there is an approach for that I just could not find one with my research).

An easy example might be lets consider we have height and weight data of many children and we have to distribute a given number of pizza slices amongst them so that skinny children gets more pizza as compared to obese ones because pizza is more beneficial for skinny as compared to obese. So might have to find out the optimum number of slices for each child out of the total number of slices so that each child gets maximum possible nutrients. A more complex version could incorporate more features like age, overall health, blood sugar content, physical activity index, daily calorie consumption and others.

A similar example might be to find out the optimal value of fuel to be allocated to each vehicle if we have a total of 100 gallons. Features might be distance they have to travel, mpg, driver competency, engine horse power etc so that all of them might travel the maximum distance possible.

So can we achieve a task like this with machine learning/deep learning approaches? If not what are the hurdles achieving this?


texts: Help me choose a Data Science book in Python

texts:
I've been a Data Scientist for a few years now, but I've only recently started to do most of my work in Python (boy, do I miss ggplot2! But altair is coming to the rescue).

I want to improve my Python skills and, since most of my work is related to developing Data Science & Analytics applications, I'd rather learn from a book about these topics than from a book on, say, Application Server frameworks. Also, since I mostly develop Deep Learning models, I was looking for a PyTorch book. However I could only find two, of which one is really crappy, and the other one is from authors I really respect, but on a topic I don't work with (NLP):

https://www.amazon.com/Natural-Language-Processing-PyTorch-Applications/dp/1491978236

So I've done a bit of research and found about the following books:

Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems

Data Science from Scratch: First Principles with Python

Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython(second edition recently came out)

Python Data Science Handbook: Essential Tools for Working with Data

Which book would you choose? You're free to suggest other titles. Requirements, in order of importance:

It should be a good book about Python, so the code should be Pythonic
It should be about about PyTorch, but not only about NLP (this requirement will probably be unfulfilled, for reasons above). Not about Keras, or mainly about Keras: I already have a reference for that.
It should be about Data Science

Price is not an issue.


texts: How to Fit S shape (Sigmoid Function) in my scatterplot

texts:
How to interpret my chart? I want to get the maximum likelihood in logistic regression with this result (I'm really not sure if this is how it looks like):

I am currently using logistic regression to National Achievement Test(a performance exam for students,NAT -GRADE-REMARKS the Y axis) and their scholastic grade(In the example above 2CLPW(21st Century Literature in the Philippines and the World)-(Grade 12)-Q1(Quarter 1), the x Axis).I am using Rapidminer.
I wanted to know the likelihood of students to Pass the National Achievement Test or to get VLM or LM. For my example in the image above the category in the National Achievement Test is VLM(Very Low Mastery) which is set to 1 and the LM(Low Master) Set to 0. VLM and LM are the only categories all the student gets.
I wanted to know the maximum likelihood in this graph in order to fit a S like line (sigmoid function).Can I fit a sigmoid S shape? If there are no relationship then that means I can't fit an S shape line right? I just really dont know how to interpret the scatterplot above. Do I need to zoom it out? I really cant interpret it.Green is Very Low Mastery, Blue is Low Mastery


texts: Training a CNN to convert ellipses into circles

texts:
My current project has to do with modeling the effects of blurring/convolution of objects in various imaging processes. Right now, I am starting off with a preliminary, artificial model. I am using Keras to accomplish this.

I create artificial ideal data as a set of circles in random locations in a 128 x 128 image. I then have a routine that takes the coordinates and sizes of these circles as input and replaces the circles with ellipses at the same locations.

I am trying to train a convolutional neural network to perform the inverse of this function, i.e. read the images with ellipses as input and replace them with circles. I create training pairs by grouping these two image types together with the images with ellipses as inputs and the images with circles as outputs.

How can I design a neural network which accomplishes this? Right now, the one that I am using merely returns distorted copies of the images of ellipses instead of outputting images with circles.

I have played around with multiple different CNN structures, as you can see in the commented-out code:

model = Sequential()

#model.add(Dense(10, activation = 'relu', input_shape = (80, 128, 128, )))
model.add(Conv2D(nb_classes, kernel_size=3, padding = 'same',
                 activation='relu',
                 input_shape=(128,128,1)))
#model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
model.add(Conv2D(32, kernel_size = 3, activation='relu', padding = 'same'))
#model.add(MaxPooling2D(pool_size=(2, 2), padding='same'))
#model.add(Dropout(0.25))
#model.add(Flatten())
#model.add(Dense(128, activation='relu'))
#model.add(Dropout(0.5))
#model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))

#model.add(UpSampling2D((4,4)))

model.add(Conv2D(nb_classes, (3, 3), activation = 'relu', padding='same'))
#model.add(Dense(10, activation = 'relu', input_shape = (128, 128,)))
model.add(Activation('softmax'))


texts: How to predict number of orders for coming weeks using previous week's data?

texts:
I'm getting confused as how to proceed for this problem:

I need to predict a city's food supply, for number of orders, for the upcoming weeks based on previous weeks' data. What should I do? Do I need to use a time related algorithm or a simple algorithm?


texts: How to choose an appropriate Machine Learning algorithm?

texts:
I am currently new in Data Science field especially in ML domain. There are various approaches to a problem ie. Supervised, Unsupervised learning. I want to know what is the approach and how to determine using exactly which algorithms in real life test cases.

Even if a particular algorithm is selected, how do we know other algorithm might have better accuracy and precision than this one


texts: Exploratory analysis and feature engineering for time till failure prediction using sensor data of engines

texts:
I am trying to do some data exploration and analysis on a dataset of engine sensor readings. I would like to determine if the data I have is good enough to predict a time till failure and possibly classify the type of failure.

Background:

The data is collected at random time intervals of about 5 to 10 minutes in 1-minute bursts. Each 1-minute burst contains 4 groups of distinct sensors for about 15 seconds each. These groups are organized by sensors commonly used together for manual diagnosis by a technician. The idea is to hopefully be able to capture important relationships between these sensors and the 'degradation' over time. There are many engines that are missing data because of unreliable communication interfaces during collection. Another layer of complexity is that there is not a feature for defining the type of each engine. EX: Manufacture or different engine stats
There are not a lot of data on engines with a failure as there are only 800 engines, in 8 possible classes of failures, out of the 20,000 engines.

I might have gotten ahead of myself, but here is what I have done so far:

resampled the data into days by getting the min, max, mean, std, 25%, 50%, 75%
graphed a correlation matrix
graphed some sensor data to verify the data shows signs of 'degrading' over time
ran some simple ML classifiers and regressors to get a baseline auc and accuracy
used some unsupervised learning techniques to create new features
EX: used dynamic time warping as a distance metric for DBscan clustering

Right now, I don't see much correlation between the engineered features and didn't get many good results with any supervised ML. I have not yet tried LSTMs or variations on the OneVsRest classifier, but from what I have read those are popular techniques for these types of problems.

Any ideas how to explore the data further?
Any ideas how to create new features, resample, or extract the relationships of the sensor of time series data?


texts: Why am I getting accuracy of Xgboost model 0.00%?

texts:
I am trying to build Job Recommender System using Deep Learning.

dataset used
From this dataset i have taken only users.tsv, user_history.tsv, apps.tsv and jobs.tsv to build a hybrid recommender system.

The steps i have performed till now are:

One hot encoding on this dataset
Embedding on one hot encoded features to reduce dimension and sparsity
Concatenated all the 4 datasets(Embedded matrix and non-categorical features of dataset)
Created Xgboost model for feature regularization

But I am getting accuracy of xgboost model equal to 0.00%. After this i also need to perform CNN to get the top-k recommendations, but since i am getting accuracy of xgboost model zero i cannot proceed further. So, please help me.

Google drive URL to my code:
code URL


texts: Can HDF5 be reliably written to and read from simultaneously by separate python processes?

texts:
I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using click to gather the data.

My concern is what will happen if the data gathering script is writing to the HDF5 file and the yet-to-be ML application tries to read data from the same file?

I took a look at The HDF Group's documentation about HDF5 parallel I/O, but that didn't really clear things up for me.


texts:
I have a dataset with noisy labels on which I train a binary classifier. Inspecting the loss I see some samples were misclassified with high confidence and others were classified with indecision around 0.5 confidence.
If I have a budget to inspect wrongly predicted samples and potentially relabel them, is it better to pick the ones where the model was certain but got it wrong or the ones where the model is uncertain?

On the one hand, if I select the samples where the model has high confidence it could either be a sample which was classified correctly but has a bad label or the sample was indeed misclassified. The latter is a very valuable sample, because the classifier misclassified with high confidence. The former however is useless, as the model was in fact correct.
On the other hand, if I select the samples where the model was uncertain I will always gain insights (samples might have 50% bad label ratio, cannot learn label from sample, etc.) but it might not be as valuable as the high confidence false-positives/false-negatives.
Which group should I pick?


texts:
I am confused on how to represent the confusion matrix -- where to put the FP and FN. Link1 and Link2 show different confusion matrix for binary classification. The rows represent the actual and columns represent the predicted values. Based on my understanding, the correct confusion matrix should have been:
                        | Pred Neg| Pred Pos
                        |___________________
Actual Negative(class0) |  TN     |  FP
------------------      |-------------
Actual Positive(Class1) | FN      | TP

where  TN (class 0) indicates the amount of correctly identified normal patterns. TP (class 1) indicates the amount of correctly identified malignant patterns. On the other hand, FP indicates that the classifier predicted the signature to be malignant when infact it was normal.
$  \text{Precision} = \frac{TP}{TP+FP}$,  $\text{Recall or TPR} = \frac{TP}{TP+FN}$,
Question 1: Is my understanding and construction of the confusion matrix correct?
Question 2: What is the intuitive difference between Precision and recall? What happens if precision < recall?


texts:
I have database with three columns, y,x1 and x2:
>>>y      x1       x2
0  0.25   -19.3   -25.1
1  0.24   -18.2   -26.7
2  0.81   -45.2   -31.4
...

I want to create more features based on the x columns.
until now I have just created random functions and tries to check their correlation with the y, but my question is if there is any propeer way/ common functions in order to create thise new features.
I have used PolynomialFeatures of scikit learn but as I understood is not common to do more than 3.
from sklearn.preprocessing import PolynomialFeatures
#split x y....

poly = PolynomialFeatures(3)
poly=pd.DataFrame(poly.fit_transform(X))

My end goal is to use those new columns in random forest algorithm (I have more columns than x1 and x2 but those two that are interesting for me  and would like to investigate them and their relationshop more).


texts: How to correctly interpret the table function to determine logistic regression accuracy?

texts:
I have a set of data, that gives the length of a species of abalone, and its corresponding type (male, M, female, F, or infant, I). (https://archive.ics.uci.edu/ml/datasets/abalone)

I have constructed a logistic regression to create a model that will determine whether the abalone is M/F or I, given the length. (M and F are classed as the same.)

So I write the following in R to generate and test the model on data points:

g <- glm(sex ~ length, family="binomial")
pairs <- paste(round(predict(g, type="response")), sex)
table(pairs)

The output table is:

pairs
0 F  0 I  0 M  1 F  1 I  1 M
218    6  210 1089 1336 1318

How can I correctly interpret this?

These are the options I have come up with:

1089 females correctly identified, 218 females incorrectly identified; 1318 males correctly identified, 210 males incorrectly identified; 1336 infants correctly identified, 6 incorrectly identified.
218 females correctly identified, 6 infants and 210 males incorrectly identified; 1089 females incorrectly identified, 1336 infants and 1318 males incorrectly identified.


texts: classification of bugs ownership by their content (improve score for log analysis)

texts:
I'm doing a little project in which given a dataset of bugs and their relevant owner, I'd like to predict the "final owner" of a non-analyzed bug (bugs tends to assign back and forth between different owners and I'm referring to the "real/final owner" as the final one based on the dataset).
Number of classes/labels is 6
The data I use to make these decisions relays mostly on the bug's content which can be a subset of the following:

Error logs
Python tracebacks
Panics
Panics traces

I'm also using SMOTETomek to balanced the dataset (it creates artificial training data) - this alone increased my results by 10-15%.
However, I'm still getting a very low score (around 65%) when applying a set of classifiers (Each is being trained against the above mentioned data)
Example for classifiers I've tried: SGDClassifier, MultinomialNB, BernoulliNB, LinearSVC, RandomForestClassifier, LogisticRegression, DecisionTreeClassifier, ExtraTreeClassifier, etc.
My pipeline looks as follows:
def train(self, algorithm, X_train, y_train):
    model = Pipeline([
        ('vect', CountVectorizer()),
        ('tfidf', TfidfTransformer()),
        ('smote', SMOTETomek()),
        ('classifier', algorithm)
    ])
    model.fit(X_train, y_train)
    return model

where each algorithm in the pipeline is one of the above mentioned classifiers (it's being run in a loop for each one)
Initially I thought that the root cause for this fairly low score is the way I'm sanitize the data.
For example: For errors, I'm relaying on regex to clean timestamps, thread ID, etc. For python tracebacks I'm getting rid of line numbers and taking just func names
Later on I was surprised to find out that even without cleaning the data, I'm getting the same score, so it made me think I'm missing something
My features are simply text I cleaned from the logs and (each feature is just a long string) and the labels are (each) simply a number that correlates to a class.
How can it be that no matter what I do (cleaning the logs vs leave it untouched) faces me with the same score? Am I using the wrong steps in the pipeline? missing steps?
Any help will be appreciated as I've no clue where/how to proceed.


texts: When is Gradient Descent invoked on the objective function while running XGboost?

texts:
is it at the end of every tree? or only after all trees are build? I tried to think in both ways but didn't get a clear picture.

Can we focus more the part "the loss function is applied between training models" I am trying to understand how is Gradient Descent is applied to minimize the objective function function.

slide no 36 in http://www.saedsayad.com/docs/xgboost.pdf

  Data points reach to a leaf will be assigned a weight. The weight is
  the prediction

how is the weight assigned ? how is the weight calculated ? why predict the weight again ? Can someone shed some light on this?


texts:
I am new to the field of machine learning. I have just recently learnt Decision Trees and started solving Titanic Survival problem from Kaggle Competition. I understood the algorithm behind decision trees and how it actually works but I can't really match it with how python functions work this way. Say what exactly fit method does?

Well correct me if my concept is wrong. I think that looking at the train set i can build a model on "if a passenger survived or not" . Then I split the model into train set and test set and then the fit method (x_train, x_test) (this is what i think it works) actually tries to find the connection between x_train and y_train. I BUILT using my Train dataset and then tries to predict my test set. Say if i change my model, the fit method will try again to find the connection between my x_train and y_train using my new model.
(Sorry for some terms as I am a self-learner. I don't really have the grasp of machine learning terms)

It will be of great help if you could suggest some simple data science problems which can be solved using Decision trees(besides Iris dataset)


texts: how do I find out if my random forresrt is overfitting?

texts: I used 100 trees on time series data to predict and the r square score is 0.9249886998972637. Is this over fitting? How do I know? Also, how do I determine the forest size? My data size is 1400 and variable 27 $(1400*27 =37800)$