texts: Negative R2_score Bad predictions for my Sales prediction problem using LightGBM

texts:
My project involves trying to predict the sales quantity for a specific item across a whole year. I've used the LightGBM package for making the predictions. The params I've set for it are as follows:
params = {
'nthread': 10,
'max_depth': 5, #DONE
'task': 'train',
'boosting_type': 'gbdt',
'objective': 'regression_l1',
'metric': 'mape', # this is abs(a-e)/max(1,a)
'num_leaves': 2, #DONE
'learning_rate': 0.2180, #DONE
'feature_fraction': 0.9, #DONE
'bagging_fraction': 0.990, #DONE
'bagging_freq': 1, #DONE
'lambda_l1': 3.097758978478437, #DONE
'lambda_l2': 2.9482537987198496, #DONE
'verbose': 1,
'min_child_weight': 6.996211413900573,
'min_split_gain': 0.037310344962162616,
'min_data_in_bin': 1, #DONE
'min_data_in_leaf':2, #DONE
'num_boost_round': 1, #DONE
'max_bin': 7, #DONE
'extra_trees': True, #DONE
'early_stopping_rounds':-1
}

My dataset consists of daily sales data (columns= date, quantity) for the years 2017, 2018, 2019 and 3 months of 2020. I've been trying to use the 2017 and 2018 data for training and cross-validation and trying to test it for 2019 data. However my predictions for the year is way off the mark while considering the quantities on a weekly, monthly, quarterly or yearly basis (error ~ 40-50%)(I've tuned the params to bring the error down to this values). Moreover while considering the predictions, my r2_score is giving me a negative value of around -2.9148426301633803. Any suggestions on what can be done to make it better?
Script for lightgbm:
lgb_train = lgb.Dataset(train_x, train_y)
lgb_valid = lgb.Dataset(test_x, test_y)
model = lgb.train(params, lgb_train, \
                  valid_sets=[lgb_train, lgb_valid],\
                  verbose_eval=50)
test_df_pred = df[(df.date >= '2019-01-01') & (df.date < '2020-01-01')]
#test_df_pred = df[(df.date >= '2019-01-01') & (df.date < '2019-02-01')]
#test_df_pred = df[(df.date >= '2019-01-15') & (df.date < '2019-01-22')]
test_df_pred['month'] = test_df_pred['date'].dt.month
test_df_pred['day'] = test_df_pred['date'].dt.dayofweek
test_df_pred['year'] = test_df_pred['date'].dt.year
col = [i for i in test_df_pred.columns if i not in ['date','id', 'qty']]
y_test_pred = model.predict(test_df_pred[col])
test_df_pred['qty_pred'] = y_test_pred
mse = mean_squared_error(y_true=test_df_pred['qty'], y_pred=test_df_pred['qty_pred'])
mae = mean_absolute_error(y_true=test_df_pred['qty'], y_pred=test_df_pred['qty_pred'])
mape = mean_absolute_percentage_error(y_true=test_df_pred['qty'], y_pred=test_df_pred['qty_pred'])
qty = test_df_pred.qty.sum()
qty_pred = test_df_pred.qty_pred.sum()
diff = qty_pred - qty


texts: I have transformed my cyclic variables into sin-cos variables; will I need further normalization/standardization?

texts:
I have a dataset where there are both numerical, categorical and cyclic(month-quarter) variables.

I will run a regression model, but I may also use Random Forest, XGBoost etc. So I will preprocess my data. During preprocessing stage, I transformed Cyclic variables into sin-cos representation. Both columns are now scattered between -1 and 1.

I will one-hot-encode the categorical variables, this is for sure.

But I have some questions for numerical transformations:

1) In general, can I transform some of the columns by standardization and some of the columns by normalization? Or should I choose a method and apply for all numerical columns?

2) It is also similar to 1. Now I have sin-cos columns, already transformed from a cyclic variable.

Should I further apply numerical transformation to this column, for example If I normalize all numeric variables between 0-1, will I apply this transformation also sin-cos columns?

May be the best way is normalizing every numerical column between -1 and 1?

Thanks


texts: How to find out the subject of an email (in the form of a sentence)  or a pdf document in NLP using Python. If I do topic modelling and get different groups of topic, how do I pick out the only topic best suited for the document and join back all the words to form a meaningful sentence as the email/document subject.


texts: How to set thresholds to automate the management of model drift?

texts:
I have a trained ML Model and new data coming in every week. The new data sometimes vary too much (in a statistical sense) and, therefore, the performance of the ML model degrades. The data-set schema consists of a few numerical columns, few categorical columns, and few text columns.
I have an infrastructure to calculate necessary statistics (mean, variance, skewness, null-values, unique-values) for each feature but am unaware, how much change should I allow between the statistics of the training set and the new data set before I consider retraining the model?


texts: Attractive clustering experiment for missing value datasets

texts:
I have an upcoming publication, where I present an algorithm for clustering tabular datasets with missing values. I want to do quantitative evaluations of my algorithm and qualitative evaluation. In this question I am asking for ideas for attractive qualitative experiments that "wow, the audience".

Quantitative evaluation: An obvious experiment is to take a complete dataset, remove some values and compare the clustering results of the complete dataset with the incomplete dataset. With this nice quantitative evaluation can be done, that is why I will do it. However this is not very attractive.
Qualitative evaluation: Ignoring the missing values problem, an attractive use-case for clustering is image segmentation like this and that. However, there would be little meaning values in those dataset - I mean, in practice all values are available. There might be noise (regarding the color values), but not missing data. Setting noise to NA, would first require a noise detector, but that is not part of my publication. I have trouble finding another use-case that one can "see" and is not the setup I mentioned in 1.


texts: Can Sequence to sequence models be used to convert code from one programming language to another?

texts: Is it possible to convert a code from one language to another using sequence to sequence programming. if not sequence to sequence, which algorithm can i use to do this?


texts: Selecting Regressors from list for time series

texts:
I have been try to get results from a time series problem and I have used fb's prophet algo for that.
Now problem is I have to select comb of those regressors giving me least rmse.

I used granger causality test but somehow that is not giving as good results as that of combinations for rmse.

Any suggestion.

One alternative is to perform comb of features, add them as regressors via multiprocessing, which is still a lot.

any other way i can remove those features(in this case market time series ecnometric indexes)


texts: Is it ever advantageous to perform regression on integer-typed label?

texts:
Right now, my program's type-checking prevents regression analysis from running on integer labels. It assumes ints are for ordinal/ OHE classification.
I just ran a test and verified that it is in fact possible to perform regression with an int label. However, wouldn't continuous/ floats always be preferred? Or are there certain situations where regression performs better on integers?
When feeding in an int label, the results actually come out as floats:
{'test': array([[ 2.4436188],
        [ 6.4358497],
        [ 3.6724093],
        [ 2.534666 ],
        [20.369778 ],
        [ 4.7333455],
        [ 3.9125016],
        [ 2.9019382],
...


texts: Data Visualisation Techniques for Multi Labelled data

texts:
I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: pitch, amplitude etc.).

So some have multi labelled emotion labels.

This is a snapshot of my dataset:

Please tell me any techniques for multi label classification visualization techniques. I searched all over the internet, but all of them are related to single label classification.


texts: I am trying to import Inception v3 in TensorFlow. I wish to apply it after reading this tutorial on object detection.


texts: Calculating the Standard Deviation by category using Python

texts:
I have a datset with Scores and Categories and I would like to calculate the Standard Deviation of these scores, per category. The data look something like this:

Category    Score
AAAA        1
AAAA        3
AAAA        1
BBBB        1
BBBB        100
BBBB        159
CCCC        -10
CCCC        9

What I would then like is the Standard Deviation of each Category. I know that with numpy I can use the following:

numpy.std(a)

But the example I can find only have this relating to a list and not a range of different categories in a DataFame.


texts: In k-fold-cross-validation, why do we compute the mean of the metric of each fold

texts:
In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.

Source : https://scikit-learn.org/stable/modules/cross_validation.html

However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.

Thanks


texts: Implementation of kmeans clustering using R

texts:
I have implemented kmeans clustering on iris dataset (inbuilt dataset) in R. The code is given below:

X=as.matrix(iris[-5]);

K=3;

prevCentroids=matrix(0,K,dim(X)[2]);
centroids=X[sample(1:dim(X)[1],K),];

dot=numeric(3);
C=numeric(150);

while(!isTRUE(all.equal(centroids,prevCentroids)))
{
  for(i in 1:dim(X)[1])
   {
    for(j in 1:dim(centroids)[1])
     {
       dot[j]=(X[i,]-centroids[j,])%*%(X[i,]-centroids[j,]);
     }
     C[i]=which.min(dot);
   }

  prevCentroids=centroids;

  for(k in 1:K)
   {
     centroids[k,]=colMeans(X[which(C==k),]);
   }
}

print(cbind(iris,C));

Sometimes, with this code, I get 85% clustering correct. But sometimes, it is just 37% as correct, if I compare it with the already clustered inbuilt iris dataset.

Could anyone please tell me where I am going wrong?


texts:
There are many blog posts and threads concerning the computation of mAP, both for PASCAL and for COCO. Still, they do not answer some fundamental questions.

I am listing here the flow as I understand it, and will be happy for people's answers and fixes:

Flow:

You run your detection and recognition model through all your testing images, for which you already have ground-truth annotations. You apply no threshold.  Question: Should each detection be duplicated |classes| times and saved for each class with its corresponding confidence? For example, a detection that is 60% dog, 40% cat and 0% for all the rest, should be saved once as dog with 0.6, once as cat with 0.4 and once for each other class with 0?
For each class, the ground-truth boxes of this class are matched with the detected boxes that intersect with them with $IoU > iou\_threshold$, where $iou\_theshold \in [0,1]$ is a predefined parameter, and which share the same class. The matches will be used in order to compute precision and recall for the class. Question: Is this process performed image-wise or over all the images at once? i.e., are precisions and recalls computed per image and then averaged, or are they computed over the total detections in all the images in one step?
The detections are sorted according to their confidence. The threshold starts from 0 and is each time raised to exclude the next detection(s) with the next lowest confidence. The remaining detections already have correct/incorrect fields, from the previous step, which are now used to compute precision and recall with the current confidence threshold (the "next threshold" is the confidence of the detection(s) that we just excluded)
The precision values are computed in certain predefined recall points, as the highest precision in all the points with recall > recall_point. If there are no such points, then we use 0 as precision. The class' average precision (AP) is then computed as simply the mean of all the precisions we found for the set of certain recalls.
mAP is computed as the mean of the classes' APs
In COCO, this whole process will be repeated with a certain set of IoU thresholds and will be averaged again.


texts: How to deal with evaluations in a multi step ML pipeline

texts:
Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data.

Ok, lets say I have one dataframe and one regressor, here what I do:

Split the data.
Apply the regressor to the training data.
Apply the model to testing data to get predictions.
Evaluate the prediction using an evaluator (e.g., binary classifier).

Now, lets say I have 3 steps and the last one is the regressors. I can not do the above because the first two also need to be applied to the testing data as well, how to deal with this? How my steps will look like?


texts:
I have a dataset of long/short equity hedge funds returns and their associated benchmarks (market indices).

I need to form multiple regression on the fund returns using the benchmarks returns as independent variables (i am allowed to form linear combination or manipulation of the indices or even non-linear combinations).

Of course, I do not know which independent variables to choose. Are techniques such as subset selection, Lasso, and Ridge supposed to be used in situation like this?


texts: convert single index pandas data frame to multi-index

texts:
I have a data frame with following structure:

df.columns
Index(['first_post_date', 'followers_count', 'friends_count',
       'last_post_date','min_retweet', 'retweet_count', 'screen_name',
       'tweet_count',  'tweet_with_max_retweet', 'tweets', 'uid'],
        dtype='object')

Inside the tweets series, each cell is another data frame containing all the tweets of an user.

df.tweets[0].columns
Index(['created_at', 'id', 'retweet_count', 'text'], dtype='object')

I want to convert this data frame to a multi-index frame, essentially by breaking the cell containing tweets. One index will be the uid, and another will be the id inside tweet.

How can I do that?

link to sample data


texts: How to save a Numpy array output of an autoencoder as an image

texts:
I have a 256*256*3 numpy array "SP" out of an autoencoder decoder layer which I want to save and open as an .jpg image. I used something like the following python code snippets:

img = Image.fromarray(SP, 'RGB')
img.save('my.jpg')
img.show()

However I have noticed the array "img" is 256*256 in dimension and the image is just a noise. What can be the right way to display the image? I have attached the array as a output.npy file: ---> https://ufile.io/410iu


texts: Need a Work-around for OneHotEncoder Issue in SKLearn Preprocessing

texts:
So, it seems that OneHotEncoder won't work with the np.int64 datatype (only np.int32)! Here's a sample of code:

import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
a = np.array([[56748683,8511896545,51001984320],[18643548615,28614357465,56748683],[8511896545,51001984320,40084357915]])
b = pd.DataFrame(a, dtype=np.int64)
ohe = OneHotEncoder()
c = ohe.fit_transform(b).toarray()

When I run this I get the following error: "ValueError: X needs to contain only non-negative integers."
As you can see, X DOES contain only non-negative integers! When I trim a few of the digits and change the datatype to int32 it works fine:

a = np.array([[56748,8511896,51001984],[18643548,28614357,56748],[8511896,51001984,40084357]])
b = pd.DataFrame(a, dtype=np.int32)
ohe = OneHotEncoder()
c = ohe.fit_transform(b).toarray()

Unfortunately, the data I need to encode has 11 digits (which can't be represented by int32). So, any suggestions would be helpful...
Also, I should mention, I don't necessarily need a one hot encoding, just need to create dummy variables. Thanks!


texts: Why is ElasticNet performs worse than both Lasso and Ridge?

texts:
I am using the following codes to build a few models on the same dataset:

X_train, X_test, y_train, y_test = train_test_split(X_in, y, test_size=0.25, random_state=42)

# Lasso regression
lasso = linear_model.Lasso()
lasso.fit(X_train, y_train)
pred_lasso = lasso.predict(X_test)

# Ridge regression
ridge = linear_model.Ridge()
ridge.fit(X_train, y_train)
pred_ridge = ridge.predict(X_test)

# ElasticNet
elastic = linear_model.ElasticNet()
elastic.fit(X_train, y_train)
pred_elastic = elastic.predict(X_test)

# R^2 Evaluation
print('R^2 for Lasso', r2_score(y_test, pred_lasso))
print('R^2 for Ridge', r2_score(y_test, pred_ridge))
print('R^2 for ElasticNet', r2_score(y_test, pred_elastic))
print('\n')

And the r2_score for the 3 models are:

R^2 for Lasso 0.28
R^2 for Ridge 0.14
R^2 for ElasticNet 0.02

This is confusing to me ... shouldn't the ElasticNet result fall somewhere between Lasso and Ridge? Why is ElasticNet result actually worse than the other two? Thanks!


texts: train_test_split() error: Found input variables with inconsistent numbers of sample

texts:
dataset_url='http://mlr.cs.umass.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv'
wine = pd.read_csv(dataset_url, delimiter = ';')

x_train, x_test, y_train, y_test = train_test_split(wine.quality,wine.alcohol,test_size=0.33, random_state=42)

training_accuracy = []
test_accuracy = []
# try n_neighbors from 1 to 10
neighbors_settings = range(1, 11)

for n_neighbors in neighbors_settings:
    # build the model
    clf = KNeighborsClassifier(n_neighbors=n_neighbors)
    clf.fit(X_train, y_train)
    # record training set accuracy
    training_accuracy.append(clf.score(X_train, y_train))
    # record generalization accuracy
    test_accuracy.append(clf.score(X_test, y_test))

plt.plot(neighbors_settings, training_accuracy, label="training accuracy")
plt.plot(neighbors_settings, test_accuracy, label="test accuracy")
plt.ylabel("Accuracy")
plt.xlabel("n_neighbors")
plt.legend()


texts:
I've come across the term "learnable parameters" recently, and googling didn't help much as most search was describing learnable parameters in a CNN instead of a DNN. Is there any difference between the two?
How would I compute the number of learnable parameters in a DNN? Could anyone please explain what those are with an example? I'm new to machine learning so I would appreciate some help on this.


texts: How to determine best parameters after grid search?

texts:
I'm optimizing the parameters for a single layer MLP. I've chosen to vary 4 parameters: hidden layer size, tolerance, activation, and regularization weights. Each of these has 4 possible values it can take (4^4 = 256 combinations).

So the question is, how does one determine that a set of parameters are statistically significantly better than another?

My stats is a little rusty, but my first thought was n-way ANOVA with 4 factors and 4 degrees of freedom in each factor. Is there something better?


texts: Possible Growth Function for a hypothesis set

texts:
I am new to ML and I am trying to resolve a Homework problem. How do I determine the possible growth function $m_H(N)$ for some hypothesis set? If my choices are $1,2^N,2^{\sqrt{N}},N^2-N+2$ and none.

My research : I understand that the growth function counts the most dichotomies on any N points and the growth function satisfies $m_H(N) \le 2^N$.

Also $m_H(3) = 2^3 = 8. m_H(4) = 14 < 2^4$. Where $H$ is the perceptron. Any leads on how I can resolve the question is really appreciated.

Thanks


texts:
I am building a bot with python and I need some system to solve captchas like these:

I think I need a deep learning algorithm, but coding one is a pain in the ass. Is there any easy solution to this?

I can code the part that screenshots the images and clicks on the correct answer, image 4 or 5 in the example. The images get rotated and change every time.

Thank you!


texts:
Getting this error while using flow_from_generator in keras

  63/3851 [..............................] - ETA: 6:41:59 - loss: 12.8586 - acc: 0.1930
  64/3851 [..............................] - ETA: 6:41:40 - loss: 12.8544 - acc: 0.1934Traceback (most recent call last):
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 551, in get
    inputs = self.queue.get(block=True).get()
  File "/tools/anaconda3/envs/py35/lib/python3.5/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/tools/anaconda3/envs/py35/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 391, in get_index
    return _SHARED_SEQUENCES[uid][i]
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 761, in __getitem__
    return self._get_batches_of_transformed_samples(index_array)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 1106, in _get_batches_of_transformed_samples
    interpolation=self.interpolation)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/preprocessing/image.py", line 364, in load_img
    img = img.resize(width_height_tuple, resample)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/PIL/Image.py", line 1743, in resize
    self.load()
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/PIL/ImageFile.py", line 233, in load
    "(%d bytes not processed)" % len(b))
OSError: image file is truncated (42 bytes not processed)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "convolutional.py", line 394, in <module>
    runCNNconfusion()
  File "convolutional.py", line 379, in runCNNconfusion
    epochs=epochs,verbose=1, callbacks = [MetricsCheckpoint('logs')])
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/models.py", line 1227, in fit_generator
    initial_epoch=initial_epoch)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 2115, in fit_generator
    generator_output = next(output_generator)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/utils/data_utils.py", line 557, in get
    six.raise_from(StopIteration(e), e)
  File "<string>", line 3, in raise_from
StopIteration: image file is truncated (42 bytes not processed)

Can someone please help, totally blank on how to proceed


texts: Feature selection and classification accuracy relation

texts:
One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your classifier and a subset of the ranked features.

For example, if your features are A, B, C, D, E, and if they are ranked as follow D,B,C,E,A, then you calculate the accuracy using D, then D, B then D, B, C, then D, B, C, E... until your accuracy starts decreasing. Once it starts decreasing, you stop adding features.

In example1 (above), you would pick features F, C, D, A and drop the other features as they decrease your accuracy.

That methodology assumes that adding more features to your model increases the accuracy of your classifier until a certain point after which adding additional features decreases the accuracy (as seen in example 1)

However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases.

In a scenario such as this one, how do you pick your features? Do you only pick F and drop the rest? Do you have any idea why the accuracy would decrease and then increase?


texts: What's the difference of Symbolic translation approach and statistical translation?

texts: Based on what I know, symbolic computing is based on hard rules in the code. Statistical translation is based on statistic and probabilities. If so, could you give an easy to understand example to demonstrate this difference? If it is not correct, please give a correct explanation of the differences.


texts:
I have a data frame, and I am only interested in the numerical variables (variable class is double):
Example
A    B    C    D    E
0.13 2.22 3.44 3.30 6.54
3.55 4.23 0.43 5.33 3.55
...

I want to build a linear regression xgboost model, so I converted the data frame to a dense matrix.
I took out all categorical variables from my data, and am left with the five variables of interest. But I can't seem to be able to use the dense matrix, despite it being a type of input that xgboost accepts. See here. I am instead getting this error:
Error
"Error in xgb.DMatrix(train_mat[, -n], label = train_mat[, n]) : [23:57:13] amalgamation/../dmlc-core/src/io/local_filesys.cc:66: LocalFileSystem.GetPathInfo 2007 Error:No such file or directory".
Code
train_mat <- as.matrix(subset(train, select = -c(Genre, Publisher, Rank, Name, Platform)))

lm.boosted <- xgb.train(data = train_mat, booster= "gblinear", max_depth=3, nthread=1, nrounds = 2, eval_metric="error", eval_metric= "rmse", objective= "reg:linear")

I tried to convert to a xgb.DMatrix, but I am not clear on what it's asking for to do the conversion.
train_mat <- xgb.DMatrix(data = train_mat, label= train$Global_Sales)

How do I input data into xgboost?


texts: Topics to cover for software developer interested in data analytics

texts:
First of all I don't know if this is the appropriate place to post this question. If it's not, I apologize in advance. It seems like the most relevant Stack Exchange sub.

A little about myself:

I'm a software developer and have been working in the field for about 5 years. I have a two year degree from a community college that didn't involve much math above pre-calc high school level stuff. I've never taken a statistics course.

Most of the software I write is business software. It's typically back end server or database stuff, never user facing. I do allot of data aggregation and normalization.

Lately I've been thinking about expanding my skillset. Data and statistics have always interested and I'd like to learn ways to measure and interpret data. Ideally this would lead to thinking of useful metrics and uses for data that would help the company I work at.

The problem is I'm not sure what I should be looking into. There are so many data science topics and I don't know what would be useful for me to learn. I prefer structured learning environments such as courses or video series.

I know this isn't a great question... I'm just not sure where to ask and I'm not sure how to ask for what I'm looking for... or even sure what I am looking for. Any guidance or suggestions would be very much appreciated. Even if the suggestion is a better place to ask the question or a better question for me to ask. Thanks much.


texts: Algorithm Suggestion For a Specific Problem

texts:
I'm working on a problem where in I have some data sets about some power generating units. Each of these units have been activated to run in the past and while activation, some units went into some issues. I now have all these data and I would like to come up with some sort of Ranking for these generating units. The criteria for ranking would be pretty simple to start with. They are:

Maximum number of times a particular generating unit was activated
How many times did the generating unit ran into problems during activation

Later on I would expand on this ranking algorithm by adding more criteria. I will be using Apache Spark MLIB library and I can already see that there are quite a few algorithms already in place.

http://spark.apache.org/docs/latest/mllib-guide.html

I'm just not sure which algorithm would fit my purpose. Any suggestions?


texts: Public datasets that show "cyclical" behavior

texts: I am looking for any publicly available dataset that has a "cyclical" structure to it, in the sense that if I plot the data in a certain way, a loop becomes visible.  A good example of this would be the Lotka-Volterra predator-prey model, which has a very pronounced cycle, shown below.  Are there any other good examples that demonstrate this?


texts:
Lately, I have been concerned to implement fixed effects and random effects (from econometrics) in deep learning.

After reading some articles, I realized that most of them just used only the neural network based on RNN with panel data. They were not considered to panel data structure such as fixed effects or random effects.

In my knowledge, in the case of LSTM, the weights in one cell are the same for all panel subjects. There is no consideration of the group average. This seems inappropriate from the viewpoint of panel data analysis. I believe that it should be considered to fixed effects, or random effects, or multi-level model in deep learning. Although there are some articles to apply those models in deep learning(https://arxiv.org/pdf/1702.06512.pdf, http://willwolf.io/2017/06/15/random-effects-neural-networks/) , most data scientists seem to think that it is not important or not necessary.

So I would like to hear opinions from experts about using fixed or random effects in deep learning.

Thank you.


texts: Extract and normalize date from string for the proper text mining

texts:
I am trying to perform string delivery classification (X: delivery_string, y: delivery_string_relevance (values 0 or 1)). I am using DTM (document term matrix) for the feature extraction, and experimenting with text-mining algorithms. However, I still believe that I can preprocess my raw strings better (before creating the DTM) by extracting delivery dates and obtaining one common format representation for all of them. Any suggestions on good date parsers? datefinder.find_dates() in Python is doing quite a decent job, however still fails for some cases, such like when string contains "delivery time 10 - 12 days" it extracts it as 10-12-2017.

Sample data (language used Dutch):
beschikbaar vanaf 04 okt // beschikbaar vanaf: 05-08-2017 // 8 - 9 weken // 8 - 9 werkdagen // 8 dag(en)


texts: Storing large data sets for python machine learning algorithm consumption

texts:
I'm reading up on how to clean/munge/wrangle data sets in order to run machine learning algorithms on them. Lots of info on how to do the actual wrangling, but a practical detail seems to be glossed over: storage.

My question is quite simple: which is the go-to technology to store/retrieve a large data set in order to run algorithms on it in the most convenient/efficient way possible?

I'm guessing the language in which the algorithms are written is not all that relevant here.


texts: How to interpret the loading values of a pca?

texts:
Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics:

ger = c(2,4,1,3,2,4,4,1,2,3)
phi = c(3,4,1,2,2,3,3,2,2,2)
mat = c(1,3,2,4,1,2,2,4,3,1)
phy = c(2,2,2,5,2,2,3,4,3,3)
A = cbind(ger,phi,mat,phy)

I combine everything to a matrix and scale the data:

As = scale(A)

Now, I perform a summary  on the PCA:

summary(princomp(As), loadings = TRUE)

Which returns the following output:

Importance of components:
                       Comp.1    Comp.2     Comp.3     Comp.4
Standard deviation     1.3257523 1.1657791 0.59600603 0.35793402
Proportion of Variance 0.4882275 0.3775114 0.09867311 0.03558799
Cumulative Proportion  0.4882275 0.8657389 0.96441201 1.00000000

Loadings:
     Comp.1 Comp.2 Comp.3 Comp.4
ger  0.496 -0.502  0.519  0.482
phi  0.548 -0.443 -0.423 -0.570
mat  -0.430 -0.572 -0.546  0.435
phy  -0.518 -0.474  0.503 -0.503

I have a few hints for the first component (based on the loadings):

There is a high positive correlation between german and philosophy and there is also a high positive correlation between math and physics.
Who is good in language (german and philosophy) is often worse in MINT (math and physics) and the other way around.

And an idea about the second one, which I cannot interpret:

It's a weighted arithmetic mean over all four variables.

But I have no idea how to interpret the Comp. 2, Comp. 3 and Comp. 4 based on the loadings. Especially because all values of Comp. 2 are all negative, or have the same orientation. Can someone help me? Thanks in advance!


texts: Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model?

texts:
I have experimented with a number of different machine learning models used for pose estimation. Most of them output a heatmap and offsets for the detected person(s) in the image. I really like the performance/accuracy of the multipose estimation model here.

What I would like to do next is to create a model similar to this one, except it should label each pose of the person(s) detected. There are multiple different implementations caffe/pytorch/tensorflow to choose from. I've thought about how to approach this and I have thought of a few different ways:

Create a completely new machine learning model and use the labeled output of the pose estimation model to train it.
Change or add layers to the machine learning model to change the output. (Not sure how this is done)
Ditch the pose estimate model and train a new model to directly estimate using raw images/labels of cropped people. This would rely on another method to detect each person.

I want to take the path of least resistance here but I also care about the time it takes to gather/process data, and most importantly the accuracy/performance of the model. Are there any experienced Machine Learning/Data Scientists who answer the following?

Which approach should I take? advantages/disadvantages
Which machine learning library offers the functions to do this.
My assumption is that option 1 or 2 would be more accurate than option 3. Am I correct?


texts: how to deal with columns that has different value in only 1 or 2 rows?

texts:
I have very high dimensional data. Almost 20% of the columns has different value in less than 1% of rows. All of these are binary columns and many columns has 0s filled in more than almost 98% of rows.
Some more info:
Target variable is an imbalanced(91.9%:8.1%) binary variable.
Every variable I have, except 3, are binary.
I would like some ideas on how to deal with columns like this? drop them or smote to have more data?
Thanks in advance.


texts: Should I use a regression or classification framework to predict rankings?

texts: My dependent variable is a performance ranking (relative to the other individuals). For example, one individual would be ranked first, another second, another third, etc. Would I use a classification or regression framework to model this?


texts: Predict customer behaviour with Transformer(attention is all you need)

texts:
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop through time.
So in this case N would be the number of users (or number of user sessions), T / time window I could set myself and D I could set also myself taking only event type (purchase, view, etc.) or something else like price, brand etc.(please see screenshot below)

Please share your opinion
Many thanks in advance


texts: Question about AlphaGo Zero's Neural network architecture?

texts:
The following text is quoted from the AlphaGo Zero Paper 2017 from Nature. My question is regarding the eight features.

The input to the neural network is a 19 × 19 × 17 image stack
comprising 17 binary feature planes. Eight feature planes, Xt, consist
of binary values indicating the presence of the current player’s
stones (Xit=1 if intersection i contains a stone of the player’s
colour at timestep t;

This sounds like the t means time and the eight features are the latest 8 board states. If this is the case, this will be my question.

0 if the intersection is empty, contains an
opponent stone, or if t < 0). A further 8 feature planes, Yt,
represent the corresponding features for the opponent’s stones. The
final feature plane, C, represents the colour to play, and has a
constant value of either 1 if black is to play or 0 if white is to
play. These planes are concatenated together to give input features st
= [Xt, Yt, Xt−1, Yt−1,..., Xt−7, Yt−7, C].

I have the feeling the eight features should be the combination of rotated (4) and reflected (2
) board states of the current board state. Please correct me if I am wrong.

History features Xt, Yt are necessary, because Go is not fully observable solely from the current
stones, as repetitions are forbidden; similarly, the colour feature C
is necessary, because the komi is not observable.

Board repetition can be easily prevented by keeping tracking of all previous board states and when a new move is attempted, just compare against all previous board states. The same is for komi. Keeping 8 previous board states to prevent repetition doesn't sound right to me. What if the repetition happened to the last 9th board state? If this is not a serious paper published to the Nature, I would highly doubt this is wrong.


texts: What math topics are most needed to become data scientist?

texts:
I am trying to learn data science from very different sources like Coursera, edx and many other sources. But I can not able to find any proper syllabus that what actually topics are needed. So, if anyone can please list out topics that I can follow and able to complete data science math skills.

Even if you post the name of books which can able to complete these criteria, then also it will be a great help. Thanks in advance.


texts:
What are the tools, practices and algorithms used in automated text writing?

For example, lets assume that I have access to wikipedia/wikinews and similar websites API and I would like to produce article about "Data Science with Python".

I believe that this task should be divided into two segments. First would be text mining and second would be text building. I'm more or less aware how text mining is performed and there are lots of materials about it in Internet. However, amount of materials related to automated text building seems to be lower. There are plenty of articles which says that some companies are using it, but there is lack of details. Are there any common ideas about such text building?


texts: Inconsistancy in Sklearn SVM predict() and predict_proba()

texts:
Actually I have two questions. One of them is related the bug of sklearn SVM model and the other one is about ROC-AUC score.

My first question is related to ROC-AUC score but also includes a bug similar to that (https://github.com/scikit-learn/scikit-learn/issues/13662) and my sklearn versiton is: scikit_learn-0.24.2-cp36-cp36m-macosx_10_13_x86_64.whl

I have imbalanced data which have 4672 positive and 14459 negative samples. I trained SVM model by following that code:
kernel='rbf'
c = 0.01
Gamma = 0.5
svm_model = SVC(kernel=kernel,C=c, gamma= Gamma, probability=True)
ss = svm_model.fit(X_train,y_train)
y_true_SVM1, y_pred_SVM1 = y_test, ss.predict(X_test)
y_true_SVM,yy_pred_SVM = flat(y_test), ss.predict_proba(X_test) #flat like a converting one hot representation

When I look at y_pred_SVM1 which is the output of predict function, I saw that all samples were predicted as negative. It is normal because of the imbalance. However, when I investigate yy_pred_SVM variable which keeps the predicted probabilities I saw that there are 6 samples which were predicted as positives:
[3.0000009e-14 1.0000000e+00]
[0.39734363 0.60265637]
[0.04542387 0.95457613]
[0.04544127 0.95455873]
[0.30997869 0.69002131]
[4.13349563e-09 9.99999996e-01]

I thought that it is a bug but I couldn't be sure. Because If there is a such bug how can we sure about the other probabilities. In their github page such a bug was reported for cross validation but I didn't use CV.
Moreover, I tried to evaluate my results with respect to some metrics:
SVM Accuracy score on our Test data: 0.7558795860771401
SVM Sensitivity score on our Test data: 0.0
SVM Specificity score on our Test data: 1.0
SVM MCC score on our Test data: 0.0
SVM ROC-AUC score on our Test data: 0.8524902491867827
SVM Average Precision Score score on our Test data: 0.7477108938150628
SVM F1-Score on our Test data: 0.0

However, I couldn't figure it out that even all classes were classified as negatives how can ROC-AUC score will be 0.85. I used this code for that:
rocAuc = metrics.roc_auc_score(y_true_SVM[:,1], yy_pred_SVM[:,1])

I know that there can be a threshold different from 0.5 and it could create such a case. Because of that reason I plotted the probs of positive and negative classes:

As you can see on the image above, there is not a distinct threshold for classes. Also we can understand from that prob array:
array([[0.63341647, 0.36658353],
       [0.7541376 , 0.2458624 ],
       [0.75364027, 0.24635973],
       ...,
       [0.75417909, 0.24582091],
       [0.91853779, 0.08146221],
       [0.75418213, 0.24581787]])

probabilities for being a positive were gathered around 2.45.
How should I approach that results and how should I fix that inconsistency between pred() and predict_proba() functions ?
By the way when I use class_weight
svm_model2 = SVC(kernel=kernel,C=c, gamma= Gamma,class_weight='balanced',probability=True)

pred() and predict_proba() gives same results in reverse order like in the link above. But this time my result became:
SVM Accuracy score on our Test data: 0.24412041392285982
SVM Sensitivity score on our Test data: 1.0
SVM Specificity score on our Test data: 0.0
SVM MCC score on our Test data: 0.0
SVM ROC-AUC score on our Test data: 0.8521257552159206
SVM Average Precision Score score on our Test data: 0.7472358224145087
SVM F1-Score on our Test data: 0.3924385633270321

Thank you for your help!


texts: Decision Tree Induction using Information Gain and Entropy

texts:
I’m trying to build a decision tree algorithm, but I think I misinterpreted how information gain works.

Let’s say we have a balanced classification problem.
So, the initial entropy should equal 1.

Let’s define information gain as follows:

info_gain = initial_entropy weighted_average(entropy(left_node)+entropy(right_node))

We gain information if we decrease the initial entropy, that is,
if info_gain > 0. If info_gain == 0
that means

weighted_average(entropy(left_node) + entropy(right_node)) == initial_entropy.

Let’s say we have 4 features such that

weighted_average(entropy(left_node) + entropy(right_node)) is in this order

  wa_of_feature_0 > wa_of_feature_1 > … > wa_of_feature_4.

The more info_gain is larger than 0, the more the feature makes order in system.

So, based on our 4 features maximum information gain will be

info_gain_max = initial_entropy - wa_of_feature_4

since that would give us bigger number than using wa_of_feature_n where 1<=n<4.

Is this the correct interpretation?


texts:
I'm wondering why sequence batching in RNNs's target value loops back (I'm not sure what you call it), but let's take for example:

We want to learn a sequence of numbers (our input) from 1 to 16:

$$
        \begin{bmatrix}
        1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16
        \end{bmatrix}
$$

Batches: 2, Sequence Length: 4

First, we can divide the data to 2 batches:

$$
        \begin{bmatrix}
        1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\
        9 & 10 & 11 & 12 & 13 & 14 & 15 & 16
        \end{bmatrix}
$$

Then we can divide this into mini batches:

$$
        \begin{bmatrix}
        1 & 2 & 3 & 4\\
        9 & 10 & 11 & 12
        \end{bmatrix}
$$
$$
        \begin{bmatrix}
        5 & 6 & 7 & 8\\
        13 & 14 & 15 & 16
        \end{bmatrix}
$$

Then we need to create targets for the inputs, and intuitively we want to to targets to be the next value of the input, so:

$$
        \begin{bmatrix}
        2 & 3 & 4 & 5\\
        10 & 11 & 12 & 13
        \end{bmatrix}
$$

However, this is not what I usually see, instead I see the last value in a mini batch is swapped with the first value:

$$
        \begin{bmatrix}
        2 & 3 & 4 & 1\\
        10 & 11 & 12 & 9
        \end{bmatrix}
$$

So what is the intuition in doing so?

Since if we want to learn the sequence of 1, 2, 3, 4, but 1 was given as the target for the value 3, so 4 was not learnt but instead of 1.


texts: How to use Cohen's Kappa as the evaluation metric in GridSearchCV in Scikit Learn?

texts:
I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better metric than F1 score for class imbalance.

But I couldn't find kappa as an evaluation_metric in scikit learn here
sklearn.metrics.

Questions

Is there any workaround for including kappa in gridsearchcv?
Is there any other better metric i can use instead of kappa in scikit learn?


texts: I'm trying to use Tensorlow on İris dataset using sublime as the text editor. I got this error "TypeError: float() argument must be a string or a number, not 'generator'"when running the session, but I haven't any Idea about it. Please help me out. Attached is the error screenshot.


texts: Using neural nets with just one input variable (without response/feature)

texts:
Hi i am using following data from R library ftnonpar:

library(ftnonpar)
data(djdata) #this generates four samples, each size of 2048
data1 <- djblocks
head(djblocks)
[1] -0.11140257 -0.05594395 -0.29779225 -0.10952726 -0.03012277  0.12101243

As you can see, it is only vector of data, but it seems to interesting for me how can i fit those data. I want use neural nets for fitting this (especially RBF network). But in this case i do not know how to prepare data for this, because i have only one input variable. I tried create data.frame like this:

n <- seq(1,length(data1),1)
df <- as.data.frame(cbind(n,data1))
head(df)
    n    djblocks
  1 1 -0.11140257
  2 2 -0.05594395
  3 3 -0.29779225
  4 4 -0.10952726
  5 5 -0.03012277
  6 6  0.12101243

And my idea was to normalize n:

df$n <- (n - min(n))/(max(n) - min(n))

and then use n as feature and djblocks as response, but i think this is not the best idea. Next idea is use RNN and the ouput from RNN connect to RBF layer. This is interesting problem for me, can you somebody help me, how to prepare data for this problem ? Thank you


texts: Churn definition for non-contractual services

texts:
I am computing churn definition i.e, the number of days after which we will say a customer has churned in fashion retail, etc. Currently, I am using the transaction dates to get the average days between two purchases of each customer, other stats such as number of days he has been customer.
If anyone has any idea on this , kindly give some pointers.


texts: Matching new set of data with pre-defined sets

texts:
I have sets of data describing sets of levels of requirements needed for certain sets of tasks. The following is a tabulated example:

Note that the data values are on a scale from 0 to 10.

My problem here is that I have a set of employees whose skills (analysis, patience, comprehension ...) have been analyzed, like the following employee:

Analysis --> 8.5
Patience --> 5
Comprehension --> 7
Communication --> 7.5
Creativity --> 8

How to match this employee to the best task according to his skillset and the required ones for each task and also find a matching percentage.

Please note that the number of tasks is much bigger (around 1000 tasks) with more requirements that have been found statistically.


texts: Correlation among features (e.g. doc length, punctuation, ... ) in classifying spam emails

texts:
I extracted some other features from my dataset regarding punctuation, capital letters, upper case words. I got these value:

looking at the correlation with my target variable (1=spam, 0=not spam), using .corr() in python.
BT stands for binary text, e.g., and BS stands for binary summary, where I assign 1 or 0 based on the presence of a capital letter in the text/summary, or upper case word, or...
Do you think that features like these can be useful in model building? I cannot see very strong correlations, but I would like to determine if an email can be spam or not based also on features like these (number of character/text length; presence of !, upper case words,....).
I have around 1000 emails, but only 50 are spam (maybe too small to extract useful information).
However, I had to extract these information, so it is a new dataset, built on my own, so I could not get many more spam emails (and I would like to not use datasets from kaggle, for instance).
What do you think?


texts: scale_pos_weight using XGBoost's Learning API

texts:
I see it is possible to add a weight for unbalanced problems in XGBoost's Scikit-Learn API through scale_pos_weight. Does it have an equivalent in the Learning API? If not,

is there a reason behind this?
Could this corrective factor/weight also be somehow implemented using the learning API?


texts: How to create correlation matrix but with only part of the rows?

texts:
I would like to have correlation matrix like this one, but with only 3 bottom rows but all the columns. How can I generate it?

corr = corrdata.corr()
sns.heatmap(corr, mask=np.zeros_like(corr), annot=True, cmap=sns.diverging_palette(220, 10, as_cmap=True),
            square=True, ax=ax, fmt='.2f' ,  )


texts: Converting HTML tables to a data.frame in R

texts:
I am trying to convert an HTML table of daily data into a R data.frame for trend analysis.  Using xml2 and rvest I am able to download the table into RStudio, but am struggling to A) create the dataframe logically B) bind new data to the dataframe.

Here is my code to import the data:

# Loads the rvest, dplyr and xml2 packages
library(xml2)
library(rvest)
library (dplyr)

today <- Sys.Date()

# Creates the URL to be searched
base_url <- paste("https://services.tcpl.ca", "cor", "public", "gdsr/", sep = "/")
daily <- paste("GdsrNGTLMetric", format.Date(today, "%Y%m%d"), ".htm", sep = "")

tc_url <- paste(base_url, daily, sep = "")

#Gets the website data, user agent tells the server who it is

gdsr <- tc_url %>%
read_html( options = "NOBLANKS") %>%
html_node(xpath = '//*[@id="oReportCell"]/div/table') %>%
html_table(header = FALSE, trim = TRUE)

#Massages the data into a useable dataframe
gdsr <- gdsr %>%
select(X1, X3)

gdsr <- subset(gdsr, !X1 == "" | !X3 == "")
gdsr <- gdsr[-c(1:3, 6), ]

And this is my data.frame that I want to insert the scraped data into:

gdsr_df <- setNames(data.frame(matrix(ncol = 25, nrow = 0)), c("date", "Empress", "McNeil", "ABC", "Gordondale", "GroundbirchEast", "OtherBorders", "TotalExports", "Intraprovincial", "TotalStorAndIntra", "TotalNetStor", "StorDel", "StorRec", "NGTLDel", "NGTLRec", "FieldRec", "EODLinePack", "LPRateOfChange", "LPTarget", "LowTol", "HighTol", "TolLastChange", "SDAccount", "OBAAccount"))

Please advise on how I can make this happen.


texts: Using Binary Image Classification on Video

texts:
Using tensorflow/Keras, I have built a good model which is currently binary classification.  For example, training labels would be images of a person's knees bent or knees not bent.

The idea behind such a model could be using a continuous video feed, and when it detects either knees bent or not, a certain probability would output.  It works, however, since it's binary, I have probabilities close to 0 or 1 even though a person is not even in the video feed.

Any ideas on how I can change my model (perhaps multi-class) in over a continuous feed - which could be hundreds of images over a minute would only give me probabilities for those events of knees bent or not?  If going multi-class, would I then have to also train all other potential images?  That seems close to impossible, even with using transfer learning with ImageNet.  Any suggestions?


texts: Preprocessing for finetuned CNN model from pretrained models

texts: Is it necessary to preprocess the images the same way as they were during the training of pretrained models in our finetuned model to use it for a different classification task ? Say, I have a pretrained VGG16 model which I am finetuning to use it for a different classification task ? Now, before feeding my dataset to this finetuned model, should I necessarily use vgg16.preprocess_input() for training on this dataset or I can use my own preprocess_input ?


texts: When memory errors occur with model.fit(), is it due to GPU RAM or normal RAM?

texts:
With respect to this question,

https://stackoverflow.com/questions/51895278/how-to-know-when-to-use-fit-generator-in-keras-when-training-data-gets-too-big

when memory errors are reported due to excessively large training data, are the memory errors caused by lack of normal PC RAM or lack of GPU RAM? I would like to know which one so that I can buy the right kind of RAM.

Keras was configured to use GPU for training.


texts: IsolationForest Decision Function vs. Anomaly Prediction Question

texts:
I'm currently working on an unsupervised anomaly detection project, and for it I'm using IsolationForest through scikit-learn. My question is, why/how is it possible for the model to predict something to be an anomaly when it is within the decision function space for inliers?

I've attached my results here:

Could the size of the decision function space be due to my input dimension vs this 2 dimensional projection?

I also made a quick plot of anomaly score vs. prediction (0 = inlier, 1 = anomaly):

As seen, there exists outliers above the threshold score, which doesn't make sense to me. Can someone explain?


texts: Finding correlation between different columns

texts:
I have a dataframe in following shape:
cart_rank|cart_id|no_of_users|Art Gallery|Art Museum|Asian Restaurant|Athletics & Sports|Bakery|Bar
0|30|42|0.0|0.0|0.2|0.0|0.0|0.00|0.0
1|6|40|0.0|0.0|0.0|0.0|0.0|0.00|0.0
2|10|29|0.0|0.0|0.1|0.0|0.0|0.00|0.0
3|12|28|0.0|0.0|0.0|0.0|0.0|0.00|0.0
4|16|26|0.0|0.0|0.0|0.0|0.1|0.00|0.0

It has a cart id, number of users a cart has and depending on it its rank. Cart with highest number of users is on top. Next to it is a set of venues which are one hot encoded.
For e.g. record one indicates a cart with 42 users and 0.2 art museums near it (it's a mean value).
What I want is to find a possible correlation in between nearby venues and success of carts. Such may be if carts are placed near to a Bar, they get more users.
But in current shape, I am stuck and can’t seems to understand what to do to move ahead with this.
One thing I believe I can do is df.groupby(["American Restaurant"]).mean().reset_index() for each venue.
This will give me mean rank of cart for each venue and then I can see for which mean rank is high and for which its low.
But the number of unique venues is really high and it would practically impossible or at least not elegant to do this one by one for each of them.
Is there an easy way or better way of doing this? Or overall some different strategy i should follow?


texts:
I need to calculate the accuracy but at a false positive rate of 1%.
I am not sure if it is the normal accuracy that we can calculate with sklearn or I need a customized formula?


texts: Reshape Time series data for Conv2d Block

texts:
I am modelling my time series data into a supervised learning problem for the input to a conv2d block in pytorch from this tutorial.

timesteps = 5

def to_supervised(train,n_input=timesteps,n_output=3):
    n_start = 0
    train_X , train_y = list(),list()
    for _ in range(len(train)):
        n_end = n_start + n_input
        n_out = n_end + n_output
        if n_out < train.shape[0]:
            x = train[n_start:n_end,:]
            y = train[n_end:n_out,0]
            train_X.append(x)
            train_y.append(y)
        n_start += 1
    return np.array(train_X),np.array(train_y)

How ever this code turns the input shape of (100,5) to (92,5,5). But what i am expecting it to make it a 4d array(batchsize, samples, Height, width) for the input of conv2d block.
So i have changed the code a little bit

timesteps = 5

def to_supervised(train,n_input=timesteps,n_output=3):
    n_start = 0
    train_X , train_y = list(),list()
    for _ in range(len(train)):
        n_end = n_start + n_input
        n_out = n_end + n_output
        if n_out < train.shape[0]:
            x = train[n_start:n_end,:,:] #Here
            y = train[n_end:n_out,:,0] # and here
            train_X.append(x)
            train_y.append(y)
        n_start += 1
    return np.array(train_X),np.array(train_y)

but it throws error IndexError: too many indices for array

I saw many guides for multivariate sequence but I don't understand how to apply this on this case


texts:
In my dataset I have a 'text' column and a 'followers' column containing lists of follower IDs, i.e. '1093777852477116417, 936194589043683328,...'. Some of the 'followers' values contain thousands of IDs.

I am preprocessing the data for LSTM, and I will do word embedding on the text column.

My question is, should I add the follower IDs to the word embedding of the text column, or should I hash the follower IDs and add an extra LSTM input layer for the IDs?

Thanks in advance!


texts: Cluster titles or ingredients of food into n-categories

texts:
I have a dataset which has information about food recipes (in german), that looks like this:

Here is a link to a small .csv file (first 1000 rows of my data)
https://drive.google.com/file/d/1C7thFlOnDn-oTc6AaDWA3CXXcX8m9NRu

The idea is to cluster the recipe names into n-categories so that afterwards I can assign every recipe to a category. Of note, there are tags and ingredients for every recipe, maybe this information helps to refine the clusters?
For example the algorithm (maybe a semantic analysis?) should output: categorised recipes into 200 biggest found categories: (hamburger, soup, pizza, ...)

Is there a way to do this?

Note: I have for every recipe min. 1 image. The idea is to label my images with n-categories, afterwards to train a convolutional neural network with my data.
The input would be an food image, the output would be a category.


texts: How duplicated items can be deleted from dataframe in pandas

texts:
I have the following data:

userid itemid
  1       1
  1       1
  1       3
  1       4
  2       1
  2       2
  2       3

How can I drop duplicated items (not the index) to make the following DataFrame:

userid itemid
  1       1
  1       3
  1       4
  2       1
  2       2
  2       3

Can someone help me?

import pandas as pd
df = pd.DataFrame({'userid':[1,1,1,1, 2,2,2],
                   'itemid':[1,1,3,4, 1,2,3] })


texts: Make Keras run on multi-machine multi-core cpu system

texts:
I'm working on Seq2Seq model using LSTM from Keras (using Theano background) and I would like to parallelize the processes, because even few MBs of data need several hours for training.

It is clear that GPUs are far much better in parallelization than CPUs. At the moment, I only have CPUs to work with. I could access 16 CPUs(2 Threads per core X 4 cores per socket X 2 sockets)

From the doc of multi-core support in Theano, I managed to use all the four cores of a single socket. So, basically the CPU is at 400% usage with 4CPUs used and the remaining 12 CPUs remain unused. How do I make use of them too. Tensorflow could also be used instead of Theano background, if it works.


texts: What should be the criteria to select features using correlation factors between features?

texts:
For the titanic dataset, I have done some feature engineering (one-hot encoded the features) and now I have developed a heatmap to view the correlation between different features.

I'm not able to understand what to do with them. Lets say two features are highly correlated, eg, in the image, Name_title_mr and sex_0 (ie. male) are having correlation factor of 0.84. So, does that mean I should drop Name_title_mr, since sex_0 is a very important feature. (This we know with experience, that sex in titanic is very important but is there any way we could view this as well by just observing the heatmap?).
One more doubt I have is: How would I know that I can just add two features, like sibsp and parch ? I have seen many kernels where they just create one feature with no_of_family_members by just adding sibsp and parch.

Will that adding of two features helpful? and should I drop the individual features sibsp and parch in that case?

sibsp_1 and sibsp_0 (sibsp_1 is when sibsp=2 and sibsp_0 is when sibsp=1) seems to be highly negatively correlated. Should I consider some action on this as well?


texts: How to get xgbregressor feature importance by column name?

texts:
xgb = XGBRegressor(n_estimators=100, learning_rate=0.08, gamma=0,
  subsample=0.75, colsample_bytree=1, max_depth=7)

xgb.get_booster().get_score(importance_type='weight')
xgb.feature_importances_

This will return the feature importance of the xgb with weight, but how to return it with column name?


texts: sklearn.GridSearchCV predict method not providing the best estimate and accuracy score

texts:
I was playing around with the credit default dataset in UCI
("https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls")

These are the steps i have undertaken so far:

Created a Pipeline for Perceptron
Parameter values were provided for learning rate and epochs.
Passed the estimator and param grids to GridSearch to get the best estimator
GridSearch provided me with best score for a particular learning rate and epoch
used predict method on the gridsearch and recalculated accuracy score

Parameters provided for gridsearch

{'perceptron__max_iter': [1,5,8,10], 'perceptron__eta0': [0.5,.4, .2, .1]}

Question: The best parameters provided by GridSearch are not really the best parameters. I am not getting same scores using predict method on the gridsearch. Why is this the case? WHat am i doing wrong?

So acc to gridsearch best param are :
{'perceptron__eta0': 0.5, 'perceptron__max_iter': 8}

Accuracy score : 0.7795238095238095

However if i use these best parameters and call predict on gridsearch gives a totally different value, accuracy score dips to  0.5882222222222222

Please find code below.

import pandas as panda

from sklearn.model_selection import learning_curve, train_test_split,GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn.metrics import accuracy_score, mean_absolute_error, classification_report
from sklearn.linear_model import Perceptron, LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

from matplotlib import pyplot as plot
import seaborn as sns

from numpy import bincount, linspace, mean, std

remote_location = "https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls"
data = panda.read_excel(remote_location,sheet_name = "Data", header = 1)

data.rename(str.lower, inplace = True, axis = 'columns')

_y_target = data['default payment next month'].values

columns = data.columns.tolist()
columns.remove('default payment next month')

_x_attributes = data[columns].values

## meaning of stratify = _y_target. returns test and training data having the same proportions of class label '_y_target'
_x_train,_x_test,_y_train, _y_test = train_test_split(_x_attributes, _y_target, test_size =0.30, stratify = _y_target, random_state = 1)

## lets check the distribution. we can see 4times the lower value as was the case before as well. train/test set distributed well
print("label counts in y train %s" %bincount(_y_train))
print("label counts in y test %s" %bincount(_y_test))
parameter_grid = {'perceptron__max_iter': [1,5,8,10], 'perceptron__eta0': [0.5,.4, .2, .1]}

pipeline = make_pipeline( StandardScaler(),Perceptron(random_state = 1))
gridsearch = GridSearchCV(estimator = pipeline, param_grid = parameter_grid, cv = 10, n_jobs = 1, scoring = 'accuracy')

search = gridsearch.fit(_x_train, _y_train)
print(search.best_params_)
print(search.best_score_)

_y_prediction = gridsearch.predict(_x_test)

print("Accuracy score %s" %accuracy_score(_y_test,_y_prediction))
print("Classification report  \n %s" %(classification_report(_y_test, _y_prediction)))

My output is as below:

{'perceptron__eta0': 0.5, 'perceptron__max_iter': 8}
0.7795238095238095
Accuracy score 0.5882222222222222
Classification report
              precision    recall  f1-score   support

          0       0.78      0.66      0.71      7009
          1       0.22      0.35      0.27      1991

avg / total       0.66      0.59      0.62      9000


texts: How can I calculate Kernel matrix K for clustering based Kernel Principal Component Analysis?

texts:
In practice, a large data set leads to a large K, and storing K may become a problem. One way to deal with this is to perform clustering on the dataset, and populate the kernel with the means of those clusters for speeding up and reducing storage.
How can I calculate the kernel using clustering?


texts: How to classify a new email as spam/not spam?

texts:
I am working on a small exercise for determining if an email is spam or not. My dataset is the following:
                       Email                                                   Spam
0   Hi, I am Andrew and I want too buy VIAGRA                                   1
1   Dear subscriber, your account will be closed                                1
2   Please click below to verify and access email restore                       1
3   Hi Anne, I miss you so much! Can’t wait to see you                          0
4   Dear Professor Johnson, I was unable to attend class today                  0
5   I am pleased to inform you that you have won our grand prize.               1
6   I can’t help you with that cuz it’s too hard.                               0
7   I’m sorry to tell you but im sick and will not be able to come to class.    0
8   Can I see an example before all are shipped or will that cost extra?        0
9   I appreciate your assistance and look forward to hearing back from you.     0

where 1 means spam, 0 not spam. What I have tried is the following:
#Tokenization

def fun(t):

# Removing Punctuations
remove_punc = [c for c in text if c not in string.punctuation]
remove_punc = ''.join(remove_punc)

# Removing StopWords
cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')]

return cleaned

So I applied the function: df['Email'].apply(fun). Then I converted the text into a matrix as follows:
from sklearn.feature_extraction.text import CountVectorizer
mex = CountVectorizer(analyzer= fun).fit_transform(df['Email'])

and split the dataset into train and test:
X_train, X_test, y_train, y_test = train_test_split(mex, df['Email'], test_size = 0.25, random_state = 0)

I applied a classifier (I would apply the logistic regression to determine if an email is spam or not, but I have currently only used Naive Bayes:
from sklearn.naive_bayes import MultinomialNB

classifier = MultinomialNB()
classifier.fit(X_train, y_train)

Finally I applied the classifier first to the train set, then to the test set:
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score

pred = classifier.predict(X_test)

print(classification_report(y_test ,pred ))
print('Confusion Matrix: \n', confusion_matrix(y_test,pred))
print()
print('Accuracy: ', accuracy_score(y_test,pred))

The code works but I would like to know how to visually see with an example of new email if this has label 1 or 0. For example: if I have a new email 'Hi, my name is Christopher and I like VIAGRA', how could I determine the label/class?
I feel I am missing something or probably I am following a wrong way to demonstrate this.
My question is the following:
Given this new email: Hi, my name is Christopher and I like VIAGRA, how can I see if this is spam or not? I have thought about classification but probably my approach is wrong. I would like to have something like:
Email                                        Spam
...
Hi, my name is Christopher and I like VIAGRA 1

as this is very similar to the email  'Hi, I am Andrew and I want too buy VIAGRA' (if included in the train set or correctly predicted in the test set).
Maybe what I want to do requires only tf-idf algorithm or a different approach.
Any advice will be appreciated.


texts: How to predict months/time using neural network?

texts:
Assume I have a problem where I have to predict time let it be years, months or days. So I think it's a regression problem but the problem is how to do it. I mean usually dense layers prediction will be in floating number right? But months or days are usually interger.

That is we can predict 8.65 months when the y_true is 8 months right? How do deal with this? Should I just round it?

What type of loss function can I use in this case?


texts: Best method for similarity searching on 10,000 data points with 8,000 features each in Python?

texts:
As mentioned in the title I am attempting to search through 10,000 vectors with 8000 features each, all in Python. Currently I have the vectors saved in their own directories as pickled numpy arrays. The features were pulled from this deep neural network.

I am new to this space, but I have heard about M-Trees, R-Trees, inverted tables, and hashing. Is any one of them better for such a large number of features?

This implementation needs to be done quite rapidly and is just a prototype, so simplicity is valuable.

Thank you for your help.


texts: Ranking of feature sets which will be used for binary classification

texts:
I have some feature sets say X1 and X2 ...
Each feature set have some variable amount of features and there is no intersection between different feature sets
Say X1 have 100 features and X2 have 500 features and none of the features they had is same .
Though the file from which these features are extracted is same for both x1 and x2.
Each feature set can be used individually for binary classification.
Like X1 can also be used for classification and X2 can also be used for classification. They can also be used together combined i.e X1 U X2 which will have 600 features
But I want to compare the feature sets and hence want to know if there is any statistical method which can be used to rank these feature sets other than classification accuracy
Like X1 is better than X2 and so on ...


texts: How to fix inconsistent (variable spelling) categorical data and "fill in" missing data

texts:
I am a newbie data science engr. My first challenge is to (1) normalize inconsistent values in categorical features and (2) fill any missing information.

To describe inconsistency lets say we have a country field and there can be multiple entries for USA. e.g. USA, United States of America, US, US-50, United States, America, etc.

How do I normalize all these entries and just say USA for all entries. I can think of some set of full text search rules to match against using lucene or something. Is there any specific technical term for this problem?


texts: Where is the majority of Tensorflow memory usage?

texts:
Trying to wrap my head around where Tensorflow starts to use a lot of memory.

If I have a batch_size=10 and my samples are 150MB each, then that’s only ~1GB at a time or less if the samples are processed 1 at a time.

Is it more about the accruing size of the neural net itself as I go through more epochs? If I try to train against 10,000 of these 150MB samples, am I more likely to run out of memory toward the end of the training?


texts:
On the following lines of code I am getting

clf = neural_network.MLPClassifier(hidden_layer_sizes=(5, 12))
parameters =[ {'solver': ['lbfgs'],'max_iter': [500,1000,1500], 'alpha': [1e-1,1e-2,1e-3,1e-4,1e-5,1e-6,1e-7],  'random_state':[0,1,2,3,4,5,6,7,8,9]}]
model = GridSearchCV(clf,param_grid=parameters,n_jobs=-1)

On the last line I am getting the following error

ValueError: Parameter values for parameter (solver) need to be a sequence(but not a string) or np.ndarray.

now I know from reading

Parameters in GridSearchCV in scikit-learn

That this means everything must be in an array but all my params are in an array so what am I doing wrong.  thanks


texts: implementing conv2d in fourier domain using np.einsum --> ValueError: einstein sum subscripts string contains too many subscripts for operand 0

texts:
According to the convolution theorem, convolution operation changes to pointwise multiplication in fourier domain - here I have 'fft_x' of shape (batchsize, height, width, in_channels) which is the fft of input data and similarly 'fft_kernel' of shape (height, width, in_channels, out_channels) which is fft of the kernel after being padded to image size. To get pointwise multiplication of these in efficient way, I was using einsum in the following way -

...
    print(fft_x)
    print(fft_kernel)
    output = 0
    n=int(self.no_of_kernels/2)+1      # n = out_channels here
    for i in range(n):
        output += np.einsum('ijkl,jkl->ijk', fft_x, fft_kernel[i])
    return output
...

It gives the following output and error-

Tensor("input_11:0", shape=(?, 28, 28, 1), dtype=complex64)
Tensor("fourier__conv2d_11/transpose:0", shape=(28, 28, 1, 17), dtype=complex64)
...
...
ValueError: einstein sum subscripts string contains too many subscripts for operand 0

Could anyone please explain why this error is arising? Thanks in advance for any help.


texts: How to work with large amount of data overcoming RAM issues in python

texts:
I use numpy arrays to work with deep learning images. But as the data gets bigger, I'm facing issue with RAM even before training the model when using techniques like data augmentation.

Can someone suggest me how to work with large data for eg. 30GB of data in my system which has 16gb ram.

P.S. I'm worried about RAM during preprocessing and training, while i do batch processing with my GPU


texts: How to visualize multivariate regression results

texts: Are there commonly accepted ways to visualize the results of a multivariate regression for a non-quantitative audience? In particular, I'm asking how one should present data on coefficients and T statistics (or p-values) for a regression with around 5 independent variables.


texts: I am new to Machine learning. While reading SparkMLLib java code, I found Binary_classification dataset. But I am not able to understand how this data is modeled and if I want to model same type of data, what I have to do?


texts:
I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data:

It looks like outliers are 4,5,6 and should be removed.

Now plotting the field "A" in test file also shows somewhat similar results to the image shown. In that case, should I consider 4,5,6 as outliers, and remove them from train data, or keep them? Any pointers are appreciated.


texts: How can I determine the relationship between spam and weekdays?

texts:
I am trying to check if there is a correlation between spam emails and weekdays.
My dataset looks like as follows:
  Spam? Day
0   1.0 Saturday
1   1.0 Saturday
3   0.0 Saturday
5   1.0 Saturday
7   0.0 Friday
... ... ...
346 0.0 Friday
348 1.0 Friday
361 0.0 Saturday
383 1.0 Thursday
387 1.0 Friday

where 1 means spam and 0 not spam.
I have tried as follows
corr = (numpy.corrcoef(df['Spam?'],df['Days']))

I do not know how to explain a possible relationship between these two variables and if a plot could help to better visualise data and relationship.


texts: Implementing Multiclass Dice Loss Function

texts:
I am doing multi class segmentation using UNet. My input to the model is HxWxC and my output is,
outputs = layers.Conv2D(n_classes, (1, 1), activation='sigmoid')(decoder0)

Using SparseCategoricalCrossentropy I can train the network fine. Now I would like to also try dice coefficient as the loss function. Implemented as follows,
def dice_loss(y_true, y_pred, smooth=1e-6):
    y_true = tf.cast(y_true, tf.float32)
    y_pred = tf.math.sigmoid(y_pred)

    numerator = 2 * tf.reduce_sum(y_true * y_pred) + smooth
    denominator = tf.reduce_sum(y_true + y_pred) + smooth

    return 1 - numerator / denominator

However I am actually getting an increasing loss instead of decreasing loss. I have checked multiple sources but all the material I find use diceloss for binary classification and not multiclass. So my question is is is there a problem with the implementation.


texts: Non-Gaussian like distributions - Classifier of source data fails on target data

texts:
I ask you for help on a classification problem (classes are represented by the numbers 0,1 and 2). All features are extracted from time series data (fundamental is sinus shape).
I have a source dataset with features, which do not follow a gaussian distribution:

I can train classifiers with high accuracies with less effort. BUT: My validation data (target domain) has features with a distribution which is also not gaussian but which differ from the distribution the classifier was trained on.
E.g., we compare the distribution of the same feature for class 0 in source and target domain:

For classification models I used several different types of models which do not assume the features to be gaussian distributed (Trees, KnearestN,...) and I also tried a neural network (MLP with 3 hidden dense layers).

What my problem is
The problem I am facing is the same: The model trained on the source data (to ~98% valid. acc) fails on the target data with null-accuracy like (~30%). To validate that the problem is not only caused by the feature set (some features shown above) I tried a neural network approach. I am having the same problem if I train a neural network on the raw time series data, train/test is okay but performance on unseen instances is about null-accuracy only.

What I currently tried to solve it

Transforming the features to a gaussian like distribution using e.g. quantile and box-cox transformation -> The problem with that is, that I cant fit a well performing classifier on that transformed data since the models do not converge
Different optimal transport strategies to transform the source distributions to the target shapes and train a supervised model with the transformed source data and then evaluate on the target data. Acc. also in range of null-accuracy
Domain Adaptation models using non-parametric distance metrics for loss functions, like MMD (maximum mean discrepancy) and CORAL (correlation alignment). The problem is, I can only get the target accuracy to ~50% until the source accuracy drops


texts:
I am playing with convolution network, assembling something between AlexNet and ResNet. Not very deep, about 10 conv. layers including 2 through residual connection, and 3 fully-connected layes at the end. Relu activations, but also tried ELU and sigmoid a bit. Achieved good results with 25 classes (up to 80% top-1 accuracy) and now trying on ImageNet with 1000 classes. And what I get is often stucking near random-quality choice (i.e. accuracy of 0.1% on 1000 classes and corresponding loss) or improving up to like 0.5% accuracy and then dropping back.

This happens with recommended for Adam learning rate of 0.001, a bit less, 2e-4, as well with much smaller l. rates, like 1e-5. What is strange, with small l. rates everything is even more ugly, in contrast to expected slow but reliable learning.

What I have found is that activations starting from middle levels start to decrease a lot. So different channels look like

(base image is ).

If we plot channels’ min-max changes through epochs, we’ll see
 This is for two different images, with dotted line and on the right axis – standard deviation of activations.
The corresponding train loss improvement and worsening is dimly visible here, blue line:

What is this? Vanishing gradients? Is the network too deep? Shouldn't batch normalization struggle with that? What is the solution, to adjust learning rate continuously?

And everything happens quite fast. I use mini-epochs, 20000 images each, so 1 on my plots is actually 1/60 of normal epoch.

I was able to train 4 times less wide variant of the network on 110 classes, got 42% accuracy, looks reasonable. There I don’t see such big drops. Didn’t notice improvement with SGD, different batch sizes, but noticed that without augmentation learning is faster and less prone to this problem.

Update: I was able to reproduce this on smaller net :) . It learns well with learning rate 5e-4 and batch 64, then 2e-4 and then doesn’t learn much with l.r. 1-e4 and batch like 384 or bigger or quickly degrades with l.r. like 5e-5 or less.
I didn’t mention one of tricks I use: squeeze-and-excitation. So it can apparently lead to massive activations changes. And really there are excitation layers with activations in order of 10-50, even while everything looks fine.
Now trying to struggle with this without success for some reason. I added

def near1Regularizer(activationVector):                  # <----
    return 0.01 * K.mean((1 - activationVector) ** 2)

…
se_feature = Dense(channel // ratio,
                       activation='relu',
                   name='dense_exc_%d' % (g_excLayerCount * 2 - 1),
                       # kernel_initializer='he_normal',
                   kernel_initializer=My1PlusInitializer(1.0 / 256),
                       use_bias=True,
                       bias_initializer='zeros')(se_feature)
assert se_feature._keras_shape[1:] == (1,1,channel//ratio)
se_feature = Dense(channel,
                   activation='sigmoid',
                   name='dense_exc_%d' % (g_excLayerCount * 2),
                   kernel_initializer='he_normal',
                   use_bias=True,
                   bias_initializer='zeros',
                   activity_regularizer=near1Regularizer)(se_feature)    # <----
se_feature = keras.layers.multiply([input_feature, se_feature])

and getting

The regularization applied near 90 by x. The error also started to increase after a dozen of mini-epochs...

Update 2: reproduced on simpler and smaller net (3 convolution blocks with squeeze-and-excitation and 3 dense blocks) and on small AlexNet (1/16 wide and 110 classes). Very surprizingly, convolution layers activations in order of 100s are very typical, they exist even in pretrained full AlexNet.


texts: Standarizing the Label to postive intergers in Machine Learning Classification task, why is it recommended?

texts:
When doing a classification task suppose we have 3 targets with notation -2, -1, 0.
I read somewhere it is a good practice to Standardize the labels to positive integers.. In this case suppose we change it to 2, 1, 0. Why is it recommended?
Theoretically I don't think it should matter since each class is unique. But in practice it matters, why?


texts: Implemented AR to predict a time series, get nonsense out

texts:
I have this time series, it's an industry metric, the data is reported daily, and the data spans just over 1 year. Sometimes, the data does not get reported, and for that day, we would just use the previous day's number for today.
I differenced the data and ran the augmented dickey fuller test, and the diff-data is indeed stationary. I tried to use autoregression to make predictions. I computed and plotted the autocorrelation, the lag contributions all seem very low, but nevertheless, i selected the highest lags and did
from statsmodels.tsa.ar_model import AutoReg
model = AutoReg(diff_data, lags=lags_list)

where lags_list are the lag indices in a python list, e.g. [1, 2, 5, 9, 19]
I checked to see if the model can give me fair numbers for the diff_data I have, and it gave me very wrong numbers. I don't know if I implemented AutoReg incorrectly, if I missed something or misunderstood something. I am sorta out of tricks now...
Perhaps the diff_data is basically white noise, the lag contributions were all very low
using
from pandas.plotting import autocorrelation_plot
autocorrelation_plot(diff_data)
plt.show()

pretty much every lag is inside the 99% confidence band, except for 2 out of 300+ lags, and just barely over the band. Does that mean the diff_data is white noise, if so, what should I do now?
On a separate topic, the prediction is supposed to be used when the real number doesnt come in, instead of copying yesterday's number. I used to think we can do better than this "copyover" method, but now... I'm not so sure anymore.


texts: What are the best practices/architectures to solve a Hierarchical multi-label classification problem using Neural Networks?


texts: Why should I use data augmentation as Keras layer

texts:
I've seen in several Tensorflow/Keras tutorials that data augmentation functions are added as keras layers. When I converted my Keras Python model (for production purpose) to TensorflowJS I faced the issue that e.g. the RandomFlip layer is not available in TensorflowJS. So I have to use the ImageDataGenerator.
I don't understand why should I put the data augmentation layers at all in the model, I mean when I want to use my model in production I'll still have them in my model which doesn't make sense for me.


texts: Learning to parse elements from text and assigning them to categories

texts:
I am trying to generate a table with values parsed from unstructured text. Below are a couple of examples of possibly thousands of entries. For each entry, I would like to identify the title, assign the movie to a category (e.g. "SciFi"), and so on.

"The Lord of the Rings Trilogy, Blu-Ray, Extended Edition",
"Lord of the Rings, DVD",
"Blu-Ray LOTR: The Two Towers",
"Star Wars: Episode 4"

Results should look like this:

Title                       Format    Category   Franchise
=====                       ======    ========   =========
Lord of the Rings Trilogy   DVD       Fantasy    LOTR
LOTR: The Two Towers        Blu-Ray   Fantasy    LOTR
Star Wars: Episode 4        NaN       SciFi      Star Wars

A naive approach would be to search for patterns such as "Lord of the Rings" or "Star Wars". Pattern search could be fuzzy as well. This, however, would require some kind of pre-defined mapping.

However, I wondered whether there is a smarter approach from data mining. I know text classification algorithms (e.g. spam classifiers), but they have only two classes.

I do not know much about natural language processing, and this does not really seem to be "natural language" in the stricter sense.

Can a parsing/mapping of such unstructured text be learned in any way (with sufficient training data)?

I am looking forward to get a bit of general inspiration on how to approach the problem.


texts: How to draw diagram like this one (errorsbar)

texts:
I would like to draw a graph that looks like this one from:

The closest I have come is this graph:

by this code:

df = pd.read_csv(csv_fname)
df['cluster'] = [3, 3, 1, 2, 1, 1, 3, 1, 1, 2, 1, 3, 2, 1, 1, 1, 2, 1, 3, 1, 3, 1, 3, 2, 1, 2, 3, 3, 2, 2, 1, 1, 2, 2, 2, 2, 2, 3, 2, 2, 3, 3, 3, 3, 1, 1, 1, 2, 3, 2, 1, 1, 1, 1, 1, 2, 3, 1]

def mean_plus_std(x): return np.mean(x) + np.std(x)
def mean_minus_std(x): return np.mean(x) - np.std(x)

dfeb = df.drop(columns=['COUNTY_NAME', 'PARTY_NAME']).groupby('cluster').agg([np.mean, mean_minus_std ,mean_plus_std])
fig, ax = plt.subplots(figsize=(20,10))
sns.catplot(data=dfeb.reset_index().melt(id_vars='cluster'), x='variable_0', y='value', hue='cluster', ax=ax)
ax.set_xticklabels(ax.get_xticklabels(),rotation=30, ha='right');

Data (csv) are from https://exploratory.io/data/kanaugust/2016-California-Election-Data-oTv4Hgd1UT

Output looks like this:

So for each column graph should present mean - std, mean, mean + std.
These three values should be drawn as three dots connected with line for each column.
Does someone have a clue how to do it in python by using matplotlib/seaborn?

EDIT 1:

How to draw errors bar for each ("Adult Film Condom Requirements" and "Ban on Single-use Plastic Bags") column if I know column's means and standard deviations with cluster as hue as it is presented in the first picture in this question?


texts: How do I correctly build model on given data to predict target parameter?

texts:
I have some dataset which contains different paramteres and data.head() looks like this

Applied some preprocessing and performed Feature ranking -

dataset = pd.read_csv("ML.csv",header = 0)

#Get dataset breif
print(dataset.shape)
print(dataset.isnull().sum())
#print(dataset.head())

#Data Pre-processing
data = dataset.drop('organization_id',1)
data = data.drop('status',1)
data = data.drop('city',1)

#Find median for features having NaN
median_zip, median_role_id, median_specialty_id, median_latitude, median_longitude = \
    data['zip'].median(),\
    data['role_id'].median(),\
    data['specialty_id'].median(),\
    data['latitude'].median(),\
    data['longitude'].median()

data['zip'].fillna(median_zip, inplace=True)
data['role_id'].fillna(median_role_id, inplace=True)
data['specialty_id'].fillna(median_specialty_id, inplace=True)
data['latitude'].fillna(median_latitude, inplace=True)
data['longitude'].fillna(median_longitude, inplace=True)

#Fill YearOFExp with 0
data['years_of_experience'].fillna(0, inplace=True)
target = dataset.location_id

#Perform Recursive Feature Extraction
svm = LinearSVC()
rfe = RFE(svm, 1)
rfe = rfe.fit(data, target) #IT give convergence Warning - Normally when an optimization algorithm does not converge, it is usually because the problem is not well-conditioned, perhaps due to a poor scaling of the decision variables.

names = list(data)
print("Features sorted by their score:")
print(sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), names)))

Output

Features sorted by their score:

[(1, 'location_id'), (2, 'department_id'), (3, 'latitude'), (4, 'specialty_id'), (5, 'longitude'), (6, 'zip'), (7, 'shift_id'), (8, 'user_id'), (9, 'role_id'), (10, 'open_positions'), (11, 'years_of_experience')]

From this I understand that which parameters have more importance.
Is above processing correct to understand the feature important. How can I use above information for better model training?

When I to model training it gives very high accuracy. How come it gives so high accuracy?

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

dataset = pd.read_csv("prod_data_for_ML.csv",header = 0)

#Data Pre-processing
data = dataset.drop('location_id',1)
data = data.drop('status',1)
data = data.drop('city',1)

#Find median for features having NaN
median_zip, median_role_id, median_specialty_id, median_latitude, median_longitude =\
    data['zip'].median(),\
    data['role_id'].median(),\
    data['specialty_id'].median(),\
    data['latitude'].median(),\
    data['longitude'].median()

data['zip'].fillna(median_zip, inplace=True)
data['role_id'].fillna(median_role_id, inplace=True)
data['specialty_id'].fillna(median_specialty_id, inplace=True)
data['latitude'].fillna(median_latitude, inplace=True)
data['longitude'].fillna(median_longitude, inplace=True)

#Fill YearOFExp with 0
data['years_of_experience'].fillna(0, inplace=True)

#Start training

labels = dataset.location_id
train1 = data
algo = LinearRegression()
x_train , x_test , y_train , y_test = train_test_split(train1 , labels , test_size = 0.20,random_state =1)

# x_train.to_csv("x_train.csv", sep=',', encoding='utf-8')
# x_test.to_csv("x_test.csv", sep=',', encoding='utf-8')

algo.fit(x_train,y_train)
algo.score(x_test,y_test)

Output: 0.981150074104111

from sklearn import ensemble
clf = ensemble.GradientBoostingRegressor(n_estimators = 400,
                                         max_depth = 5,
                                         min_samples_split = 2,
                                         learning_rate = 0.1,
                                         loss = 'ls')
clf.fit(x_train, y_train)
clf.score(x_test,y_test)

Output: 0.99

What I want to do is, predicting  location-id  correctly.
 Am I doing anything wrong? What ithe s correct way to build model for this sort of situation?

I know there is some way that I can get Precision, recall, f1 for each paramteres. Can anyone give me reference link to perform this.


texts: How to fight underfitting in a deep neural net

texts:
When I started with artificial neural networks (NN) I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even beat my score on random forest!

I'm seeking some very general or not so general advice on what should one do to make a NN start capturing trends in data.

For implementing NN I use Theano Stacked Auto Encoder with the code from tutorial that works great (less than 5% error rate) for classifying the MNIST dataset. It is a multilayer perceptron, with softmax layer on top with each hidden later being pre-trained as autoencoder (fully described at tutorial, chapter 8). There are ~50 input features and ~10 output classes. The NN has sigmoid neurons and all data are normalized to [0,1]. I tried lots of different configurations: number of hidden layers and neurons in them (100->100->100, 60->60->60, 60->30->15, etc.), different learning and pre-train rates, etc.

And the best thing I can get is a 20% error rate on the validation set and a 40% error rate on the test set.

On the other hand, when I try to use Random Forest (from scikit-learn) I easily get a 12% error rate on the validation set and 25%(!) on the test set.

How can it be that my deep NN with pre-training behaves so badly? What should I try?


texts: I have heard the term 'feature coverage' in machine learning. However I found no relative infomation after I googled this term. Could some one give me a definition of 'feature coverage' and some references?


texts: Why would a fake feature with random numbers get selected in feature importance?

texts:
I'm using a sklearn.ensemble.RandomForestClassifier(n_estimators=100) to work on this challenge:
https://kaggle.com/c/two-sigma-financial-news

I've plotted my feature importance:

I created a fake feature called random which is just numbers pulled from np.random.randn(). Unfortunately, it seems to have quite significant feature importance.

How am I supposed to interpret this? I had expected it to be at the bottom.

PS xgboost seems to discard this feature, as it should.


texts: Is padding the right way to allow your model to make prediction with test sequences of shorter lengths?

texts: Say I have a RNN-lstm encoder-decoder model trained on fixed timesteps (no padding when training, all sequences are treated as if having the same lengths). My testing criteria requires me to provide input sequences with lengths shorter than the fixed length on which my model was trained. In this case, if I want to use model.predict() to make predictions, I would have to pad the test sequences in order to provide arrays with the same dimensions (which is required by the model.predict() method). I am wondering if it's ok to just stick to model.predict() and pad the shorter sequences if I want to make predictions with shorter test sequences?


texts: Can I use the Softmax function with a binary classification in deep learning?

texts:
I want to create a deep learning model (CNN) for binary classification, can I used the softmax function instead of the sigmoid function in binary classification?

Adding the classification layer to the model, will be like this

model.add(Dense(2, activation='softmax'))


texts: Assessing model performance on different sub-segments

texts:
I am currently working on a credit risk related project where i built a binary logistic regression model for an imbalanced dataset.
According to the regulations i have to prove that the model performs well on different subsets of data (e.g. age-group [18, 25] compared to age-group [26, 40], mortgages compared to consumer loans, high/low income). Typically, the sub-Segments would be indicated by a binary variable, but it could also be that there are more than two Segments to compare.
I spent the whole day looking for possible solutions but so far I did not find something particular useful for this challenge. Unfortunately, it is not enough to show that AUC does not drop significantly on each of the sub-segments.
Are there any of you who already have experience with these kind of problems?
Thank you very much!


texts: Why is neural language model performing so poorly?

texts:
I am trying to create a word-level Haiku generator using an LSTM neural network. I am scraping haikus from Reddit's r/haiku, and wanted to start with a "simple" model: my training data is the set of all haikus, flattened, and split into trigrams, such that the first two words of the trigram are the features and the last word is the label. I chose a "context size" of two just because I wanted to start with a small size, considering how small the haikus themselves are. So this is essentially a text prediction task with a small context window.
E.g the first couple (X,y) pairs (There are ~ 90,000 such pairs) look like:
[('delicate savage', '/'),
 ('savage /', "you'll"),
 (" / you'll", 'never'),
 ("you'll never", 'hold')... ]

(I am using the '/' character to designate a new stanza, and I also added '$' to indicate the end of the haiku, to see if my model will learn such distinctions present in a haiku's format)
I then encode my sequences using an index mapping and feed this into an embedding layer when I train my model.
The model:
Sequential([
    Embedding(input_dim=vocab_size, output_dim=60, input_length=seq_length), #seq length is 2
    LSTM(100,return_sequences=True),
    LSTM(100),
    Dense(100,kernel_regularizer=l2(0.01),activation = 'relu'),
    Dense(vocab_size, activation = 'softmax')
])

My performance is pretty bad: about 45% training accuracy and 15% validation accuracy. When I generate haiku sequences using a fit model, the model is often clueless and returns None.
I recognize that there are several places my model could be going wrong, but am really not sure where. I've tried various regularization techniques (dropout, L2) and modifying my neural network's architecture (# of layers and nodes) but without any luck. Before I try using pre-trained embedding, what can I do to improve my ability to predict text with my data?
If it helps, here's my notebook, using a language model heavily based on this implementation.