texts:
This might be an unusual question. I have a situation where I am creating paraphrases with a rule based system. One transformation that I'd like to implement would one that gets rid of light verbs, like so:

a) Steven made an unwilling concession to us.

b) Steven unwillingly conceded to us.

To go from (a) to (b) requires some inflectional changes such as:

unwilling -> unwillingly

concession -> conceded

Is there something out there that can take care of such inflectional changes reliably for a given word? Seems like there should be, but I don't recall seeing anything like this.


texts: Difference between OrdinalEncoder and LabelEncoder

texts:
I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing:

In the Documentation it is given about sklearn.preprocessing.OrdinalEncoder() whereas in the book it was given about sklearn.preprocessing.LabelEncoder(), when I checked their functionality it looked same to me. Can Someone please tell me the difference between the two please?


texts: How to implement N-Dimensional Convolution in TensorFlow / Keras? (with N > 3)

texts:
I had an idea for a model that requires multidimensional convolution (i.e. 4D or more). However, I can't find anything higher than 3D Conv on the TF 2.0 / Keras module.

Is there a way to implement multimensional, ND Conv? (N > 3)


texts: Which machine learning approach/algorithm do I choose for path validation?

texts:
I apologize for lack of terminology, I'm no computer scientist.
I have a problem of validating paths in a directed graph with complex nodes.

The full description is the following:

I have a decent set (about 1K) of directed graphs;
Each node contains a complex data structure (it is a hierarchical data structure, not a picture or sound);
I have some of paths in those graphs known as "correct" paths (based
mostly on data in nodes);
And I have some of paths in those graphs known as "incorrect" paths
(with classification why it is incorrect).

I'd like to predict given a graph with those complex nodes and a path, is this path "correct".

Which machine learning algorithm will suit me best?
In general, what approach I should use?

Edit:

Each full graph is either have app paths processed
(correct/incorrect) or completely blank (no path is processed);
Correctness depends on both position of node in a graph AND data in
the node;
Humans would need heuristics to decide or guess which paths are
correct;
Most of the paths are "correct";
I hope to convert human heuristics to some kind of "correctness"
recognition.


texts: Is there a way to cluster words based on how similarly they sound?

texts:
I have a list of words for a fictional world I've made (don't judge lol).

My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build a model that will classify words into their region by similarity to the other words in that region.

My first attempt used Levenshtein distance, and this performed... well, not very good.

I then tried using the Jaro-Winkler distance plus the Jaro-Winkler distance for the inverted string divided by two, to get the average distance (to refresh, Winkler considers the beginning of the word to be more important, so I did this to get the end as important too). This was better, with 80% train/25% test accuracy, but obviously not good enough at all.

As you can see, for the multiclass models I'm trying to use, these are poor predictors of the regional class I've assigned to each word. I have only found this:

https://www.oreilly.com/library/view/python-cookbook/0596001673/ch17s09.html

As any sort of mechanism that takes sound into account, but it seems like words have to be similar length, and mine aren't necessarily within one or two character lengths of each other.

I mean, might be able to apply it, but I wanted to see if there was a better solution. My research hasn't turned up anything useful.


texts: Question about "1x3 and 3x1 conv is equivalent to 3x3 conv"

texts:
I see a lot of sites talk that we can substitute 1x3 conv + 3x1 conv for 3x3 conv.

In order to demonstrate easily, we use a 3x3 image as an example.

From the point of view of parameters, I know that we will get fewer parameters and end up with the same result.
If the image(3x3) perform a 3x3 conv, we will get a 1x1 scalar, and this kernel size is 3x3, means we have 9 parameters.
On the other hand, if the image first performs a 1x3 convolution, and then performs a 3x1 convolution on its output. It will intuitively get the same result, a 1x1 scalar, and this kernel size is 2x3x1 = 6, which means we have 6 parameters.

However, from the point of view of calculation time, I don't think 3x3 should be substituted by 1x3 & 3x1.

For example, if we use the image to perform a 3x3 kernel, we should do 9 times of pixel-wise multiply operations, and then one time of addition. Totally 10 operations to get the result.

But, if we use the image(3x3) to perform a 3x1 kernel, we have to implement [3(multiply) + 1(add)] * 3 times to get a 1x3 feature map, then we still need to perform a 1x3 kernel. It will then takes 3(multiply) + 1(add) operations. Totally 16 operations throughout the whole process.

What I want to show is that although we reduce the parameters from replacing 3x3 conv to 3x1 & 1x3 conv, we still need more time to calculate during the training process.

Training spends lots of time and implements 3x1 & 1x3 doesn't save our time. So, should we substitute 1x3 conv + 3x1 conv for 3x3 conv?


texts: Practical way to convert jupyter notebook to MS Word document?

texts: What would be a practical way to convert a Jupyter Notebook to a Word document (.doc) ? I am asking this in a professional context, so I'd like to avoid manual solutions, do it in an efficient way (fast), avoid third parties... etc. Something that works like Rmarkdown to produce .doc would be very welcome.


texts: Beyond one-hot encoding for LSTM model in Keras

texts:
I have an LSTM model in Keras for categorical classification (20 possible categories). In many cases, my data can fit multiple categories.

Obviously, my current model uses one-hot encoding and fits on that - that gives me accuracy and validation rates in the 50-60% but I want to improve that by comparing how the model does against the top 3 categories that the algorithm chooses.

Right now, I use Keras with categorical_crossentropy. I presume that this checks to see if the label is the top match and bases the accuracy on that matching. How can I modify the fit/training of the model to allow the labeled category to be in the top 3 (or top X-number) of matches for the accuracy score?


texts: Overfitting results with Random Forest Regression

texts:
I have one image that contains for each pixel 4 different values.
I have used RF in order to see if I can predict the 4th value based on the other 3 values of each pixel. for that I have used python and scikit learn. first I have fit the model, and after validate it I used it to predict this image.
I was very happy and scared to see that I got very high accuracy for my model : 99.95%!
but then when I saw the resulted image it absolutly wasn't 99.95% of accuracy:
original image:

result image:

(I have makrd the biggest and most visible difference).
My question is- why would I get this high accuracy when the visualization shows very well that there is much less accuracy? I understand it might come from overfitting but then how this different is not detected?
edit:
Mean Absolute Error: 0.048246606512422616
Mean Squared Error: 0.00670919112477127
Root Mean Squared Error: 0.0819096522076078
Accuracy: 99.95175339348758


texts: How to build a model on a dataset having 40% missing values in most of the variables?

texts:
I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category) and addresses of the companies.
What may go wrong if I build a model on other 60% records(Non-missing values)? Can you suggest any measures/actions to take before spending time on building the model?

Couple of things I planned to do to ensure 60% records represent the population:

Distribution of companies by industry category/state for 60% records(Non-missing values) vs 100% records

Target variable distribution of 60% records(non-missing) vs 100% records


texts: How relevant is Self Organizing Maps in today's science?

texts: Self-Organizing Maps is a pretty smart yet fast & simple method to cluster data. But Self-Organizing maps were developed in 1990 and a lot of robust and powerful clustering method using dimensionality reduction methods have been developed since then. What are some of its applications in today's world of science and engineering?


texts: cannot use sklearn.naive_bayes MultinomialNB to predict from one attribute

texts:
I tried to create a simple example for Naive Bayes by learning from one attribute.  It seems like I cannot use sklearn.naive_bayes.MultinomialNB to predict from one attribute.  This is because the predict_proba is the same for every input.

Here is the code:

import numpy as np
X = np.array([1, 1, 1, 0, 0, 1, 1])
X = X.reshape(-1,1)
y = np.array([0, 0, 0, 0, 1, 1, 1])

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB()
clf.fit(X, y)

print(clf.predict(X))
print(clf.predict_proba(X))

and its result.

[0 0 0 0 0 0 0]
[[0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]
 [0.57142857 0.42857143]]


texts: Exploration in Q learning: Epsilon greedy vs Exploration function

texts:
I am trying to understand how to make sure that our agent explores the state space enough before exploiting what it knows. I am aware that we use epsilon-greedy approach with a decaying epsilon to achieve this. However I came across another concept, that of using exploration functions to make sure that our agent explores the state space.
Q Learning with Epsilon Greedy
$sample = R(s,a,s') + \gamma \max_{a'}Q(s',a')$
$Q(s,a) = (1 - \alpha)*Q(s,a) + \alpha*sample$

 Exploration function
$
f(u,n) = u + k/n $
$Q(s,a) = R(s,a,s') + \gamma*max_{a'}f(Q(s',a'), N(s',a'))$
Where N(s',a') counts the number of times you have seen this (s',a') pair before.
Now my question is, are these 2 above strategies just 2 different ways of Q learning? Because I can't seem to find much details on exploration function but can find plenty on epsilon greedy Q learning. Or can the exploration function be used along with the epsilon greedy Q learning algorithm as a form of some optimization? I am confused as to where exactly would we make use of this exploration function Q learning strategy. Any help/suggestions are much appreciated!


texts: Why do I have leakage while using Stratified Group K Fold?

texts:
I have the following case:
Training data in the form of x, y coordinates on different frames (from a video). Based on this I computed some features, using only the training data and labels. A model is being trained with GridSearchCV.
It's a multiclass classification problem. There are too many coordinates available, some of which are not used to make features. There are also categorical features. The data is not balanced.
My pipeline consists of a preprocessor, that scales the numerical features with StandardScaler() and encodes categorical features with OneHotEncoder (actually those features are already encoded, but I didn't find another way to neglect those while scaling)
By applying SelectFromModel I reduce the number of features usually with threshold of '1.1*mean'
I am using stratified_group_k_fold from Kaggle (https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation)
My data consists of different people, so I set the different people as groups. It seems that no matter which model I train, my CV score is on average 0.14 higher than the test score and never less than 0.12 higher.
Here's a snippet of my code.
# number of features
num_features = X_tr.shape[1]

# number of categorical features
num_categorical = int(np.sum([np.array_equal(X_tr[:,i], X_tr[:,i].astype(bool)) for i in range(X_tr.shape[1])]))

#start index of categorical features
offset_cat = num_features-num_categorical

numeric_features = slice(0, offset_cat, 1)
numeric_transformer = Pipeline(steps=[('scaler', StandardScaler())])

onehot_features = slice(offset_cat, num_features, 1)
cat_list = [[0,1] for i in range(num_categorical)]

onehot_transformer = Pipeline(steps=[('onehot', OneHotEncoder(categories=cat_list))])

preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
    ('onehot', onehot_transformer, onehot_features)
])

from sklearn.svm import SVC
from sklearn.feature_selection import SelectFromModel

scorer = make_scorer(custom_loss_function, greater_is_better=True, needs_proba=True)
cv = stratified_group_k_fold(X_tr, y_tr, all_people, k=5, seed=42)

pipe = Pipeline(steps=[('preprocessor', preprocessor),
                       ('select', SelectFromModel(SVC(kernel='linear'), threshold='1.1*mean')),
                       ('clf', SVC(probability=True, random_state=42))])

param_grid = {
    'clf__C': Cs,
    'clf__gamma': gammas,
    'clf__random_state': [38]
}

search = GridSearchCV(pipe,
                      param_grid,
                      cv=cv,
                      scoring=scorer,
                      iid=True, n_jobs=-1,
                      return_train_score=True,
                      verbose=10)

search.fit(X_tr, y_tr, groups=all_people)

best_params = search.best_params_
cv_results = search.cv_results_

for k in param_grid:
    print(k, '=', best_params[k])
print('\n' +
      'Training set score (map@3):' + str(search.score(X_tr, y_tr)))
print("Cross validation score:   ", search.best_score_)

clf__C = 12.50075
clf__gamma = 0.001
clf__random_state = 38
Training set score (map@3):0.9586247086247086
Cross validation score:    0.7175901022054868

While the test score on Kaggle is only 0.5713
I want to know if I'm doing something wrong here that could cause leakage. Specifically if there is something wrong with the stratified_group_k_fold because it doesn't seem to matter if I use this or just normal 5 fold cv by setting cv=5 in GridSearchCV
From the feedback I got, I know this gap could be closed. Unfortunately I only find ways to increase leakage instead of decreasing it. For example I tried removing outliers based on z-statistics of the features columns, but that only increased my CV-score and not the test score.
Main question:
Why is there leakage? Am I doing something wrong with Stratified Group K fold?
Side questions:
Is it normal for the CV score to be that much higher? What else could cause leakage in this case? Why does removing outliers increase leakage? Is there something I can do better?
Thanks for any help!


texts: What is the simplest neural network for the simplest nonlinear function $f(x,y) =xy$

texts:
How do I capture $y = x_1 x_2$ using a simple neural network with commonly used activation functions? I assume that I need at least one hidden layer. What mix of commonly activation functions should I use?

So far, I have used $max(0,x)$ and $tanh$ activation for the hidden layer, but the gradient descent diverges very quickly.

Some thoughts which may be useful:

$$ (x_1 y_1)^2 = \exp(\log(x_1^2) + \log(x_2^2)) $$


texts: Papers presenting results that are worse than random chance

texts:
Is it me or has there been an increasingly large amount of object detection papers describing models that are performing worse than chance.  Here is an example (an extract so not to name names):-

AP represents the average precision.  No mention of recall.

The paper goes on to say that YOLOv3-Lite reaches state of the art performance in detecting the specified object. This table would suggest to me that the models are worse just flipping a coin. What am I missing here?


texts: How can I imporve accuracy for text classification and mapping using SVM?

texts:
I am working on a problem where I need to predict the text corresponding to another text in my training data file. For example: if I have value like the software in one of my columns and another corresponding column holds a value adobe pdf for it then my algorithm should be able to predict the same for my test data as well. For example, if my test data has Tableau then the predicted category should be software corresponding to it. here are data options like Software, Tax, etc and corresponding to them there are subcategories. My job is to predict the right sub-category against the primary category in my test data.

What should I do to increase my accuracy? Is n_ngram affecting the accuracy a lot? I could see some but not a lot.

from sklearn.featureextraction.text import CountVectorizer from sklearn.featureextraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import LinearSVC
#preparing the final pipeline using the selected parameters
model = Pipeline([('vectorizer', CountVectorizer(ngramrange=(1,4))), ('tfidf', TfidfTransformer(useidf=True)),
('clf', OneVsRestClassifier(LinearSVC(class_weight="balanced")))])

I have already used SVM and linear SVC for doing the classification but my accuracy is only 78%. My training data set is in German with no stop words as these are categories and not plain long text.


texts:
In Keras custom loss function, I often see axis parameter is set to -1.

def custom_loss(y_true, y_pred):
    intersection = K.sum(K.abs(y_true * y_pred), axis=-1)

I always thought that it should be 0, as 0 axis represents the batch. So why is set to -1?


texts: larger batches decrease learning rate because of a technical artifact?

texts:
I'm training a neural network for a classification task and experimenting with different batch sizes. I'm using the negative log likelihood loss averaged over the samples in the batch.
I realized that because I'm keeping the number of epochs and the learning rate constant, and because I'm averaging the loss over the samples in the batch, I get slower convergence when I use larger batches simply because when I double the batch size, I'm doing half the learning steps...
How can I fix this technical artifact and study the real effect of batch size (and homogeneity) for my task like in here? Should I just stop averaging and instead sum the loss over the batch samples?


texts: Convert Numpy array with 'n' and 'y' into integer array of 0 and 1

texts:
I have a NumPy array of strings: 'n', 'y', wanna convert it into integer array of 0, 1, how to convert it?

imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
X = imp.fit_transform(X)

X

array([['n', 'y', 'n', ..., 'y', 'n', 'y'],
       ['n', 'y', 'n', ..., 'y', 'n', 'y'],
       ['n', 'y', 'y', ..., 'y', 'n', 'n'],
       ...,
       ['n', 'y', 'n', ..., 'y', 'n', 'y'],
       ['n', 'n', 'n', ..., 'y', 'n', 'y'],
       ['n', 'y', 'n', ..., 'y', 'n', 'n']], dtype=object)


texts: How to select variables based on the mean correlation in a correlation matrix?

texts:
I have a set of independent variables and I am calculating the correlation matrix between them using the Pearson Correlation Coefficient in Python. A part of the matrix looks like this:

From this matrix, suppose I want to find out the strongly correlated components between the variable NoOfDoors and the rest(Symboling...Compression Ratio). The process I have adopted is that I have taken the mean of that column(which is calculated as 0.039604) and based on that, I have only considered those values greater than 0.039604.

Based on that, the following variables have been selected as strongly correlated:

(Make, Aspiration, Wheel Base, Length, Width, Height, Curb Weight, Engine Type, Bore, Compression Ratio)

I want to ask, is this selection correct? If yes then is there an efficient way to do this? And if no, what is the correct way? Since I am new to this field, a well explained article would be appreciated. Thanks!


texts: Using SMOTE for Synthetic Data generation to improve performance on unbalanced data

texts:
I presently have a dataset with 21392 samples, of which, 16948 belong to the majority class (class A) and the remaining 4444 belong to the minority class (class B). I am presently using SMOTE (Synthetic Minority Over-Sampling Technique) to generate synthetic data, but am confused as to what percentage of synthetic samples should be generated ideally for ensuring good classification performance of Machine Learning/Deep Learning models.

I have a few options in mind:- 1. The first option is to generate 21392 new samples, with 16904 majority samples of class A and remaining 4488 minority samples of class B. Then, merge the original and synthetically generated new samples. However, the key drawback I believe is that the percentage of minority samples in my overall dataset (original+new) would remain more or less the same, which I think defeats the purpose of oversampling the minority samples. 2. The second option is to generate 21392 new samples, with 16904 majority and remaining 4488 minority samples. Then, only merge the original data with the newly generated minority samples of the new data. This way, the percentage of minority (class B) samples in my overall data would increase (from 4444/21392 = 20.774 % to (4444+4488)/(21392+4488) = 34.513 %. This I believe is the purpose of SMOTE (to increase the number of minority samples and reduce the imbalance in the overall dataset).

I am fairly new to using SMOTE, and would highly appreciate any suggestions/comments on which of these 2 options do you find better, or any other option which I may consider alongside.


texts: Will oversampling help with generalization (small imbalanced dataset)?

texts:
I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features.
I performed Recursive Feature Elimination (RFE) and stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic regression and/or SVM. I don't fully trust the AUC I got because I think it will not generalize correctly because of such a small positive class.
So, I was thinking on oversampling (K-means + PCA) the minority class and re-run the RFE approach, would this help?
Thanks.
My question is more or less the same as this one: Why will the accuracy of a highly unbalanced dataset reduce after oversampling?
but I do use AUC.


texts: How to cluster Houses on the basis of similarity of features+location?

texts:
I have a dataset of houses like this:

HouseID  Latitude Longitude PriceIndex
  1          1.4     103.120    1.21
  2          1.42    103.112    2.01

I want to find houses which are similar to each other both on the basis of their position as well as their price index.[Also would need to Rank in order of similarities, given one house] I tried using hclust package in R  and was able to extract 9 classes. However the groups don't seem to have any interpretable similarities (for example the points are spread all across the city etc). I haven't done clustering based projects before so any help in the right direction will be helpful. Thanks!

Edit: Removing the price index column from the clustering data-set actually clusters spatially. But adding the price shows only price-based clustering


texts: What quantile is used for the initial DummyRegressor for Gradient Boosting Regressor in scikit-learn?

texts:
According to the documentation of Scikit-Learn Gradient Boosting Regressor:

init: estimator or ‘zero’, default=None: An estimator object that is used to compute the initial predictions. init has to provide fit and predict. If ‘zero’, the initial raw predictions are set to zero. By default a DummyEstimator is used, predicting either the average target value (for loss=’ls’), or a quantile for the other losses.

So what quantile is used for the DummyRegressor if the loss function is 'huber'? Is it the 50-quantile, ie. median?
I need this information because I am reconstructing the predictor for the Gradient Boosting Regressor for use in another software environment.


texts: How to Monitor ML Classification Models in production?

texts:
I've often heard of measures like Population Stability Index and Characteristic Stability Index. I might be mistaken, but these seem to be more applicable towards looking at the changes in univariate distributions and are more linear.

Are there any methods that are more robust in capturing multivariate relationships and interactions?


texts: Is there a Feature selection process for ARIMA model?

texts:
I have a dataset representing sales per day for certain products. It contains 30000 observations and 6 features (target included). Since my task is to make a prediction about the number of pieces sold, I decided to use an ARIMA model (following this tutorial here).

I realized that the only preprocessing step for auto ARIMA is removing all the columns except target. (since auto ARIMA trains on the previous values).

So, my question is: is it common to not perform any feature selection and only keep the Target variable in ARIMA?

Thank you.


texts:
Suppose a training dataset contains the following inputs:
company size
number of employees
turnover
average salary
country
years of operation

...and outputs:
budget allocated to marketing
budget allocated to sales
budget allocated to R&D
budget allocated to training
budget allocated to shareholders

It also contains the column:
profit

...which is what I want to optimise against (i.e. the primary motivation of the model).
For any given set of inputs (company size, number of employees, turnover, average salary, country, years of operation), I want to be able to predict values for the 5 outputs which are most likely to achieve the highest 'profit'.
In this example, would 'profit' be an input, an output, or neither?
I don't believe it's an input, as profit won't be available to input into the model, and I don't believe it's an output, as am not trying to predict profit for a given set of inputs. Rather I am trying to predict the outputs that are most likely to generate the highest profit for that set of inputs (based on previous training data examples).
Is it possible to therefore specify some sort of 'motivation' variable which the model strives to maximise?


texts: Multi-label text classification with minimum confidence threshold

texts:
Building out a system that tries to apply zero or more predefined labels to text.

For each label, we've:

built out a reasonably good vocabulary of high-value words/features
developed a corpus containing thousands of labeled entries
trained a NaiveBayesClassifier for each topic that does a good job of classifying valid vs noisy content

The problem seems to be that the individual classifier is great at differentiating between valid & noisy content WITHIN a topic:

"the green energy bill will revolutionize..." (green = "green energy")
"the green bay packers went on to lose their..." (green != "green energy")

...but when classifying content that shouldn't match ANY topic it has a very high rate of false positives. There's no "everything else" label!

tl;dr it's good at subtle, in-topic differentiation, but terrible at broad topic labeling

Are there any algorithms that help you classify into N categories, but allow for "everything else" which might not fit into ANY of the categories?


texts: How to approach the numer.ai competition with anonymous scaled numerical predictors?

texts:
Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web.

The system has changed from time to time and the set-up today is the following:

train (N=96K) and test (N=33K) data with 21 features with continuous values in [0,1] and a binary target.
The data is clean (no missing values) and updated every 2 weeks. You can upload your predictions (on the test set) and see the log-loss. Part of the test data is even live data and you get paid for good predictions.

What I would like to discuss:

As the features are totally anonymous I think that there is not much feature engineering we can do. So my approach is very mechanical:

inspired by this I use a classification algorithm to filter out those training data which fit to my test data best.
Figure out some nice preprocessing
train nice classification algorithms
build ensembles of them (stacking, ..).

The concrete question:

Concerning step 1: Do you have experience with such an approach? Let's say I order the probability of train samples to belong to test (usually below 0.5) and then I take the largest K probabilities. How would you choose K? I tried with 15K .. but mainly to have a small training data set in order to speed up training in step 3.

Concerning step 2: The data is already on a 0,1 scale. If I apply any (PCA like) linear transformation then I would break this scale. What would you try in preprocessing if you have such numerical data and no idea that this actually is.

PS: I am aware that because numer.ai pays people discussing this could help me make some money. But as this is public this would help anybody out there...

PPS:
Today's leaderboard has an interesting pattern:
The top two with logloss of 0.64xx, then number 3 with 0.66xx and then most of the predictors reach 0.6888x.

Thus there seems to be a very small top field and lot of moderately successful guys (including me).


texts:
In here Mentioned:

  If you apply SGD to features extracted using PCA we found that it is
  often wise to scale the feature values by some constant c such that
  the average L2 norm of the training data equals one.

I don't get how can i scale the feature values by some constant c. Would you please write some line of python code for this scaling?


texts: Is it possible to detect which field does a rotated "kaggle" contest data come from?

texts:
Imagine I setup a Kaggle competition with normalized stock data (e.g. price, volume, etc) plus a random rotation matrix (i.e. so that it's less obvious what the features are). Is it possible for a contestants to figure out that the data is stock-related? If yes, how?

EDIT:

The contest will be presented as a general machine learning contest with no specific background info. The data will not be time-series - just a set of samples, each containing a list of unnamed features plus a binary label.

When I asked the question, I was wondering more about whether contestants might figure out by examine the data using data/statistical analysis (e.g. PCA) or unsupervised machine learning methods (e.g. clustering).


texts: Suitability of Jaccard's distance for count vectorized features

texts:
I had an assignment in which we had to classify the cuisine and also give back the top-5 recipes based on given input. I did a count vectorization (countVectorize.transformer()) for the following data and then used Jaccard's distance to calculate the closest matches. Is this approach right or are there better distance metrics for my purpose?

Dataset : https://www.kaggle.com/c/whats-cooking/data

  {
  "id": 24717,
   "cuisine": "indian",
   "ingredients": [
       "tumeric",
       "vegetable stock",
       "tomatoes",
       "garam masala",
       "naan",
       "red lentils",
       "red chili peppers",
       "onions",
       "spinach",
       "sweet potatoes"
   ]
   },


texts: Data Extraction from images using NLP and ML

texts:
Hi i'm trying to extract data like name, planType, phone#, .. from images like insurance cards or licence cards with GoogleVision / Textract using some conditions  but it does not extract the correct content .

is there a way to use ML and NLP together to solve this problem

Thank you


texts: In backpropagation, scale is also important?

texts:
I think backpropagation is needed to find the direction of gradient decent method.

I also wonder, the scale is also important?

I heard some issue of vanishing(or exploding) gradient problem.
If the direction of the backpropagation is remained,
we still could apply gradient decent method(cause we still know the direction to update) and could finally get optimal answer.

If I'm correct, there is no real limit in deep learning?
I mean, although the speed may be slow, we can always finish the train work of neural network?


texts: Including spatial spillover for categorical data in R

texts:
I did a regression analysis with categorical data with a glm model approach, which worked fine. I have longitude and latitude coordinates for each observation and I want to add their geographic spillover effect to the model.

My sample data is structured:

Index DV IVI IVII IVIII IVIV Long  Lat
1     0  2   1    3     -12  -17.8 12
2     0  1   1    6     112  11    -122
3     1  3   6    1     91   57    53

with regression eq. DV ~ IVI + IVII + IVIII + IVIV

That mentioned, I assume that the nearer regions are, the more it may influence my dependant variable. I found several approaches for spatial regression models, but not for categorical data. When I try to use existing libraries and functions, such as spdep's lagsarlm, glmmfields, spatialreg, gstat, geoRglm, 'spatialprobit' and many more (I used this list as a reference: https://cran.r-project.org/web/views/Spatial.html). For numeric values, I am able to do spatial regression, but for categorical values, I struggle.

The data structure is the following:

library(dplyr)
data <- data %>%
  mutate(
    DV = as.factor(DV),
    IVI = as.factor(IVI),
    IVII = as.factor(IVII),
    IVIII = as.factor(IVIII),
    IVIV = as.numeric(IVIV),
    longitude = as.numeric(longitude),
    latitude = as.numeric(latitude)
  )

My dependant variable (0|1), as well as my independent variables, are categorical and it would be no use to transform them, of course. I want to have an other glm model in the end, but with spatial spillover effects included. The libraries I tested so far can't handle categorical data.

Any leads/ideas would be greatly appreciated.


texts: How to find False discovery rate of a ML model? Situation explained below

texts:
Lets say:- ML Model denied 1000 transactions
FDR= FP/(FP+TP)
I randomly picked 400 accounts and checked if the model denied good user or bad user.
Good user denied- 110
Bad user denied - 290
How will I calculate False Discover rate of the model. Can it be based on the sample that I checked manually and then extrapolate ?


texts: What loss term might enforce interpretability?

texts:
A neural net can be interpreted as a weighted directed graph. Interpretability of this neural net could be measured using properties of that graph. In particular, "modularity" might be useful, which I could formalize as "A graph is modular if a small number of edges can be removed in order to split it into smaller graphs that are themselves modular.". This would allow a reader to understand small portions of the neural net at a time because they only interface with the rest in limited ways.
My question is: What differentiable numeric properties of weighted directed graphs encapsulate interpretability and can thus be used as a loss term to enforce interpretability?


texts: Match between objective function and evaluation metric

texts:
Does the objective function for model fitting and the evaluation metric for model validation need to be identical throughout the hyperparameter search process?
For example, can a XGBoost model be fitted with the Mean Squares Error (MSE) as the objective function (setting the 'objective' argument to reg:squarederror: regression with squared loss), while the cross validation process is evaluated based on a significantly different metric such as the gamma-deviance (residual deviance for gamma regression)?
Or should the evaluation metric match the the objective function as much as possible, hence the root mean square error need to be selected as the evaluation metric?


texts:
I know about BERT and other solutions when you masking some words and try to predict them.
But let say I have a text:

  Transformer have taken the of Natural Processing
  by storm, transforming the field by leaps and bounds. New,
  bigger, and better models to crop up almost every ,
  benchmarks in performance across a wide variety of tasks.

And I cannot in advance say to BERT where masking is. I am looking for an algorithm which can understand where missing words are and after that predict them.


texts: Suggestions on non-linear dimensionality reduction for small, one-hot encoded dataset

texts:
I wish to apply non-linear dimensionality reduction on a very small dataset (less than 100 observations). The dataset is very sparse, of approx 20 columns, each containing either 0 or 1. It's the result of a set of non-exclusive on-hot encoding of features. Observations are of this kind:

---------------------------------
| a | b | c | d | e | f | g | h |
---------------------------------
| 0   1   1   1   0   0   0   0 |
| 1   1   0   1   0   1   1   0 |
| 0   1   1   1   1   0   0   1 |

...

I tried with t-SNE, but its stochastic nature leads to very different results each time I run my code.

Any suggestions for what shall I use? I also need a technique that is strictly non-hierarchical (such as PCA, for example, in which each factors explains just the "leftovers" of the variance of the previous factor; I don't want that).


texts:
I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based on how much their heart rate change over time. I am using machine learning to predict the subjects' emotions and i am getting acceptable result when tested separately on each dataset. However, i get even better result if i merge the two datasets.

I am not however sure if combining the two datasets is acceptable. As I am combining two somehow different datasets, will it create statistical bias? How should i report my finding in a journal paper?


texts: Which colour channel from a TIFF image do I have to use?

texts:
I'm going to use the following dataset to do semantic segmentation with U-Net network.
LGG Segmentation Dataset
This dataset contains brain MR images together with manual FLAIR abnormality segmentation masks. The images were obtained from The Cancer Imaging Archive (TCIA). They correspond to 110 patients included in The Cancer Genome Atlas (TCGA) lower-grade glioma collection with at least fluid-attenuated inversion recovery (FLAIR) sequence and genomic cluster data available. Tumor genomic clusters and patient data is provided in data.csv file.

I have found that brain images are in TIFF format, and they are in RGB (with three channels). I have opened one with Gimp:

I don't know if there is any special information on each colour channel because this question is strange (I don't understand it).
Which channel do I have to use if I want to use greyscale images? Or maybe I can convert them into greyscale.


texts: converting column from a factor to date format YYYY-MM-DDThh:mm:ss.SSSZ


texts: how to build a predictive model without training data neither historical data

texts:
I m trying to score "how much a product is expected in the market". I created some features:
How much this product is used each year.
Where was it used .
how many product for each country.
the main component of this product
compared the product with other similar products.
etc.
I built a big matrix/table with all these information.
I want to create a prediction model, where the input is a main component of a product and the output is ( expected with a pourcentage).

I dont have a training data ( just features without related results which indicates if the product is expected or no)
I dont have historical data to confirm the results of my work.

Some advices ?

Thank you all


texts: What all should I learn in sequence to be expert in Tensorflow and Google Keras provided I know Python and Numpy, Pandas, Pyplot basics? Please provide the resources too as most online tutorials are chargeable


texts: Technique to make NN (specifically autoencoders) more robust by artifcally making input data more sparse

texts: I believe some month ago I have read somewhere that autoencoders can respond better to sparse input data when trained with such. For example when the training data is not sparse, but many features of the test data are zero. Apparently randomly setting some features of the training data to zero can help to perform the model better on the test set. I desperatly tried to find the book/paper where I read it again, but I was not able to and I cannot remember the name for this. Does somebody know what this technique is called?


texts: Generate new sentences based on keywords

texts:
For example, for a domain specific neural network in Fashion, with the Keywords light, dress, orange, cotton. It could output: This gorgeous orange summer dress is great for wearing on sunny camping days. It's cotton fabric makes it very comfortable to wear.

Can someone please suggest the simplest way to achieve this?


texts: solving quadratic equations using machine learning?

texts:
I am attempting to solve quadratic equations using machine learning. I tried to write code using mxnet but fails because it is using linear regression while my problem is non-linear (as far as I know).

It is possible to predict the solution for arbitrary equations provided the at least one real root exists?


texts: Generating random numbers from best probability distributions?

texts:
I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test):

Best Distribution fits with parameters:
---------------------------------------

Distribution: lognorm
Parameters: (0.5921510072108613, -0.006454418407435666, 0.021090953536473916)

Distribution: expon
Parameters: (2e-06, 0.018877666177431276)

Distribution: pareto
Parameters: (11.057961210885452, -0.20440106940633995, 0.20436985884797232)

Here, i want to achieve two things:

Using the best parameters,how to generate the new synthetic data that represent our original data?.
Does the rvs method in python is useful?
Compare the actual and generated data distribution fitting as shown in below figure.


texts: So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level of complexity. Many courses in time series only deals with univariate time series, and now I'm supposed to work on a time series with 600 different factors with extreme loads of nan-values and present it in a simple and illustrating way. Plotting single univariate time series sort of doesn't make sense as there are still 600 other time series to consider. I was just wondering if anyone here have any tips on how to proceed? Any input would be appreciated.


texts: I am attempting to train an autoencoder on data that is extremely sparse.  Each datapoint is only zeros and ones and contains ~3% 1s.  Being that the data is mostly zero the autoencoder learns to guess zero every time.  Is there a way to prevent this from happening?  To give context this is extremely sparse data when you consider that the number of features is over 865,000


texts:
The YOLO model splits the image into smaller boxes and each box is responsible for predicting 5 bounding boxes.

My question is how does the model make these bounding boxes for every grid cell ? Does each box have a predefined offset with respect to say the center of the grid cell.

I AM NOT TALKING ABOUT THE FINAL BOUNDING BOX THAT ENCLOSES THE OBJECT
I am talking about the 5 predicting bounding boxes that are present for each grid cell.

Like for example if the smaller grid cell is located at say 50x50 (the center of it) then the bounding boxes should be at (50+5)x(50+5) or something like that

If not then how do the bounding boxes come to be ?

Paper - https://arxiv.org/pdf/1506.02640.pdf


texts: How to find anomalies/outliers in Panel Data?

texts:
I have panel data based on 900000 different entities with 384 time steps and the data is not normally distributed. I am looking for outliers/anomalies, this is unsupervised as I have no examples of anomalies/outliers.

Apart from clustering methods such as K-means, DBSCAN/HDBSCAN, what options do I have?


texts:
I have been looking for a dataset that includes:

Age
Hospitalized
ICU
Disposition (Recovered,Death)

Has anyone located one ?
Preferably from a western country as China's data is suspect.
I find it hard to believe CDC is not gathering this data.


texts:
I have a number of weather stations, and I know their positions. I would like to interpolate measurements from them for various other positions as a weighted average of these stations, but of course I need weights for this. I am thinking the most logical choice here from a physics point of view will be weighting by the inverse of the distance to the station, but I haven't quite convinced myself that that is right, and am not sure if I should maybe be using the distance squared, or maybe something in between.

Any comments? Are there any other reasonable alternatives I should be considering?


texts: Orange widget with OpenCV setMouseCallback

texts:
I am designing an Orange Widget for facial recognition.

I would like to draw a rectangle around faces captured in video by setting mouse events with cv2.setMouseCallback. I can capture the video in the image window (cv2.namedWindow("image")) with cv2.VideoCapture(0) but I am unable to call the MouseCallBack function in the image window.

I am new to Orange widget development and unsure if thecv2.setMouseCallback function can work inside of an OrangeWidget class. Any suggestions or help would be greatly appreciated.

Here is my code:

class MouseCallBackWidget(OWWidget):

    name = "MouseCallBack"
    # Short widget description
    description = "Draw rectangle"
    icon = "icons/DataSamplerA.svg"

    want_main_area = False
#   with a fixed non resizable geometry.
    resizing_enabled = False

    def MouseCallBack(self,event, x, y, flags, param):
        #global boxRefPt, boxCX, boxCY, boxW, boxH, click, mode_learn, completed_rect, ix, iy

        # check if the left mouse button is clicked.
        if event == cv2.EVENT_LBUTTONDOWN:
            if self.completed_rect == False:
                self.boxRefPt = [(x, y)]                     # record the starting (x, y) coordinate of a new rectangle.
                self.ix, self.iy = x, y
                self.click = True

        # check if the left mouse button is released.
        elif event == cv2.EVENT_LBUTTONUP:
            self.boxRefPt.append((x, y))                     # record the ending (x, y) coordinate a new rectangle.
            self.click = False
            self.completed_rect = True
            self.mode_learn = True

            (x1, y1) = self.boxRefPt[0]
            (x2, y2) = self.boxRefPt[1]

            self.boxCX = min(x1, x2)
            self.boxCY = min(y1, y2)
            self.boxW = abs(x1 - x2)
            self.boxH = abs(y1 - y2)

    def __init__(self):

        super().__init__()

        from AnyQt.QtGui import QIntValidator

        self.boxRefPt = []
        self.click = False
        self.mode_learn = False
        self.completed_rect = False

        self.ix = 0
        self.iy = 0
        self.boxCX = 0
        self.boxCY = 0
        self.boxW = 0
        self.boxH = 0

        self.cap = cv2.VideoCapture(0)
        cv2.namedWindow("image")

        cv2.setMouseCallback("image", self.MouseCallBack)

        while (True):

            self.ret, self.frame = self.cap.read()

            if self.completed_rect == True:

                cv2.rectangle(self.frame, self.boxRefPt[0], self.boxRefPt[1], (0, 255, 0), 2)
                cv2.putText(self.frame,'LEARN AREA', (self.boxCX, (self.boxCY - 12)), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 0, 255), lineType=cv2.LINE_AA)

            cv2.imshow("image", self.frame)
            cv2.waitKey(1)

```


texts: Does mini-batch gradient descent nullify the effect of stratification on the training data set?

texts:
In data pre-processing, stratified shuffle is used to ensure that the distribution of the original dataset is reflected in the training, test and validation dataset.
Mini-batch gradient descent uses random shuffling to ensure randomness in the mini-batches.
My doubt is- Why should we implement stratified shuffle on our dataset if it is going to be shuffled in a random manner later during training?


texts:
When using K-Fold CV, is it still useful to have a Train/Validation/Test split?

Or simply just a Train/Test? I.e. split up data into k bins, and leave one out for testing, train on the rest, and take average of the scores.


texts:
Consider a dataset A which has examples for training in a binary classification problem. I have used SVM and applied the weighted method (in MATLAB) since the dataset is highly imbalanced. I have applied weights as inversely proportional to the frequency of data in each class. This is done on training. I have used 10 folds cross-validation for training. After training, I get the confusion matrix on A:

80025 1
0 140

where the first row is for the majority class and the second row is for the minority class. There is only 1 false positive (FP) and all minority class examples have been correctly classified giving true positive (TP) = 140.

PROBLEM: I train again using more data points. Then, I  run the trained model on the a new unseen test data set B which was never seen during training. This is the confusion matrix for testing on B .

50075 0
100 0

As can be seen, the minority class has not been classified at all, hence the purpose of weights has failed. Although, there is no FP the SVM fails to capture the minority class examples.
I have not applied any weights or balancing method on B. What could be wrong and how to overcome this problem?


texts: Derivative of multi-output Gaussian Process

texts:
I am working on a project where I estimate transition and measurements models for a kalman filter using Gaussian Processes.
In order to linearize the models I require the Jacobian of the estimated Guassian Process.

For the single-output case this is no problem, But I am a little confused about how to do this for the multi- output case.

The posterior mean of the gaussian process would be

\begin{equation}
 \begin{aligned}
  \bar{f}_* &= \mathbf{k}(\mathbf{x}_* \, \mathbf{X}) K(\mathbf{X}, \mathbf{X}) ^{-1} \mathbf{y}\\
  &\stackrel{\triangle}{=} \mathbf{k}(\mathbf{x}_* \, \mathbf{X}) \mathbf{\alpha}
 \end{aligned}
\end{equation}

And the posterior Covariance

\begin{equation}
\begin{aligned}
C &= \begin{bmatrix}
k(X, X) & k(X, x_*) \\
k(x_*, X) & k(x_*, x_*)
\end{bmatrix}
\end{aligned}
\end{equation}

\begin{equation}
 \begin{aligned}
  \text{covariance} &= k(\mathbf{x}_*, \mathbf{x}_*) - k(\mathbf{x}_*, \mathbf{X})k(\mathbf{X}, \mathbf{X})^{-1}k(\mathbf{x}_*, \mathbf{X})^T \\
 \end{aligned}
\end{equation}

For the single-output case this is no problem

\begin{equation}
 \begin{aligned}
  \frac{\partial\bar{f}_*}{\partial \mathbf{x}_*} &= \frac{\partial k(\mathbf{x}_* \, \mathbf{X})}{\partial \mathbf{x}_*} \mathbf{\alpha}\\
  &= \left[ \Lambda^{-1} \tilde{\mathbf{X}}^T_* (\mathbf{k}(\mathbf{x}_* \, \mathbf{X})^T \odot \mathbf{\alpha}) \right] \in \mathcal{R}_{D \times 1}
 \end{aligned}
\end{equation}

However, my GP in this case maps $ f: R^2 \rightarrow R^2 $

e.g.

\begin{equation}
 \begin{aligned}
        \mathbf{x}_* &= \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \\
  P\left(\begin{bmatrix}
   y_1\\ y_2
  \end{bmatrix} \ \vert \cdots \right)
  &= N\left( f(
   \begin{bmatrix}
   x_1\\ x_2
  \end{bmatrix})
  \vert
  \bar{f}_*,  \mathbf{K}
  \right)
 \end{aligned}
\end{equation}

But I am unsure how to formulate the Jacobian matrix of this function since I do not know the individual functions for each output dimension as I would in a deterministic case.

Any help would be greatly appreciated.


texts: Constructing an image from a dense layer output

texts:
I am trying to reconstruct an image from a dense layer with is a concatenation of outputs from a
1) convolutional network with image inputs; and
2) dense layer with numerical inputs

The concatenated 1D tensor is fed to a dense layer which I need to reconstruct as an image.

The code I am using right now is as so:

merge_output = tf.keras.layers.concatenate([convolutional_model_output, numerical_model_output])
densem1 = Dense(8092, activation='relu')(merge_output)
#densem2 = Dense(512, activation='relu')(densem1)
densem2 = Dense(self.image_width*self.image_height*3, activation='relu')(densem1)
reshapem1 = Reshape(target_shape=(self.image_height, self.image_width, 3))(densem2)

convm1 = Conv2D(filters=32, kernel_size=3, padding="same" , activation='relu')(reshapem1)
convm2 = Conv2D(filters=3, kernel_size=3, padding="same" , activation='relu')(convm1)

However, the training is not able to converge to the true output and looks like a kaleidoscope of weird colors.

Is there something wrong with my approach? I do realize that concatenating convolutional and numerical features and reshaping them as an image might be a possible pain point, is there a better way to represent the layer or the specific problem?

Thanks in advance.


texts: Theoretical Question: Data.table vs Data.frame with Big Data

texts:
I know that I can read in a very large csv file much faster with fread using the data.table library than with read.csv that reads a file in as a data.frame. However, dplyr can only perform operations on data.frame.

My questions are:

Why was dplyr built to work with the slower of the two data structures?
When working with big data is it good practice to read in as data.table then convert to data.frame to perform dplyr operations?
Is there another strategy I am missing?


texts: What does IBA mean in imblearn classification report?

texts:
imblearn is a python library for handling imbalanced data. A code for generating classification report is given below.
import numpy as np
from imblearn.metrics import classification_report_imbalanced
y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
target_names = ['class 0', 'class 1', 'class 2']
print(classification_report_imbalanced(y_true, y_pred,target_names=target_names))

The output for this is as follow
                pre       rec       spe        f1       geo       iba       sup

 class 0       0.50      1.00      0.75      0.67      0.87      0.77         1
 class 1       0.00      0.00      0.75      0.00      0.00      0.00         1
 class 2       1.00      0.67      1.00      0.80      0.82      0.64         3

avg/total      0.70      0.60      0.90      0.61      0.66      0.54         5

What is the meaning of iba in this classification report. Here pre stands for precision, rec stands for recall, spe stands for specificity, f1 stands for f1 measure, and geo stands for geometric mean. All these are metrics for measuring performance of imbalanced classes.


texts: Steps in exploratory methods for mild-sized data with mixed categorical and numerical values?

texts:
Experienced in signal/image analysis, and new to data science, I recently was challenged with a relatively simple dataset: 100 to 200 items, about 10-20 numerical variables (in the [0-1] or percentage range), with only one variable used at present time for ranking, and 5 to 10 categorical variables, each with few options. A categorical variable takes about 2 to 4 different values.

I would like first to get insight on potential structures in such data. I have browsed Agresti's Analysis of Ordinal Categorical Data, some have advised me to invest on TDA (Topological data Analysis). Yet I do not know where to start from.

Do you have guidelines, best practices on such REAL data to gradually address the aforementioned issues, from visualization to genuine processing/inference?


texts: the relationship between the number of filters/kernels and the number of feature maps

texts:
The above image is from "Deep Learning Tutorial" by Yann LeCun and Marc'Aurelio Ranzato (see pages 73 and 81).

I don't understand why 64 kernels (from the input to layer 1) produce 64 feature maps while 4096 kernels (layer 2 to layer 3) give 256 feature maps.


texts: Clasification - ROC Curve with very high number of false negatives

texts:
I have the following curve as the result of a azure machine learning classification experiment.

This obviously shows a lot of false negatives, but I'm struggling to interpret this, as my data has a high number of negatives and a low number of positives (I'm doing a binary classification).
I am interpreting this as saying my data is almost entirely made up of positive values, with the majority being identified as negative. I believe that the false negatives on the chart are actually true negatives. Am I reading the chart correctly?


texts: How to teach algorithm to mimic paths in a certain enviroment

texts:
I have a set of scenarios which represent the movement of a car in a certain environment containing some obstacles. So for each scenario I have the position of the car (x,y,t) and a description of the environment (a grid where holes to represent the presence of obstacles). Therefore for each element of the training set I have:

label: List of car positions for each time instant (basically a trajectory)
features: List of grids describing the environment around the car for each time instant (changing environment)

My goal is to use some machine learning tool to teach it how the car should behave given a new sequence of environments (list of grids).

What machine learning tool should i use? Can you provide some example similar to what I am trying to do in order to have some inspiration?


texts: How to treat missing data for survival analysis

texts:
I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, after two weeks and so on till after 3 months. Now some patients dropout in between the treatment stage. The dataset I have consists of around 50 columns(10 questions repeated over 5 times during the course of treatment), but there are missing data for some patients as they dropout from the treatment.

My questions are:

how do I handle the missing data as it is not filled by the patient?

Should I impute that with mean values or is there any other way?

P.S.: I am new to survival analysis. So any help will be appreciated. Thanks in advance.

id  age sex dropout s1_q1 s1_q2 s1_q3 s1_q4 s1_q5.... s5_q10
217 50  m   0       2     3     3     3     2         3
202 58  f   0       4     9     10    10    10        N/A
222 72  m   1       3     8     9     10    9         N/A
207 50  m   0       2     7     6     7     7         6
277 55  f   0       2     4     5     5     5         6
281 62  m   0       4     10    10    10    10        10


texts: Create a new column based on two columns from two different dataframes

texts:
I have one column in the first dataframe called 'id' and another column in the second dataframe called 'first_id' which refers to the id from the first dataframe. I need to create a new column which has value 1  if the id and first_id match, otherwise it is 0.  I tried this but I am getting an error ValueError: Length of values does not match length of index.

df2['new'] = np.where(df2[df2.first_id.isin(df1.id.values)], 1, 0)

I understand why is it happening, because df2 and df2[df2.first_id.isin(df1.id.values)] are of different lengths but I can't make them the same. Any ideas?


texts:
What are xml datasets? Is it possible to convert them to csv files?
I'm working on a Java program and I sometimes download datasets wich are in a binary format, are those xml?

Thank you.


texts: Hill Climbing Algorithm - Optimum Step Size

texts:
I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach:

Apply grid search with large values applied to the hyper-parameter to find a 'best' starting point
Apply hill climbing algorithm in this space with a large number of different, random start points to find a local optimum

The large values that are passed in the first step, the grid search are

1*10^seq(-4, 5, by=1)
 [1] 1e-04 1e-03 1e-02 1e-01 1e+00 1e+01 1e+02 1e+03 1e+04 1e+05

So I am struggling to choose an optimum step size that isn't too large that is skips the peak or too small that it takes too long to converge. I don't think a single value for the step size is appropriate for all values passed in the grid search since, for example the difference between 1e-04 and 1e-03 is vastly different to 1e03 and 1e04. So I want the step size to be proportionate to the grid search start point. I know the search space I'm looking at is

grid_search_optimum/10 to grid_search_optimum*10

My question, therefore, is what is an accepted value for the step size in relation to the search space? I haven't found any opinions in literature around this and, in general, the only advice is to choose a step size that is "sufficiently small". Any advice or pointers to relevant papers would be greatly appreciated!


texts: Merging Training and Validation Sets for better accuracy

texts:
I am training an Image Classification Model and my train-test set distribution is 80-20.

After Training my train-test loss curve looks like this

As the model is converged after around 20-30 epochs and is not overfitting.
Can I also try to train the same model by merging my training and testing set hoping it would become more accurate with increased size of training data ?

Is it a good idea to merge train, test set in this case ?
What are the disadvantages if I do so ?


texts: High / low resources language : what does it mean?

texts:
In NLP, languages are often referred as low resource or high resource.

What these terms mean ?


texts:
I'd like to test a new algorithm for link prediction on graphs. I'm interested in  both weighted/unweighted and directed/undirected graphs.

What are some common benchmark datasets that researchers often use to test their algorithms? I know that within Computer Vision people often use MNIST or CIFAR, but I haven't found similar datasets for link prediction.


texts: Technique to determine variation in metric due to varying parameters

texts:
So basically I have a large set of features corresponding to a metric - like many ML problems.

What I want to know is: can we correlate the variance of metric with the variation in each feature.

ex:

I have features x, y, z that produce an output say 10. When I vary x, no matter how much I vary it, the output stays relatively close to 10. However, when I vary y the output is heavily influenced.

Is there a good technique to be able to assign a value correlating x and/or y to the metric?

I'm mostly looking for direction here.. i.e. techniques or relevant papers. In my experience I haven't really come across this problem. I don't have a good solution in my toolbelt.

Thanks!


texts: How to improve the accuracy of Random Forest for Text Categorization

texts:
I am working on a text Categorization problem, the objective is to classify related companies into their corresponding categories. This is a single category classification problem and not multi-class classification. Below is the details of the method I used.

Data Preparation -

Broke the documents in the list of tokens/ words.
Removed numbers, stop words, punctuations.
Performed stemming.
Convert corpus into DTM and,
specify the word length in-between(3, 10)
Transformed the documents into TF-IDF vectors
convert it into a matrix and further into a data frame

These will be used a feature list for the
classification algorithm. Used Random forest ("ranger" by using mlr).

I have 853 categorized documents with 20 categories, and I used 75:25 ration for training and test dataset. I used mlr package in R. The classification accuracy I have managed to get is 45% and I need it to be at least 80%.

I also applied xgboost but got only 38% accuracy.

Data Looked like the below-mentioned image: -

Here is an example code I used so far: -

terms <-DocumentTermMatrix(clean_corp,
                                      control=list(wordLengths=c(3, 9),
                                                   weighting = weightTfIdf))

m <- as.matrix(terms)
vertical_market <- process$label
termsDf <- as.data.frame(m)
termsDf$label <- as.factor(label)
names(termsDf) = lapply(names(termsDf), as.name)

words <- colnames(termsDf)

# taking words with length more than 4
data <- termsDf[, str_length(words) < 16 & str_length(words) > 3]
colnames(data) <- c(paste0("w", 1:(ncol(data)-1)), "label")

    task <- makeClassifTask(data = data, target = "label")
    lrn <- makeLearner("classif.ranger", mtry=100,respect.unordered.factors = FALSE,
                       predict.type = "response")
    n <- nrow(data)
    set.seed(123)
    train.set <- sample(n, size = 2/3*n)
    test.set <- setdiff(1:n, train.set)
    model <- mlr::train(lrn, task, subset = train.set)
    pred <- predict(model, task = task, subset = test.set)
    performance(pred, measures = list(mmce,acc))

Any help on how I can improve the accuracy would be greatly appreciated. Thanks a lot. Please let me know if you need any more details.


texts:
I am trying to develop a mispronunciation detection model for English speech. I use TIMIT dataset, this is phoneme labeled audio dataset.
A phoneme is any of the perceptually distinct units of sound. So, my dataset looks like an audio file and string of phonemes corresponding to that audio. Ex:
SX141.wav -> p l eh zh tcl t ax-h pcl p axr tcl t ih s pcl p ey dx ih n ax v aa dx ix z ix kcl k w aa dx ix kcl k ah m pcl p tcl t ih sh ix n

So, the problem is overfitting. My model is very good at training, but poor on testing. So because of this, I want to try synthetically increase my dataset. Maybe change the speed of audio or add some background noises etc.
Are there any already-ready solutions for doubling the audio dataset? Or, how to change speed and add some noises on the audio file? Will be it helpful?


texts: Deep learning test loss curve won't go down

texts:
I've been working with Deep Learning projects for this current project that I am working on and it's basically a time series classification problem. Where given an array of time series data I need to classify the customers as being either honest or dishonest.
The current model that I have now only uses CNNs, but I'm planning on expanding it with LSTMs or other models in the future. Here is the code for my model.
model = Sequential([
    Input(batch_input_shape = (None, 1036, 1)),

    Conv1D(
        filters=32,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    Conv1D(
        filters=16,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    MaxPooling1D(),

    Conv1D(
        filters=8,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    Conv1D(
        filters=8,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    MaxPooling1D(),
    Flatten(),

    Dense(10, activation='relu'),
    Dropout(0.25),
    Dense(1, activation='sigmoid'),
])

model.compile(
    # optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=[f1_m,precision_m, recall_m, matthews_correlation, 'accuracy', fpr_m]
)

After training the model for 100 EPOCHs the loss curve is this

I've tried a number of things which are:

Reducing the network size. This still results in the same issue but at a different point in the network
Reducing the learning rate of the optimizer. Same with the point above. This seems to work but it is in fact changing when this happens.

I am open to any input or recommendations I could get on how to make the testing curve follow the training curve more. I should note that my dataset is imbalanced and I'm ONLY balancing the x_train and y_train using SMOTE and not making and balancing to the testing and validation datasets to keep the data as clean as possible.


texts: Looking for the proper algorithm to compress many lowres images of nearby locations

texts:
I have an optimization problem that I'm looking for the right algorithm to solve.
What I have: A large set of low-res 360 images that were taken on a regular grid within a certain area. each of these images is quite sparsely sampled and each of these 360 images has an accurate XYZ position assigned of its center. There are millions of these small images, clusters of close-by images obviously share a lot of information while images farther apart can be completely different.
What I want to do is to compress these small 360 images.
If two 360 images are close by each other, they can be 'warped' into each other by projecting it onto a sphere of finite distance and then moving that sphere (so a closeby 360 image can be a good aproximation of another 360 image when it has been warped that way).
Based on this idea, I want to compress these small low-res 360 images by replacing each of them with:

N (N being something like 2-5) indices into an archive of M (M being something like 50-500) different 'prototype' images (of possibly higher resolution than the low res 360 images), each of which has an XZY location assigned plus a radius
N blend weights

Such that if I want to reconstruct one of the small, sparsely sampled 360 images I take the N indices stored for this image, look-up the corresponding prototype images from the archive, warp them based on the radius of the archive image and the delta vector of archive XZY and compressed image XYZ location, and then blend the N prototype images based on the N blend weights (and possibly scale down in the prototype images are higher res)
I guess this goes into the direction of Eigen Faces, but with Eigen faces each compressed face has a weight stored for each eigen-face, whereas I want that each compressed sphere only has N non-zero weights.
So my input is:
a lot of small 360 images plus a XYZ location each
my output should be:

an archive of M "prototype" images, each assigned an XYZ location and a projection radius
all compressed spheres, with each sphere compressed to N indices and N weights

This seems to be some non-linear least squares problem, but I wonder if someone can point me into the right direction on how to solve this?
As a completely alternative approach I also looked into spherical harmonics, but with those I only get enough high-frequency details at l=6 which takes 36 coefficients which is too much and also too slow to decompress.


texts:
I just built a random forest classifier and wondered about the results.

I have 4 classes: A, B, C and control. When I compare A vs control, B vs control, C vs control I get a lower average AUC score (0.8) compared to a multiclass approach with all 4 classes (0.85).

Would one expect this result and when yes why?

I would appreciate any help!


texts: PCA, covariance, eigenvector matrix and rotation

texts:
I am following the Coursera NLP specialization, and in particular the lab "Another explanation about PCA" in Course 1 Week 3.
From the lab, I recovered the following code. It creates 2 random variables, rotates them to make them dependent and correlated, and then run PCA on them:
import numpy as np
from sklearn.decomposition import PCA
import math

std1 = 1     # The desired standard deviation of our first random variable
std2 = 0.333 # The desired standard deviation of our second random variable

x = np.random.normal(0, std1, 1000) # Get 1000 samples from x ~ N(0, std1)
y = np.random.normal(0, std2, 1000)  # Get 1000 samples from y ~ N(0, std2)

# PCA works better if the data is centered
x = x - np.mean(x) # Center x
y = y - np.mean(y) # Center y

#Define a pair of dependent variables with a desired amount of covariance
n = 1 # Magnitude of covariance.
angle = np.arctan(1 / n) # Convert the covariance to and angle
print('angle: ',  angle * 180 / math.pi)

# Create a rotation matrix using the given angle
rotationMatrix = np.array([[np.cos(angle), np.sin(angle)],
                 [-np.sin(angle), np.cos(angle)]])

# Create a matrix with columns x and y
xy = np.concatenate(([x], [y]), axis=0).T

# Get covariance matrix of xy
print("Covariance matrix of xy")
covmat = np.cov(xy, rowvar=False)
print(f"{np.sqrt(covmat[0,0]):.3f} = {std1}")
print(f"{np.sqrt(covmat[1,1]):.3f} = {std2}")

# Transform the data using the rotation matrix. It correlates the two variables
data = np.dot(xy, rotationMatrix)

# Get covariance matrix of data
print("Covariance matrix of data")
covmat = np.cov(data, rowvar=False)
print(f"{np.sqrt(covmat[0,0]):.3f} = {std1}")
print(f"{np.sqrt(covmat[1,1]):.3f} = {std2}")
print(f"{covmat[0,1]:.3f} = {n}")

# Apply PCA.
pcaTr = PCA(n_components=2).fit(data)

# In theory, the Eigenvector matrix must be the
# inverse of the original rotationMatrix.
print("** These two matrices should be equal **")
print("Eigenvector matrix")
print(pcaTr.components_)
print("Inverse of original rotation matrix")
print(np.linalg.inv(rotationMatrix))

I get the following output:
angle:  45.0
Covariance matrix of xy
1.031 = 1
0.325 = 0.333
Covariance matrix of data
0.764 = 1
0.765 = 0.333
0.479 = 1
** This two matrices should be equal **
Eigenvector matrix
[[ 0.70632393  0.70788877]
 [ 0.70788877 -0.70632393]]
Inverse of original rotation matrix
[[ 0.70710678  0.70710678]
 [-0.70710678  0.70710678]]

Why does n=1 define the magnitude of the covariance?
Why we can obtain the angle between the variables as angle = np.arctan(1 / n)?
Why I don't achieve this covariance between the variable when I take element (0,1) of the covariance matrix (the second time that I run np.cov, so the line covmat = np.cov(data, rowvar=False)?
Why does the rotation change the variables variance from the initial std1 and std2?
Why "In theory, the Eigenvector matrix must be the inverse of the original rotationMatrix" ?
Why this is not the case?


texts: How to apply ensemble clustering method?

texts: I need to use ensemble clustering method by using python in my data set. I already applied k-means clustering by using scikit learn library. I also applied different classification method also find ensemble classification method in scikit-learn. Now I am confused is there any library exist in scikit learn for ensemble clustering or how I can apply ensemble clustering method on my data set?


texts: What is the best machine learning method for determine shock in time series?

texts:
I'm familiar with the Auto Vector Autorgressive (VAR) models. As you know, these models are widely used in determining shocks in time series.
Also, machine learning methods have become very popular these days.
My question is: Is there a way in machine learning methods to detecting shocks? Can something like a VAR be implemented in machine learning?


texts: tensorflow simple regression nan after >29 observations

texts:
I have code as below. If the number of data points changed to any number above 30 (example 40) then i get nan for values of final_slope , final_intercept
why?

For 25 examples it runs fine. I am using a cpu version on tensorflow on my windows machine.

The number of datapoints can be changed by changing number on the line 4th line
n= 40

import numpy as np
import tensorflow as tf

n= 40
x_data = np.linspace(0,10,n) + np.random.uniform(-1.5,1.5,n)
y_label = np.linspace(0,10,n) + np.random.uniform(-1.5,1.5,n)

import matplotlib.pyplot as plt
#%matplotlib inline
plt.plot(x_data,y_label,'*')

m = tf.Variable(0.39)
b = tf.Variable(0.2)

error = 0

for x,y in zip(x_data,y_label):

    y_hat = m*x + b  #Our predicted value

    error += (y-y_hat)**2 # The cost we want to minimize (we'll need to use an optimization function for the minimization!)

optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
train = optimizer.minimize(error)

init = tf.global_variables_initializer()

with tf.Session() as sess:

    sess.run(init)

    epochs = 1000

    for i in range(epochs):

        sess.run(train)

    # Fetch Back Results
    final_slope , final_intercept = sess.run([m,b])

print (final_slope , final_intercept)


texts:
I originally had a wide CSV dataset of about 18000 columns(and about 80 rows) that I am trying to read in R. It was stored in an Excel sheet,which unfortunately has a limit of only 16384 columns. Hence, taking the dimension I obtain:

> dim(train_set)
[1]    83 16384

i.e 1000+ columns are getting eaten up ,and this would badly affect the accuracy of the predictions. How can I read all the columns in R?

Your suggestions are much appreciated. Thanks a lot!


texts:
After splitting into test and train the glm function is used on train set. For example

m1 = glm(target ~ ., data = train, family="binomial")

Then

P = predict(m1,newdata=test, type="response")

Here I am getting warning message :

  newdata has 60000 rows but variables found have 100000 rows

And the next step:

table(test$target, P>0.5)

Here I am getting error:

  all arguments must have same length

What is the solution for this?


texts: Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

texts:
I read about NCE (a form of candidate sampling) from these two sources:

Tensorflow writeup

Original Paper

Can someone help me with the following:

A simple explanation of how NCE works (I found the above difficult to parse and get an understanding of, so something intuitive that leads to the math presented there would be great)
After point 1 above, a naturally intuitive description of how this is different from Negative Sampling. I can see that there's a slight change in the formula but could not understand the math. I do have an intuitive understanding of negative sampling in the context of word2vec - we randomly choose some samples from the vocabulary V and update only those because |V| is large and this offers a speedup. Please correct if wrong.
When to use which one and how is that decided? It would be great if you could include examples(possibly easy to understand applications)
Is NCE better than Negative Sampling? Better in what manner?

Thank you.


texts: How much of data wrangling is a data scientist's job?

texts:
I'm currently working as a data scientist at a large company (my first job as a DS, so this question may be a result of my lack of experience). They have a huge backlog of really important data science projects that would have a great positive impact if implemented. But.

Data pipelines are non-existent within the company, the standard procedure is for them to hand me gigabytes of TXT files whenever I need some information. Think of these files as tabular logs of transactions stored in arcane notation and structure. No whole piece of information is contained in one single data source, and they can't grant me access to their ERP database for "security reasons".

Initial data analysis for the simplest project requires brutal, excruciating data wrangling. More than 80% of a project's time spent is me trying to parse these files and cross data sources in order to build viable datasets. This is not a problem of simply handling missing data or preprocessing it, it's about the work it takes to build data that can be handled in the first place (solvable by dba or data engineering, not data science?).

1) Feels like most of the work is not related to data science at all. Is this accurate?

2) I know this is not a data-driven company with a high-level data engineering department, but it is my opinion that in order to build for a sustainable future of data science projects, minimum levels of data accessibility are required. Am I wrong?

3) Is this type of setup common for a company with serious data science needs?


texts: Forecasting sales of next year using sales of past years?

texts:
I have a dataset of sales of a company for different products throughout 3 years. I have to forecast the sales for each of these products for next year. A sample of the dataset is:

Here I have to forecast sales for year 4 from the sales record of each product over 3 years.

N.A is present where the particular product was not sold in that year.

Questions

1.Which algorithm should I use to forecast the sales for year 4? I have heard about ARIMA and xgboost being used for time series data. Can you please help me with that?

2.Now I am being given a new product which was not sold before. I am being asked to predict its sales for year 4, using the data for sales of other products:

What should be my approach for this problem?

Please help me out here. Thank you for your time and support in advance. I am sorry if my documentation or question is poor.


texts: Feature Engineering and prediction with R and python

texts:
I have a sequential dataset, and have 2000 rows for 300 ID. I have 20 variables (i.e in my real dataset)
ID<-c(1,1,1,2,2,2,2,3,3,4,4,4)
time<-c(0,12,36,0,7,23,68,0,23,0,32,45)
Age<-rnorm(12,45,9)
Sexe<-c("F","F","F","M","M","M","M","M","M","F","F","F")
biology1<-rnorm(12,12,3)
biology2<-rnorm (12,100,20)
biology3<-rnorm(12,45,9)
biology4<-rnorm(12,20,2)
Death<-c(0,0,1,0,0,0,0,0,0,0,0,1)
data<-data.frame(ID,time,Age,Sexe,biology1,biology2,biology3,biology4,Death)

I should predict whether the patient dies or not. I have several biological observations over several times. I want to create more variables using feature engineering to improve the prediction.
I am a beginner in machine learning. It would be useful if I could get some ideas:
What kind of transformation I could do? averages? slopes? others? How could I do them? How many variables should I create? Should I do manual or automatic transformations? What packages shoul I use? finaly should I create only one row per ID to predict? Can someone give me links? book titles to get me started? I need something focused on time-series data. I would like to work with both R and Python and compare the results. So packages related to both software will be useful for me.


texts: k-means in pandas dataframe, optimal k (number of clusters)?

texts:
I have a dataframe like this.
At first, i would like to calculate the optimal number of clusters and then I would like to use k-means without the last column (id).
After that i would like to know the cluster of every row (id).
I did this:

from sklearn.cluster import KMeans
features = list(df.columns)[:-2]
data = df[features]
kmeans = KMeans(n_clusters=4)

y = kmeans.fit_predict(data)

df['Cluster'] = y

i believe it is right but i don't know how to know the optimal number of k before all these. How can i find it? Also, is what I did right? Because i then dropped the id column and I run:

kmeans = KMeans(n_clusters=4)

y = kmeans.fit_predict(df)

df['Cluster'] = y
df[['Cluster']]

and the results were a liitle different.
But my main question is, how to find optimal number of clusters


texts: Spacy custom NER-Model - losses too high?


texts:
I have a question related to K-Means clustering and PCA. In my project, I have two target classes - 0 and 1- and I am trying to group the records that were predicted as 0 into 5 clusters.
I am using PCA strictly as a visualization technique since my data frame has 8 dimensions and I need to bring it down to 2-3 dimensions to see the clusters. My question is about the procedure I should follow~
First Way:

Extract all records with target = 0
Do PCA and KMeans on just those records

Second way:

Do PCA on all records (target = 0 and 1)
Extract PCA records with target = 0 (from the PCA data frame created in step 1)
Do KMeans on those records

The PCA1, PCA2, PCA3 values for the records(with target = 0) are different using these two ways. And since the PCA values are different, the cluster visualizations are also showing up differently. Which option should I follow?
Thank you so much!


texts: Reproducing cutoff in xgboost.train() with XGBClassifier()

texts:
(Cross-posted from https://stackoverflow.com/questions/43415724/reproducing-cutoff-in-xgboost-train-with-xgbclassifier)

I've gotten xgboost to generate good predictions by using xgboost.train().

X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=.6)
xgtrain = xgb.DMatrix(X_train, y_train)

param = {'max_depth':7, 'silent':1}
bst = xgb.train(param, xgtrain, num_boost_round=2)
y_pred = bst.predict(xgtest)
y_pred = [1. if y_cont > .28  else 0. for y_cont in y_pred]
y_true = y_test

This approach wasn't producing good results (I am trying to maximize the f1 score) until I realized that the f1 score increases significantly when setting a threshold on the outputs. This threshold turned out to be .28. Here are some of the predictions before I set the cutoff and convert to 0s and 1s:

[ 0.25447303  0.25383738  0.24621713 ...,  0.24621713  0.24621713 0.24621713]

But now I want to tune my parameters (using GridSearchCV()), which means that I'll need to reproduce what I did in xgboost.train() above with XGBClassifier().

I realize that things might get tricky because the (default) objective function in xgboost.train() is none, and for XGBClassifier() it's 'binary:logistic'. XGBClassifier() returns the class and not the probability, which is useful in most cases but not here. I've tried predict_proba() with XGBClassifier() and then setting a cutoff, but it seemed pretty useless since the probabilities I were getting were super close to 0 and 1:

[[  9.99445975e-01   5.54045662e-04]
 [  9.89062011e-01   1.09380139e-02]
 [  9.95234787e-01   4.76523908e-03]

How can I finish the code below do the equivalent of xgboost.train() but with XGBClassifier? When I've tried XGBClassifier without a cutoff, I would get a horrible f1 score.

X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=.6)
rf = XGBClassifier(max_depth=7, learning_rate=0.1, n_estimators=100, silent=True, objective='binary:logistic', nthread=-1, gamma=0, min_child_weight=1, max_delta_step=0, subsample=1, colsample_bytree=1, colsample_bylevel=1, reg_alpha=0, reg_lambda=1, scale_pos_weight=1, base_score=0.5, seed=0, missing=None)
rf = rf.fit(X_train, y_train)


texts:
Conducting a linear regression model using a loss function, why should I use $L_1$ instead of $L_2$ regularization?

Is it better at preventing overfitting? Is it deterministic (so always a unique solution)? Is it better at feature selection (because producing sparse models)? Does it disperse the weights among the features?


texts: Where can I find Business Intelligence Case Studies?

texts: I am teaching an undergraduate course on business intelligence and data mining. I was wondering if anyone has any resources where students can search for BI case studies? I've tried google myself but I can't find anything that's too promising.


texts: OneHotEncoder vs. LabelEncoder vs. LabelBinarizer

texts:
I have a dataset that I have created in Python with a category called change_label that has values "buy, hold, sell" (or alternatively I could use -1,0,1).

I am wondering, what is the best way to fit this data so that machine learning algorithms can be applied to it to make predictions?

My options are label encoding, label binzarization, or one-hot encoding. Basically, I thought that simply using sci-kit learn's LabelBinarizer() would be enough. However, I want to be sure. Any advice?


texts: Samples in a PCA form lines, what are the reasons?

texts:
I have come across some PCA plots in which the samples are forming "lines", such as this one:

Or this one:

What kind of data can generate such PCA plots? Are there anything wrong with these PCA?


texts: Data Science Take Home Challenge Interpretation of Question

texts:
I am currently applying for a Data Science position and have to finish a take-home challenge for one of the companies. However, I don't really understand what they want me to do and hope you can help me interpret the question. Unfortunately, I can't reach anyone at the company due to Christmas holidays

I have a data set with transactions of customers and another dataset with information about age, country... of those customers. Now I am asked to define a target metric for engagement and say how I would define unengaged vs. engaged customer. Afterwards, I have to use this logic to build a model to classify engaged vs unengaged customers (they point out overfitting due to feature used in the target metric).

How would you interpret these questions?

Do I first have to say if target_metric > x: engaged and then build a supervised model to classify the users again?

OR

Would you define several metrics that could separate engaged and unengaged customers and then use an unsupervised model to find two clusters that would then be engaged vs unengaged?

I tried to keep the question as general as possible to avoid "cheating". I only want to know how you would interpret them. Thanks!


texts: What does images per second mean when benchmarking Deep Learning GPU?

texts:
I've been reviewing performance of several NVIDIA GPU's and I see that typically results are presented in terms of "images per second" that can be processed. Experiments are typically being performed on classical network architectures such as Alex Net or GoogLeNet.

I'm wondering if a given number of images per second, say 15000, means that 15000 images can be processed by iteration or for fully learning the network with that amount of images?. I suppose that if I have 15000 images and want to calculate how fast will a given GPU train that network, I would have to multiply by certain values of my specific configuration (for example number of iterations). In case this is not true, is there a default configuration being used for this tests?

Here an example of benchmark Deep Learning Inference on P40 GPUs (mirror)


texts: In handwritten digit recognition problem using logistic regression, what changes needed to add another class "Not a Digit"

texts: In handwritten digit recognition problem using logistic regression, normal implementation would forcibly classify even a picture of dog or cat as a digit. To eliminate this, what changes are needed to add another class i.e. "Not a Digit" to already existing 10 classes (0 to 9) ?


texts: Name of ML discipline for trainable clusterer - supervised clustering, or group ID assignment

texts:
Is there a name for a machine learning algorithm, that learns 'clustering approach' from examples of clustered datapoints (with different numbers of clusters each time)?

Imagine the training set as an ensemble of sets with their clusters assigned.

Has anyone please encountered a similar problem in any literature? My goal is to read something more about this problem.

Example #1:
Underlying rule: "cluster words to the same group if they do exist on the same page of an imagined storybook"
Training dataset:
1) Source values: [banana,car,rock,orange] -> Target groups: [banana and orange],[car],[rock]
2) Source values: [kiwi,audi,folswagen] -> [kiwi,audi,folswagen]
...) .. and so on...

... And that we do not know the original story, but only instances of (datapoints -> their groupping). And also we do not know the rule :)

Example #2:
Underlying rule: "be evil - cluster like kmeans, but if there seems to be a cloud of points looking like a smiley, do not cluster at all!"
Training dataset:
1) Source values: [bunch of points in a shape of two circles] -> [each circle is a cluster]
2) Source values: [bunch of points and a cloud of points in a shape of smiley] -> []
...) and so on ...

Yes it can be solved by detecting if there is the smiley, but we do not know the rule, we want the algorithm to learn it from the data.

Edit:
maybe the word clustering is not a wise choice from me. The quality of output will not be evaluated by user, but on a big validation set to see if the algorithm did learn from samples it has seen. There also supposedly does exist one best solution, from the nature of our problem (the underlying rule is always present, although not known).

Original, unclear, description before editing:
The problem is as follows:
Each row of data is a shuffled list/set of datapoints (source variable) and their assignment to groups (target variable).
It would seem as a classification task, BUT the groups make sense only to each specific datarow (and their number also varies).
It would seems as a clustering task, BUT the underlying principle needs to be applied again and again to each different row.


texts: What are the practical applications of inverse reinforcement learning?

texts: Inverse reinforcement learning is about using expert trajectories to learn a reward function. Now the most successful method is Maximum Entropy Inverse Reinforcement Learning. But in that, you need a model-based reinforcement learning. But most of the practical world problems are model-free which is hard to calculate the state transition probabilities. So how can we use this inverse reinforcement learning in real-world problems?


texts: Is text recognition by definition a part of image recognition?

texts:
I'm referring to more advanced text recognition systems that are using neural networks to find and extract text from images like the ones Google and Microsoft are offering on their ML platforms.

If not, what is the technical difference to other branches of image recognition like object or face recognition?


texts: How to get probabilities values with keras?

texts:
tensorflow version = '1.12.0'

keras version = '2.1.6-tf'

I'm using keras with tensorflow backend.

I want to get the probabilities values of the prediction.
I want the probabilities to sum up to 1.
I tried using 'softmax' and 'categorical_crossentropy' but nothing works.

This is my model:

X = pickle.load(open("X.pickle", "rb"))
y = pickle.load(open("y.pickle", "rb"))

number_of_gestures = 5
y = to_categorical(y, num_classes=number_of_gestures) #to_categorical is a function from keras - np_utils.

model = Sequential()
model.add(Conv2D(16, (2,2), input_shape=(IMG_SIZE, IMG_SIZE, 1), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))

model.add(Conv2D(32, (5,5), activation='relu'))

model.add(MaxPooling2D(pool_size=(5, 5), strides=(5, 5), padding='same'))

model.add(Conv2D(64, (5,5), activation='relu'))

model.add(Flatten())

model.add(Dense(128, activation='relu'))

model.add(Dropout(0.2))

model.add(Dense(number_of_gestures, activation='softmax'))

sgd = optimizers.SGD(lr=1e-2)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
model.fit(X, y, batch_size=500, epochs=40, validation_split=0.1)

The probabilities looks like that:

  [1. 0. 0. 0. 0.]

And I want it to look like this:

  [0.897. 0.023. 0.158. 0.780. 0.1021]

I know it does not sum up to 1 but this is just an example.


texts: Problem with data augmentation only on train set!

texts:
So I want to augment my data but unfortunately, I don't have an individual folder for train and validation set,  and of course, I want to only augment the train set but if I use this:

data = keras.preprocessing.image.ImageDataGenerator(rescale=(1/255.),horizontal_flip=True,vertical_flip=True,validation_split=0.2)

this way I can split the sets but the augmentation would also apply to validation data too, so what should I do?

I should also mention that I'm reading the data from the directory using flow_from_directory


texts:
I was going through the Andrew Ng's notes for Decision Trees. It has one section explaining the usage of categorical variables using Decision Trees in which I am not able to understand this part

" A caveat to the above is that we must take care to not allow a variable
to have too many categories. For a set of categories S, our set of possible
questions is the power set P(S), of cardinality 2^|S| . Thus, a large number of categories makes question selection computationally intractable. "

Q1 -> How is the possible set of questions is $2^{|S|}$ ? It should be equal to the non-leaf nodes in the decision tree .

Link to Andrew Ng notes on Decision Tree : http://cs229.stanford.edu/notes/cs229-notes-dt.pdf

Can you please give some clarification on it .


texts: How to measure variable contribution to an observation in a non-linear model?

texts:
Based on my model, if I decline someone due to their score, it should be able to provide some reasoning as to which variables mainly contributed to the decision to decline.

Typically in Logistic Regression models, this is a simple exercise where you calculate (Beta * X) for each variable and pick 1 or 2 variables which caused the biggest score drop.

However, this isn't very straightforward for non-linear models. I would appreciate any ideas on handling something like this. Thanks.


texts: How to treat undefined input space in supervised learning algorithms?

texts:
In a supervised learning problem (like a neural network) with binary output, we supply input data for which the output values are defined 0 and 1 (indicating probabilities of occurrence of an event).  But the input data not necessarily cover the entire input space. How does the algorithm treat the rest of the input space?

For example, for an annular dataset in the figure, there are two sets of data in the 2-D input space annotated as 0s and 1s. But there are a lot of input space which is not defined (neither 0 nor 1). What output can be expected at the prediction stage for an input that does not lie around the input data clouds?


texts: How to replace levels with certain value?

texts:
I have a categorical variable in my dataset. I want to replace the levels which are present in test dataset and not present in training set with a value called "others"

Here is how it looks:

levels(training$var1) has levels as "1"  "2"  "3"  "Others"

levels(testing$var1) has levels as "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10"

I want to replace all the levels in testing data which are not training data. To achieve this I take the difference between the levels first.

a <- setdiff(levels(levels(testing$var1),training$var1)).

and I get output as

levels(a) as "4"  "5"  "6"  "7"  "8"  "9"  "10"

Now I need to replace all the above difference values with "Others". Kindly note that I do not want to drop out these levels but I want to replace with "Others".

For this I tried

testing$var1[testing$var1 == "4" <- 'Others"
testing$var1[testing$var1 == "5" <- 'Others" ==> these works.

However I want to make it in run time, something like this:

testing$var1[testing$var1 == a[1,] <- 'Others"

But this is not working.


texts: Dummy variable only for character value in a column (Neglecting float and integers)

texts:
My dataset consists of 3000 rows and 50 columns, out of which one column (ESTIMATE_FAMILY_CONTRIBUTION)  contains all numerical value(around 2000 different values like 20,30,32....) but got one value as String e.g. 'No_information'.
When I create dummies(One-Hot Encoded) for the Feature (pd.get_dummies()), around 2000 new columns are created for the original column i.e. ESTIMATE_FAMILY_CONTRIBUTION.
What I want is to have only 2 columns to be created, one for 'No_information' and other with all the numerical values. How do I do it?