texts: Fitting and transforming text data in training, testing, and validation sets

texts:
I'm trying to implement a simple text classifier wherein the data is split into training (70%) and testing (30%) sets, but cross validation (k=10) to be performed on the training set.

My main concern here is the basis used for transforming a given set. I've seen tutorials where the whole data set was used to fit a Count/TFIDF vectorizer, but wouldn't this introduce bias when transforming the validation and testing sets since the previously mentioned sets were included in the whole data set? Or is the bias so small that it is acceptable to do so?

Within a fold, would it be better if the training set was used to fit a vectorizer and transform the validation set? And for testing, should the training + validation sets be used to fit a vectorizer to transform the testing set?

And on this note, should the validation set also be treated as "unseen" data similar to the test set?

A lot of the tutorials and notes online show ready made datasets, but there are certain cases with text data where values are dependent on a given set of documents (e.g. like how IDF is computed based on some group of documents) and that there are a lot more nitty gritty details when extracting features.

I think I'm just confused and am seeking a little clarification regarding this manner. Thanks!


texts: Is it possible to perform Semantic Textual Similarity without using NLTK and Genism?

texts: College restricted us to make projects in object oriented programming languages but without using any other libraries except standard ones. We can not use APIs also


texts:
While predicting, what happens if we pass the newdata along with the target variable? Do we need to isolate the target variable before feeding into predict function?

data = read.csv("diabetes.csv")  #768 observations of 9 variables

#I split the data into train and test
set.seed(123)
IDS = 1:nrow(data)
train_data = sample(x = IDS,size = nrow(data)*0.7)
test_data = sample(x = setdiff(IDS,train_data),size = nrow(data)*0.3)
train_data = data[train_data,]
test_data = data[test_data,]

#Build logistic regression model
log_model1 = glm(formula = Outcome ~ .,data = train_data,family = 'binomial')

#Predict it on test or train
train_predict = predict(object = log_model1,newdata = train_data,type = "response")


texts: Accuracy value constant even after different runs

texts:
I am using the neural network toolbox of Matlab to train a network. Now my code is as follows:

x = xdata.';
t = target1';
% Create a Pattern Recognition Network
hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
net.layers{2}.transferFcn = 'softmax';
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;
net.trainFcn = 'trainscg';  % Scaled conjugate gradient
net.performFcn = 'mse';
net.performParam.regularization = 0.5;
%net.performParam.normalization = 0.01;
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
  'plotregression', 'plotfit', 'plotconfusion'};

% Train the Network
[net,tr] = train(net,x,t);
% Test the Network
y = net(x);
e = gsubtract(t,y);
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
performance = perform(net,t,y)
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t  .* tr.valMask{1};

Now I am supposed to get different accuracies with different run of data since the sampling (division of dataset into train test and validation set) is random. But I am getting a constant accuracy (89.7%). The variable 'xdata' contains only those features selected by a feature selection algorithms. Is there any reason why my accuracy value is constant?
I have trained an SVM too with the same dataset. There too I am getting the a stale accuracy even with multiple run (94%)
The output y contains 2 values. What do those values signify?


texts: Suppose I have a dataset containing two very similar classes of data. By similar, I mean that the 'distance' between these two classes is very small. For example, one instance in Class 1 is the sum of one instance in Class 2 and some noise. If the SNR is high, we can say these two instances are similar. Because of the similarity, the dataset is inseparable. I am wondering if there exists any effective clustering algorithm that can work. Thank you very much！


texts: Catching hidden state activations of an RNN on prediction.

texts:
I'm interested in doing research similar to the work done in this blog, The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy. Specifically, where the author explores the rules that individual neurons have learned by extracting the activations or excitement of random neurons at the prediction step and visualizing them as a heatmap on top of the predicted text.

I'm having a hard time figuring out how to do this with existing platforms such as TensorFlow. My first strategy was to try a form of transfer learning and cut off a copy of the network at an activation layer and have that layer's activations output side by side with a text prediction. But the weakness of this approach is that I can't actually see the activations in the hidden state (or more than one layer at a time), just the outputs of an LSTM layer.

Can anyone recommend a strategy or starting point where I can reveal the activations of neurons in the hidden states of RNNs (LSTM or GRU), or if there's a way to do this in TensorFlow that I've overlooked? If I build the network from scratch with TensorFlow, is there a way to efficiently set it up to output this data on prediction without significantly adding to the computation expense during training? I couldn't find the answer in the source code.

Can anyone give me a nudge in the right direction?

Thanks in advance!


texts:
I am quite new in the ML field. I think I correctly understood the information leaking problem during the testing/validation phases but I am struggling to understand some François Chollet statements that I will now report. This discussion will be based on Keras only for practical reasons.

First of all: Keras initially supported various metrics now removed, like for example precision, recall, f1, fbeta. As stated here they were removed only for a clarity reason: they were calculated as local metrics and so they were (almost) meaningless.
Those metrics clearly used the labels/classes to be computed. However in that post nobody complained about this problem
Now comes the hard part: Keras allows to specify both class and sample weights during the training (with the class_weight and sample_weight arguments). These both influence the training loss and the metrics listed in weighted_metrics.
However, class_weight does not influence the metrics listed in weighted_metrics during the validation nor can be used in the evaluate method, where it is only possibile to specify the sample_weight argument but not the class_weight one.

François Chollet clearly explained this design choice (quote):

  You'll notice that evaluate has a sample_weight argument but no class_weight argument. Validation evaluation in fit follows the same pattern. The reason is that class weighting is meant to help training (by modulating contributions of different classes), but not to affect evaluation. Evaluation, because it reflects production constraints, implies that labels are not available when making predictions, and thus can't be used to modulate evaluation metrics. (link)

  Additionally I have noticed that in the case where validation_split is used, class_weight is applied to the validation data, which is not the expected behavior (classes are of course supposed unknown for the validation data). I am also fixing this. (link)

  I mean that weighting by classes in a test setup is a form of label information leak, because it cannot be reproduced in a real test situation (when you don't have the labels, such as when using model.predict()). It's acceptable during training, by definition, but should not be done at testing time. (link)

I cannot understand why this is the case during the validation phase:

A class leaking like this is not such a serious problem: there is no backprop during the validation phase and so the potential leaking is quite limited. In addition I think this is not more serious than the standard leaking due to the continuous model tuning on the same validation set
Without using the classes, during the validation we could not compute interesting quantities such as the precision, the recall, the confusion matrix and so on. Computing them on the training set would produce a biased estimate.
Using the classes info would be thus extremely useful in order to obtain good estimates.
In addition, as stated before, those metrics were available and nobody complained about them.

I know we could obtain a similar effect by playing with the sample_weight argument but I am more interested in understanding this problem.
I know that the classes will not be available (theoretically) during the testing phase but I think that, during the development and tuning, these information and statistics could help better understanding the errors done by the model.


texts: Orange 3 Heatmap clustering under the hood

texts:
I have recently used the heatmap widget in Orange 3. All the documentation says is "Clustering (clusters data by similarity)". Is this using hierarchical or k-means or some other type of clustering?

On that note, is there a way to look at the code being run by all the widgets to see whats going on under the hood? It would be nice if after you finish the workflow you would get a file with the script run to perform the analysis.


texts: Visualize/analyze data before or after imputing missing values?

texts:
My understanding is that we impute missing values in order to preserve those training examples so our Machine Learning algorithms have as many training examples as possible.

To me, it would make intuitive sense to visualize/analyze the data before imputing the missing values, as imputation will skew distributions and may lead to false assumptions about the real data before imputation. I can see a case for doing both before and after, but that adds time to the feature analysis. On the other hand, I can see a case for not imputing values if the % of missing values is high enough to affect the data's distribution.

In Python I imagine using something along the lines of pd.Series.dropna() to isolate the existing values.

TL;DR: Should one impute missing values before or after visualizing the data and pulling insights from it?


texts: Orange Linear Regression and scikit-learn linear regression gives different results

texts:
I am trying to cross-validate my scikit-learn script by mean of Orange3, thus obtaining a nice visual representation.

Doing step by step, to keep it simple I just tried to cross-validate a simple linear regression.

Following is the simple script for linear regression in python/scikit-learn

lr=LinearRegression(fit_intercept=True,normalize=False,copy_X=True,n_jobs=1)
lm=lr.fit(X_train, y_train)
lmscore_train=lm.score(X_train,y_train)  ## R2 = 0.6264021467338086
lmscore_test=lm.score(X_test,y_test)     ## R2 = -12.344747578839215

Whilst I was expecting to get the same result, this is not the case.

In Orange, I get R2=-10.792 whilst in Python, I get R2=-12.344747578839215.

Train and test split are the same in both cases.

Do you have a clue why that is?


texts:
I want to predict using a multi layer perceptron. My data have 3 attributes x,y,z. My window size (input) is 3. That is, using 3 sets of values for x,y,z to predict next set of x,y,z. What should be my number of input neurons?

Should it be 6 input neurons for 3 sets of values of x,y,z?
or should it be 3 input neurons and each neuron has 3 inputs for x,y,z?


texts: PCA Reduction resulted in an elliptical form

texts:
I have a dataset with 19 features (columns). I normalized them using sklearn.preprocessing.normalize then I used PCA to reduce them to 2 components for plotting purposes following the steps in this link:
http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_digits.html

What I saw is that my data points form a perfect ellipse shape as in the following picture:

I want to know if this has any special meaning for my data.


texts:
In a scientific paper (Neural Networks: A Systematic Introduction, page 86) about the Perceptron Algorithm I found:

  A good initial heuristic is to start with the average of the positive input vectors minus the average of the negative input vectors. In many cases this yields an initial vector near the solution region.

Can you show me geometrically why this is true?


texts: Why is my loss blowing up after adding regularization

texts:
I tried to add L2 regularization to a network class I wrote however when I train it the loss blows up even though accuracy also increases. Can someone explain where I am going wrong? (I am using the formulas from here)
The update to minibatch (The (1-eta*(lmbda/n)) coefficient to w is what I added)
def update_mini_batch(self, mini_batch, eta, lmbda, n):
    # n is the number of training samples being trained from
    # Turn the mini_batch with one dimensional samples into two matricies and then transpose them to get samples in columns
    matrix_x, matrix_y = [np.array([arr for arr in arr_list]).transpose() for arr_list in zip(*mini_batch)]

    gradient_b, gradient_w = self.backprop(matrix_x, matrix_y)

    self.bias = [b-(eta/len(mini_batch))*db for b, db in zip(self.bias, gradient_b)]
    self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*dw for w, dw in zip(self.weights, gradient_w)]

The function that evaluates cost (I am using Quadratic Cost)
def evaluate(self, data, lmbda):
    matrix_x, matrix_y = [np.array([arr for arr in arr_list]).transpose() for arr_list in zip(*data)]
    output_matrix = self.feedforward(matrix_x)

    cost = self.cost_func.apply(output_matrix, matrix_y)
    #L2 Regularization
    cost += lmbda/(2*(matrix_y.shape[1])) * sum(np.linalg.norm(w)**2 for w in self.weights)
    acc = np.sum(output_matrix.argmax(axis=0)==matrix_y.argmax(axis=0))

    return cost, acc

An example of my cost and accuracy during training
Epoch 0 done! Cost: 11.938649143175008. Accuracy 7397 / 10000
Epoch 1 done! Cost: 16.017232330762045. Accuracy 7381 / 10000
Epoch 2 done! Cost: 21.62351585060393. Accuracy 7431 / 10000
Epoch 3 done! Cost: 30.96422767377938. Accuracy 7498 / 10000
Epoch 4 done! Cost: 45.75409202821266. Accuracy 7669 / 10000
Epoch 5 done! Cost: 67.47752609972852. Accuracy 7691 / 10000
Epoch 6 done! Cost: 97.56030814767621. Accuracy 7574 / 10000
Epoch 7 done! Cost: 133.3273570333546. Accuracy 7503 / 10000
Epoch 8 done! Cost: 174.7085211732363. Accuracy 7341 / 10000

After running this for longer the cost still increases continually and no change in eta or lambda changes this fact. Once thing I noticed was that the individual MSE error was behaving normally and it was just the magnitude of the weights that was increasing.


texts: Bayesian combination of multi-dimensional experts?

texts:
I have what seems to me to be a slightly complex version of a decision tree problem, that can't figure out how to model, and I'm trying to avoid the "just dump it into an NN" solution.

I have a bunch of expert opinions (from either people or algorithms) where the features dimensions overlap. Both features (inputs) and outputs (decision) are categorical, but I also have a certainty in the decision (let's say 0-1).

For example, let's say that the input features are a,b,c,d,e and there is one output (decisions) from among v,w,x,y,z:

Features   Decision  certainty
 a,b,c       w          0.5
 b,c,d,e     z          0.4
 a,c,e,f     v          0.75
 ...

Without the certainty, this would obviously just be a trivial categorical decision tree problem. However, the certainty both simplifies and complexifies the problem. It simplifies it bcs if the features->decision data is inconsistent (which it is!), you'd have no output at all, so the certainty saves me from that failure mode, but on the otherhand, I'm not sure what to correctly do with the certainty.

The certainties suggest Bayesian combination, and if the data was unidimensional this would be trivial as well. So I'm kind of caught half way between a decision tree and a bayesian model. An obvious cop out is just to dump it into an NN (or even just an NB or regression), using the certainties either as outputs on the categories, or doing something dumb like replicating the I/O pairings per the (un)certainty.

Thanks in advance for any suggestions.


texts: Making predictions with missing numeric data

texts:
I don't know much about how to deal with missing data.  When the data is categorical it doesn't seem too bad, if I one-hot encode it without dropping any of the actual categories, the missing data is accounted for in the model because all the associated one-hot columns are zero. But for numeric data, I don't like the options as I understand them: omit the entire observation, or impute a value. Are there any other choices?

I'd lean toward omitting the data (in the data I am working on it would lead to ~2% data loss), but this doesn't help if you are trying to make predictions when data is missing, and I I can't find much useful discussion addressing this aspect of the problem (mostly it is around training the model).

I guess you could train multiple models that deal with all the permutations of missing features, and select the model matching the available predictive data?  But that seems likely to be too unwieldy and time consuming to be practical. (The data set I'm working on now is artificial; it has 100 features, 96 of which are numeric, and all of which [both training and prediction sets] have from 2-20 missing values.)

Fwiw, my plan if I have to impute data is to regress using the 98% of complete training/prediction data, predict mean for each missing value, then randomize it based on the variance/CI.

I guess this boils down two questions/requests:

1) As in bold above re omission/imputation: are there any other choices?

2) General advice on how to proceed, given the predictions must be based on missing data?

-- Edit/Add 15-Mar-20 --
I would particularly like to know what other options are, and what is considered best practice, when implementing models such as regression that do not cope with missing values?


texts:
Could any one help me know about different approaches, methods or algorithms to build a model only with positive responses.

Let's assume we have a set of customers with a 'positive' behaviour. We want to find customers with a similar profile in the database and that are more likely to have this positive behaviour in the future. This can't be modeled with 0/1 target variable because there's not a clear definition of a 'negative' behaviour.

It looks like a look-alike modeling. However I think that look-alike modeling assumes knowing the response (0/1) for the given set of people (treatment group) and finding similar people in the control group to predict their responses.

Thanks in advance of your response!

MK


texts: Need help understanding xgboost's approximate split points proposal

texts:
background:

in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective:

$$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$

where $g_i, h_i$ are first order and second order derivatives over our previous best estimation $\hat{y}$ (from iteration $t-1$):

$g_i=d_{\hat{y}}l(y_i, \hat{y}) $
$h_i=d^2_{\hat{y}}l(y_i, \hat{y}) $

and $l$ is our loss function.

The question (finally):

When building $f_t$ and considering a specific feature $k$ in a specific split, they use the following heuristic to assess only some split candidates: They sort all examples by their $x_k$, pass over the sorted list and sum their second derivative $h_i$. They consider a split candidate only when the sum changes more than $\epsilon$. Why is that???

The explanation they give eludes me:

They claim we can rewrite the previous equation like so:

$$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - g_i/h_i]^2 + constant$$

and I fail to follow the algebra - can you show why is it equal?

And then they claim that "this is exactly weighted squared loss with labels $gi/hi$ and weights $h_i$" - a statement I agree with, but I don't understand how does it relate to the split candidate algorithm they are using...

Thanks and sorry if this is too long for this forum.


texts: Weighting of features in Recommender Systems

texts:
I'm new to Recommender Systems, and wanted to figure some things out in order to make the best possible Content Ranking System. I want to make a ranking of all the content (and content providers) based on a hybrid score derived from both -

Content Based Filtering taking into account all the features of each content item.
Collaborative Filtering taking into account the ratings and comments (sentiment polarity) of the feedback left by users.

The end product is that I only want to rank the content based on this features and feedbacks left by users, and not recommend any content to the users.
For content based filtering, how do I assign a different weight (coefficient) to each feature depending on how important it is to the overall score. Is there a ML algorithm to assign dynamic weights to the features every time based on any new entries?


texts: should I normalise data of disparate frequencies

texts:
The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this

where each vertical line is a curve before being automatically rescaled.

I am preprocessing the data to input to a CNN model, thus I am thinking about normalising the data. But I am concerned about 2 issues:

1) The frequencies of data: most data are around 25 vs very little data between 25-45

2) The magnitude of data: although very little data is between 25-45, they are significantly greater than the rest(about twice the magnitude).

As such, given data is not normally distributed(maybe partially uniform?),I am not sure if it is valid to do normalisation. Please shed some light. Any help is appreciated.


texts: Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

texts:
I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm.

I made sure that the class counts are balanced. i.e., an equal number of occurrences per class.

The four features are Total Cholesterol, Systolic Blood Pressure, Diastolic Blood Pressure, and Cigraeette count. The class variable name is Stroke.

Dataset description is shown below:

       TOTCHOL    SYSBP    DIABP  CIGPDAY   STROKE
count  200.000  200.000  200.000  200.000  200.000
mean   231.040  144.560   81.400    4.480    1.500
std     42.465   23.754   11.931    9.359    0.501
min    112.000  100.000   51.500    0.000    1.000
25%    204.750  126.750   73.750    0.000    1.000
50%    225.500  141.000   80.000    0.000    1.500
75%    256.250  161.000   90.000    4.000    2.000
max    378.000  225.000  113.000   60.000    2.000

SKEW is

TOTCHOL    0.369
SYSBP      0.610
DIABP      0.273
CIGPDAY    2.618
STROKE     0.000

Python + sklearn is used here. The problem is that the classification performance gets very negatively-impacted when I try to normalize the dataset using

 X=preprocessing.StandardScaler().fit(X).transform(X)

or

 X=preprocessing.MinMaxScaler().fit_transform(X)

The classification report (before) normalizing the dataset:

            precision    recall  f1-score   support

      1       0.85      0.79      0.81        28
      2       0.82      0.88      0.85        32
avg / total   0.83      0.83      0.83        60

While the classification report (After) normalizing the dataset:

         precision    recall  f1-score   support

      1       0.47      1.00      0.64        28
      2       1.00      0.03      0.06        32

  avg/total   0.75      0.48      0.33        60

Please note that the class variable includes the outcomes 1(yes) and 2(no) instead of the usual 0 and 1.

at this point, I see the preprocessing have damaged the classification accuracy instead of helping it. Is there any logical explanation for that?
do I "have" to do the normalization step?

Another remark is the probability scores that I get (before) normalizing the dataset are much higher per class as shown below:

 [ 0.65929838  0.34070162]
 [ 0.40999878  0.59000122]
 [ 0.43592976  0.56407024]
 [ 0.40306785  0.59693215]
 [ 0.92748002  0.07251998]
 [ 0.74173761  0.25826239]

compared to the ones (after) normalizing the dataset

  [ 0.51636816  0.48363184]
  [ 0.5183946   0.4816054 ]
  [ 0.51410135  0.48589865]
  [ 0.50739794  0.49260206]
  [ 0.52645649  0.47354351]
  [ 0.5308564   0.4691436 ]

is there any explanation why I get such results?


texts: Cleaning a certain feature to predict salary using Machine Learning

texts:
Info:
I am working on a dataset, and i would like to create a model that would predict salary.
Columns are as follows:
Index(['ID', 'Salary', 'DOJ', 'DOL', 'Designation', 'JobCity', 'Gender', 'DOB',
       '10percentage', '10board', '12percentage', '12board', 'CollegeTier',
       'Degree', 'Specialization', 'collegeGPA', 'CollegeCityTier',
       'CollegeState', 'English', 'Logical', 'Quant', 'Domain',
       'ComputerProgramming', 'ElectronicsAndSemicon', 'ComputerScience',
       'MechanicalEngg', 'ElectricalEngg', 'TelecomEngg', 'CivilEngg',
       'conscientiousness', 'agreeableness', 'extraversion', 'nueroticism',
       'openess_to_experience'],
      dtype='object')

Issue:
The train dataset contains around 3000 entries. For the feature 'JobCity', there are around 200 unique values. Out of 3000 entries, 180 are cities whose value counts are below 10.
For example, the city 'Agra' occurs only twice and 'Ranchi' occurs only once.
I plan to encode all the cities to feed into the machine learning (polynomial regression) model.
How should the cities with value counts less than 10 be handled?


texts: Loss function in order to find the most certain samples

texts:
I have a classification problem where I try to predict if some measurement will go up or down in the next 10 seconds.
I have a few hundred of features and so far I simply trained a few classification models and measured their ability to seperate the two population.
The issue is that enentually I need to return only a few dozens of samples among the thousands of samples in the test set that I'm highly certain about them, let's say the top 15 samples that will go up and the 15 samples that will go down.
For this I'm using two thresholds and I'm returning only samples that the score of the trained model passed the upper threshold or if it was lower than the lower threshold.
My question is if there a more suitable loss function to train the model for this case (currently I'm using logloss and auc) where I dont really care about most of the population and I only try to return the samples that I'm the most certain about them.
Thanks


texts:
I have a tensor of shape (3,?,16). I want to tile each (1,16) sub-tensor 28 times horizontally and vertically so as to end up with a resultant shape of (3,?,28,28.16).
How do I go about doing this?


texts: How to generate ratings without training data?

texts: I am working on generating restaurant ratings automatically and I have various feature values like delivery time, cost estimate, etc. I want to generate a rating for each restaurant between 0 to 5. But I don't have any training data or ground truth to validate. This rating might vary with user. Most of the related work, mostly related to the Yelp data challenge, have some relevance score as training data. I though of using reinforcement learning to learn the rating with user feedback, but not sure how to do that. Can anyone please suggest a relevant technique or algorithm for this problem?


texts: Single image feature reduction at inference time : SVM

texts:
I am trying to train a SVM classifier using scikit-learn.. At training time I want to reduce the feature vector dimension.
I have used PCA to reduce the dimension.
pp = PCA(n_components=400).fit(features)
features = pp.transform(features)

PCA requires m x n dataset to determine the variance. but at the time of inference I have only single image and corresponding 1d feature vector.. I am wondering how to reduce feature vector at inference time in order to match the training dimension.
Or if anyone can suggest some other dimension reduction technique that can be used with single image wiil be highly appreciable.


texts: Applying convolutional neural network over text documents using 1-D tf-idf feature vectors

texts:
I want to apply a CNN over documents. I have tf-idf vectors of documents with me (one vector per document).

My question is, is 1D CNN applicable in this case? The reason I am asking this question is that I have read 1D convolutions being applied to sequence of vectors and I have sequence of scalars (tf-idf vector) as input in this case, so is it applicable?


texts: Regression vs. Expected value for campaign results

texts:
Let's say that I have 100k customers, and I can only send 10k letters for a loan, and I do have information from past campaigns and I know I can expect about 100 responders.

If I want to get in my next campaign the best possible outcome in terms of money (i.e. the responders who request the highest loans), what would be the difference between:

1) Creating a regression model using 0 for those who have not responded in past campaigins, and the loan amount for those who did, and select the 10k highest, or
2) Same as above, but also create a classification model and multiply the regression versus the probability of respond, and select the 10k highest

I'm confused as to what would be the proper procedure here
Any help will be greatly appreciated!


texts: Training, cross validation and testing accuracy (RMSE and R2) differs when using different shuffles and splits

texts: I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?


texts:
Very new to ML. I am trying to create an anomaly detector. I have thousands of graphs like the one I have attached. I am interested in the pink line. If the pink line's behavior changes drastically from the overall pattern of the line (i.e. the black circle), the graph needs to be flagged. While I am only interested in the pink line's behavior, I have included the blue line because it seems in every graph when the pink line dips downwards, the blue line dips upward.

I was thinking to use KNN, but I am not sure.

Any tips or guidance would be very appreciated! Thanks :)


texts:
I'm trying to use PyTorch BigGraph pre-trained embeddings of Wikidata items for disambiguation.  The problem is that the results I am getting by using dot (or cosine) similarity are not great. For example, the similarity between the Python programming language and the snake with the same name is greater than between Python and Django.  Does anybody know if there is a Wikidata embedding that results in better similarities? The only alternative I've found is Webmembedder embeddings but they are incomplete.

Wiki Item 1
Wiki Item 2
dot
cosine

Q28865 (Python language)
Q271218 (Python snake)
17.625
0.64013671875

Q28865 (Python language)
Q10811 (Reptiles)
8.21875
0.300048828125

Q28865 (Python language)
Q2407 (C++)
25.296875
0.919921875

Q28865 (Python language)
Q842014 (Django Python)
11.34375
0.409912109375

Q271218 (Python snake)
Q10811 (Reptiles)
11.25
0.409912109375

Q271218 (Python snake)
Q2407 (C++)
12.5390625
0.4599609375

Q271218 (Python snake)
Q842014 (Django Python)
6.05859375
0.219970703125

Q10811 (Reptils)
Q2407 (C++)
4.76171875
0.1700439453125

Q10811 (Reptils)
Q842014 (Django Python)
-0.60009765625
-0.0200042724609375

Q2407 (C++)
Q842014 (Django Python)
11.53125
0.419921875


texts: Are there ML Libs in Python robust to missing data?

texts:
So I was searching on how to handle missing data and came across this post from Machine Learning Mastery.
This article states that some algorithms can be made robust to missing data, such as Naive Bayes and KNN.

Not all algorithms fail when there is missing data.
There are algorithms that can be made robust to missing data, such as k-Nearest Neighbors that can ignore a column from a distance measure when a value is missing. Naive Bayes can also support missing values when making a prediction.

But then it says that sklearn implementations are not robust to missing data.

Sadly, the scikit-learn implementations of Naive Bayes, decision trees, and k-Nearest Neighbors are not robust to missing values.

Are there ML libs (preferably in Python, but could also be in other languages) that these algorithms are robust to missing data?


texts: Shuffling the training data for machine learning models

texts:
- For which algorithms is it necessary or at least important to shuffle the training data?

For example, compare regression, bagging algorithms, boosting algorithms and neural nets regarding this.


texts: how to add cross term in logistic regression model?

texts:
I have a data of 2000 (say locations of different fruits grow) and 10000 (say factors responsile for growth of fruits). And I also know that there are 20 different types of fruits in these locations.
I want to built a logistic model for this dataset such that if I get a new location then my model should be able to define the type of fruit.
log_reg_model = LogisticRegression(max_iter=500,penalty='l2',solver='newton-cg')
log_reg_model.fit(neighborhoodClass,target)

The shape of neighborhoodClass is (2000, 10000) and the target is an integer array of 20 integer labels of length 2000.
I believe that some of these fruit types are dependent on each other that means If I see a fruit A at location l1 then there is a high chance that I also observe fruit B in nearby location to l1.
That means fruit A and fruit B co-habitat in near proximity regions. Is there anyway I can add cross terms in logistic regression models to capture the neighborhood features?  So that when I see a new location with different features my model should be able to tell the type of the fruit?
Thanks.


texts: How to implement a Restricted Boltzmann Machine manually?

texts:
I am learning about Restricted Boltzmann Machines and I'm so excited by the ability it gives us for unsupervised learning. The problem is that I do not know how to implement it using one of the programming languages I know without using libraries. I want to implement it manually, which means that I want to use native functionalities of a language as much as possible.

The programming languages I know are Java, C, PHP (my preferred language), JavaScript, R and Python. I am not familiar with TensorFlow or Scikit-Learn or similar stuff.

Thanks in advance


texts: Orange Data Mining - Data Sampler Widget

texts: In the Data Sampler widget, there is an option for Cross validation for which one fold can remain unused. Is this option for nested cross validation alternating which fold you hold back?


texts: In Neural Networks and deep neural networks what does label-dropout mean

texts:
If you take the following sentence from an article on deep neural networks

  to regularize the classifier layer by estimating the marginalized effect of label-dropout during training.

What does label-dropout mean


texts: Rolling window on uneven time series classification

texts:
I have a univariate time series data that I would like to take about 60 seconds of, extract features using tsfresh and classify into multiclass.
So I might end up with a dataframe like:
Class  Mean      STD    Var
1        23      0.4    0.21
1        22      0.7    0.35
2        27       0      0
3        31      0.9    0.56
3        32      0.8    0.59
4        18      0.22   0.76

I saw a similar question: Python Time series: extracting features on a rolling window basis
However, the problem I am facing is that my time series data is not even. Some times are missing seconds and some even a few hours or days.
For ex. here it skips from 21:37:26 on 11-07 to 00:01:09 on 11-08.
As well it skips from 00:01:13 to 00:01:36 on 11-08:
Date                     Temp
2020-11-07 21:37:24       23
2020-11-07 21:37:25       24
2020-11-07 21:37:26       27
2020-11-08 00:01:09       24
2020-11-08 00:01:10       24
2020-11-08 00:01:11       23
2020-11-08 00:01:12       36
2020-11-08 00:01:13       18
2020-11-08 00:01:36       22

I'm wondering if there is a way to using rolling time series on such data?
Thanks so much!


texts: Hyper-parameter tuning of NaiveBayes Classier

texts:
I'm fairly new to machine learning and I'm aware of the concept of hyper-parameters tuning of classifiers, and I've come across a couple of examples of this technique. However, I'm trying to use NaiveBayes Classifier of sklearn for a task but I'm not sure about the values of the parameters that I should try.

What I want is something like this, but for GaussianNB() classifier and not SVM:

from sklearn.model_selection import GridSearchCV
C=[0.05,0.1,0.2,0.3,0.25,0.4,0.5,0.6,0.7,0.8,0.9,1]
gamma=[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0]
kernel=['rbf','linear']
hyper={'kernel':kernel,'C':C,'gamma':gamma}
gd=GridSearchCV(estimator=svm.SVC(),param_grid=hyper,verbose=True)
gd.fit(X,Y)
print(gd.best_score_)
print(gd.best_estimator_)

I've tried to search for examples for NaiveBayes, but couldn't find any. What I have right now is simply this:

model = GaussianNB()

What I want is to try different parameters and compare the scores.


texts: How exactly do Gaussian Processes (square dist kernel) enforce smoothness? (Aka how are they computed to do so?)

texts:
From:
http://www.cs.cmu.edu/~16831-f12/notes/F10/16831_lecture22_jlisee/16831_lecture22.jlisee.pdf

"Gaussian Processes artificially introduce correlation between close samples in that vector in order to
enforce some sort of smoothness on the succession of samples."

But how is this computed? Is the function f(x) ~ GP(mu,k(x,x')) performed incrementally?
e.g.
the n'th calculated value f(xn) uses values f(x-1)...f(x-n) to compute its mean and variance?


texts: Sampling n data points from high dimensional data

texts:
I have some face images(of a single person), which I ran through an embedding generator to get 128-dimensional embedding.
I have 1000 such embedding (shape of the dataset (1000, 128)). I have a restriction on the number of embeddings that can be used to train the model (100 embeddings). I want to pick 100 embeddings from all 1000  which will represent all 1000 embeddings.
My question is, how can I choose the best 100 embeddings which will represent the entire 1000 embeddings.
Some things I have tried out.

Picking the 100 farthest points in the cluster. (all the images are edge cases like blurred image, improper pose e.t.c)
Random sampling ( works ok but sometimes pick embeddings which were calculated from a face which was having some problem, like blurred e.t.c.)

There is one more thing, I have thought of but not tested.
Sampling with probability. Probability = 1/(euclidean_distance of point from cluster centroid).
I wanted to know if there are any alternatives, that I can look into which can provide better results.


texts: Source to learn/understand one-shot learning

texts:
I'm currently looking into one-shot learning and I wonder if there are any good sources/tutorials out there, which demonstrate one-shot learning in a (more or less) "hands-on" way?

Also some hints to indroductory literature to one-shot learning would be highly welcome! Are there any books which cover the topic?

I'm not interested in standard ML homepages which pop up on the first page of a Google search. I'm looking for recommendations of people who are somewhat familiar with the topic. So what are the best (serious) sources to start with one-shot learning? [Maybe like "Barto/Sutton: Reinforcement Learning" or "Hastie et al.: Elements of Stat. Learning"]

I would be happy for helpful recommendations!


texts: Detecting low-quality, user-created text content

texts:
I run a website that allows visitors to publish text content. It is very similar to a forum in terms of functionality. I'd like to automatically exclude or flag, submitted text content that is "spammy" or "low quality" (subjective I know). Obviously I want to detect content that is offensive or classic spam. I also want to detect content that is meaningless or nonsense. I know this must be a common problem for those that run web apps that allow user-submitted content, but I haven't found an obvious solution yet.

Ideally I'd like to use open source software that can run on Linux, suitable for me to install on my webservers, or an AWS service.

I've tried AWS Comprehend which, without any custom training, can detect the sentiment of text accurately, It doesn't seem to detect low-quality or spam out-of-the box. It supports training of custom classifiers, you simply provide a csv containing class/document pairs. This could be a good way to go, can anyone suggest a free dataset that I could use to train it?

It seems like text quality detection would be such a common requirement that I thought there might be some software out there that is pre-trained for that exact purpose, perhaps a Linux package. Does anyone have any suggestions?

I wasn't sure wether to post this in Data Science or Programming, I apologize if this is misplaced.


texts: Can I use categorical data and Decision Trees to regress a continuous variable?

texts:
Is there a way to take a set of data that consists of discrete values and predict a continuous value? Take for instance data that looks like:

sample matrix of jewel data
 color |  size | shape
['red' ,'large','square']
['blue','small','circle']
['blue','small','square']

sample array of price labels
[9.99, 7.00, 6.37]

Can I do Decision Tree Regression on this to predict the price of a jewel with a given set of features? What if some of the data is continuous? Also is there any way I can/should pre-process the categorical data other than onehot encoding?


texts: How to find the input variables for a classification problem?

texts: I am working on a  classification problem. I have 1000+ features in this dataset. I don't know how to select the right variables/ features that can actually contribute to predicting the output. What are the different methods through which I can identify the important variables that can be used out of these 1000+ variables.


texts: How can I feed BERT to neural machine translation?

texts:
I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how to give it to my model.
before that, I was using one-hot encoding and I got issues there and I want to use BERT.

Also, I have to note that, I am new in TensorFlow and deep learning. So please share your opinion with me about the use of BERT in NMT.

my goal is only to use BERT in my model for translation purpose

my model definition:

def build_model(in_vocab, out_vocab, in_timesteps, out_timesteps, units):
    model = Sequential()
    model.add(Embedding(in_vocab, units, input_length=in_timesteps, mask_zero=True))
    model.add(LSTM(units))
    model.add(RepeatVector(out_timesteps))
    model.add(LSTM(units, return_sequences=True))
    model.add(Dense(out_vocab, activation='softmax'))
    return model


texts: Search for keywords within csv/word files which are listed in an excel sheet

texts:
We have 1000's of csv, word files in different folders. Is there any open source data mining software which will look into the csv files for

1) First has to check for the Project names. If present..

2) Look for the County

3) Then, State -> Country

4)If all the conditions are satisfied, assign Order to the file.

I've brief experience working with Orange data mining software. Tried that, but couldn't even able to check project names within the csv's. Just installed Rapidminer and i can't even comprehend what i can do with it.

If anyone can shed light on what sort of software/process i have to look, will be greatly helpful.

Note: Most of the csv's & word documents are having more than 20 pages


texts:
I have to work on this database on R. I have tried to make a tree model, but all the variables are factors with too many levels. I tried changing the column "operator" in three levels (keeping private, having military for all "military - state" and air company for all the others) using something like getData$operator[grep("Military", getData$operator)] <- "Military" but it changes into NAs since it's not a valid factor.

I could change it to string and make it work, but it doesn't solve the "if not rivate or military, change to air company" part.
Is there any other way?


texts: Is it necessary to standardize your data before clustering?

texts:
Is it necessary to standardize your data before cluster?  In the example from scikit learn about DBSCAN, here they do this in the line:

X = StandardScaler().fit_transform(X)

But I do not understand why it is necessary.  After all, clustering does not assume any particular distribution of data - it is an unsupervised learning method so its objective is to explore the data.

Why would it be necessary to transform the data?


texts: LSTM for time series - which window size to use

texts:
I have a LSTM based network which inputs a n-sized sequence of length (n x 300) and outputs the next single step (1 x 300).

The "raw" data consists of a few thousand semi-processed sequences of variable length where each step is (obviously) 1 x 300. Hence X is (n x 300).

The way I am generating the training and test data, from this semi-processed original sequences is:

drop all original sequences shorter than K=9
apply a sliding window with stride 1 and length K=9 to each original sequence kept
shuffle the generated data
separate train/dev test/test data

Now all training/testing data is [9 x 300] and Y is [1 x 300]

The resulting network starts overfitting around epoch 10 which led me to 1. This is itself is not a problem since the results are of good enough quality.

If I try sequences as short as 6 (as in 6 x 300, that were dropped during the first phase) it gives proper enough information for us in y.

Hence the questions are about good practices and improving the network.

Technically, I am using Keras and using Google Cloud ML to execute the hyper parameter search.

The questions are:

should I stick to generating data with sliding windows since it is working properly?
if sliding window, k=9 is kind of a best guess, should I hyperparameterize this number K and search for the minimum loss with it in mind? I am afraid of doing so because hyperparameters typically affect the network itself, not the training data set, but note that K defines the data set length, would I be generating undesired bias by hyperparameterizing it?
if sliding window, should I be generating y shaped [9x300] with y=[x2, x3,...,x9, x10], instead of the current y generation? if I try both output structures and pick the one that lowers the loss in the dev test, I would check against a validation set?
if no sliding window, should I be training with one sequence per original time series input? and use attention or bidirectional lstm? 2
shuffling generates batches with shuffled partial sentences instead of batches from the same original sequence. but since shuffling is done prior to separating data for training and testing, it seems like I am not doing it the best way (or even properly). Should I first separate the original long sentences into train/each test phase, and then break them with the sliding windows (if applicable)?

References:

1 sliding window leads to overfitting in LSTM?
2 RNNs for time series prediction - what configurations would make sense


texts:
I am needing help with modeling some data to see if there is a correlation between two variables.  I am wanting to know what would be the best math model to use to analyze the data.  Right now, I am leaning toward developing a linear regression model and create some type of Histogram.  I’m curious to know if anyone had a better suggestion to visualize the data.
Essentially, I am comparing the day of week to an approximate time value by the hour such as 1 p.m.  I have a total of approximately 2400 observations.


texts: Which, if any, machine learning algorithms are accepted as being a good tradeoff between explainability and prediction?

texts: Machine learning texts describing algorithms such as gradient boosting machines or neural networks often comment that these models are good at prediction, but this comes at the price of a loss of explainability or interpretability. Conversely, single decision trees and classical regression models are labelled as good at explanation, but giving a (relatively) poor prediction accuracy compared to more sophisticated models such as random forests or SVMs. Are there machine learning models commonly accepted as representing a good tradeoff between the two? Is there are any literature enumerating the characteristics of algorithms which allow them to be explainable? (This question was previously asked on cross-validated)


texts: Computer vision: Identifying different items in screenshot

texts:
Lets say I have a screenshot like this:

I want to be able to detect/localize each item on the floor, however, there 1) can be any number of items in the image and 2) each item is different

I have a candidate list of all possible items. In reality each one is labelled and separated into individual image files:

I've thought about training something like a convnet, but that feels like it might be slow because I'd need to segment each screenshot with multiple sliding windows and feed each one forward through the net. Creating those sliding window segments for each screenshot will likely take a long time. I'd like the entire detection process to be completed quickly (<1sec)

Whats the most efficient way of doing this detection task? I will be implementing this using Javascript

The main problems are:

The item count is unknown. There could be 0 items in the screenshot, or there could be many
There are a lot of possible targets, each one of which are different in shape, colour, and structure
Javascript isn't the fastest of languages for this type of work. This is a semi strict requirement unless its absolutely impossible to do this using JS. The fallback language would be Python


texts: What is the form of data used for prediction with generalized stacking ensemble?

texts:
I am very confused as to how training data is split and on what data level 0 predictions are made when using generalized stacking. This question is similar to mine, but the answer is not sufficiently clear:

How predictions of level 1 models become training set of a new model in stacked generalization.

My understanding is that the training set is split, base models trained on one split, and predictions are made on the other. These predictions now become features of a new dataset. One column for each model's prediction, plus a column that contains the ground truth for these predictions.

Split training data into train/test.
Train base models on training split.
Make predictions on test split (according to linked answer, use k-fold CV for this).
Create a feature for each model, filling it with that model's predictions
Create a feature for the ground truth of those predictions.
Create a new model and train it on these predictions and ground truth features.

Question 1: Are these the only features used to train the "meta" model? In other words, are none of the actual features of the original data included? The linked answer says it is common to include the original data, but I have not read about it elsewhere.

Question 2: If the above algorithm is correct, What is the form of the data when making predictions? It seems like it would also have to have predictions as independent variables. If so, that means running all new incoming data through all base models again, right?

Question 3: I keep seeing an "out-of-fold" requirement for the first level predictions. It seems that doing a simple train/test split as mentioned above would fulfill this. However, would you not want a 3rd split to test the combined model's generalization? Or is this type of ensemble bulletproof enough not to worry about it?


texts: Is it ok to trust regression predictions when none of the coefficients are statistically significant?

texts:
Background to the problem: I am estimating individual treatment effects using double machine learning model. I do not know true treatment effects for my problem.

Double ML: Given Y (outcome), T (treatment) and X ( features)

Y = aT + bX + error

coefficient a is of interest (measures treatment effect) .

Double ML procedure:

Fit Y ~ X  => Compute residuals (Y* = Y – Y’) – Residuals are
treatment effects to be estimated
Fit T ~ X => Compute residuals (T* = T- T’) – This model captures
variation in T explained by X
Fitting a model (Y* ~ T* ) on residuals will give the average
treatment effects

I am fitting a linear regression model (Y* ~ T* ) and none of the coefficients are statistically significant. Instead of relying on point estimates, I am computing prediction confidence intervals and p-value to check if the predicted value is statistically significant or not.

Is this approach good?


texts: Use the LMS algorithm to train a single perceptron neural network by finding the weights for a given data

texts: Basically, I have data set contain inputs and outputs for unknown system, I want to use the LMS (Least Mean Square), to train a single perceptron (neuron) NN, and find the weights $ w_1, w_2, ... $, for the given output.


texts: Sklearn: applying cost complexity pruning along with pipeline

texts:
I have a data set with categorical variables. I have defined a decision tree algorithm and transformed these columns to numerical equivalent using one hot encoding functionality in sklearn:
Create Decision Tree classifer object:
clf2 = DecisionTreeClassifier(criterion = 'entropy')
pipe = make_pipeline(column_trans, clf2)            # (1)
pipe.fit(X_train2,y_train2)

where:
column_trans = make_column_transformer(
            (OneHotEncoder(),['ShelveLoc','Urban','US']),
             remainder = 'passthrough')

Now when I built the decision tree without using sklearn but using pandas directly for categorical feature encoding, I was able to find the suitable candidates for alpha to prune the decision tree by
path = clf.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas
ccp_alphas = ccp_alphas[:-1] #remove max value of alpha

where as now given that my model is baked into pipe argument in (1) when I try to find candidate alphas
path = pipe.cost_complexity_pruning_path(X_train2, y_train2)

I get an error message saying pipe does not have the attribute called cost complexity pruning. and looking at all the attributes available to pipe, I can't find cost complexity pruning as well.
Is it only possible to do cost complexity pruning if you are building the model without using the pipe functionality in Sklearn?


texts: Predict part of the input for a fixed target

texts:
I dispose of a data set composed of 6 features all of them are numeric and a binary target(taking 0 and 1). How should I proceed in order to predict the values of 2 features knowing the target and the remaining fixed 4 features.

Thanks in advance.


texts: I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need <SOS>, <EOS> tokens which are not trained with BERT. How should I get them ? Can I use <CLS> token as <SOS> and <SEP> as <EOS> ? Or I have to create these two embeddings as trainable Variables and concatenate them to the decoder input / labels ?


texts: how can LSTM output ever be negative with 0 initialization?

texts:
From the lstm equations, e.g. as they appear on p406 of the Deep Learning Book, it looks to me like initializing with zeros (as is common practice), must always produce a strictly positive output. When $s_0 = 0$, then the $i$th output unit at the first time step can be written as
$h_i = \sigma(A)tanh((\sigma(B)(\sigma(C))$ where $A$, $B$ and $C$ are linear functions on the current input and the previous output. But isn't then $h_i > 0$ for all real numbers $A$, $B$ and $C$?


texts: CART algorithm (Classification and regression trees) question

texts: So this is taken from an exam I just did. I'd like to know if there are any instances same as in the image where the CART algorithm could use a negative alpha and thus encourage a larger tree? Or does the algorithm state that alpha must be a non negative integer at all times?


texts: NLP to detect duplicates for very technical language

texts: I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to consider that different annotation and abbreviations might have been used for technical names, text errors in data entries,  similar/different dimensions or characteristics might still point to the same product. Therefore, I think that applying a normal fuzzy matching or other NLP text matching will not perform well in my case. I trying to approach this problem as a learning/supervised model, but still not sure how so any suggestion/idea is very appreciated.


texts:
I am training object detection using CNN

I am using MSE as loss functions, at best it throws loss around 1800 on testing and 1850 on training data, Which in my opinion is quite acceptable, considering that if the bounding box is "dislocated" by 20px on every axis, its already 1600 ( before mean ).

However, when the network on the exact data i have trained it on, it result in extremeley wrong results, almost like random no way it has around 1800 mean difference.

What woul general approach be for dealing with things like this? Thanks for help.


texts: Why Heaps' Law Equation looks so different in this NLP course?

texts:
I'm actually not sure if this question is allowed on this community since it's more of a linguistics question than it is a data science question. I've searched extensively on the Web and have failed to find an answer and also the Linguistics Beta Stack Exchange community also doesn't seem to be able to help. If it's not allowed here please close it.

Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia link is

$$V_R(n) = Kn^\beta$$

where $V_R$ is the number of distinct words in a document of size $n$, and $K$ and $\beta$ are free parameters that are chosen empirically (usually $0 \le K \le 100$ and $0.4 \le \beta \le 0.6$).

I'm currently following a course on Youtube called Deep Learning for NLP by Oxford University and DeepMind. There is a slide in a lecture that demonstrates Heaps' Law in a rather different way:

The equation given with the logarithms apparently is also Heaps' Law. The fastest growing curve is a corpus for Twitter data and the slowest is for the Wall Street Journal. Tweets usually have less structure and more spelling errors, etc. compared to the WSJ which would explain the faster-growing curve.

The main question that I had is how Heaps' Law seems to have taken on the form that the author has given? It's a bit of a reach but the author didn't specify what any of these parameters are (i.e. $C$, $\alpha$, $r(w)$, $b$) and I was wondering if anybody might be familiar with Heaps' Law to give me some advise on how to solve my question.


texts:
I have salary data of several user (Python list).
Now I am using KMeans to cluster them.
Given this data, Is there a way to figure out the best value for 'K' automatically through program?
I tried silhouette analysis
take k with highest silhouette_avg score. Is this the best way?


texts:
as I am very interested in programming and statistics, Data Science seems like a great career path to me - I like both fields and would like to combine them. Unfortunately, I have studied political science with a non-statistical sounding Master. I focused on statistics in this Master, visiting optional courses and writing a statistical thesis on a rather large dataset.

Since almost all job adds are requiring  a degree in informatics, physics or some other techy-field, I am wondering if there is a chance to become a data scientist or if I should drop that idea.

I am lacking knowledge in machine learning, sql and hadoop, while having a rather strong informatics and statistics background.

So can somebody tell me how feasible my goal of becoming a data scientist is?


texts: Pandas/Python - comparing two columns for matches not in the same row

texts:
I have this data:

I wanted to compare A and B for matches not by row but rather search A0 if it is in column B and so on. Moreover, I wanted to ignore the .AX in column A because it would not find any matches in column B anyway.
I used this, but it matches values row by row and it returns False or True. I would like to print the matches in a new Column C:
    df3['match'] = df3.A == df3.B

Thank you.


texts: How to work with Sensitive (personal) Data, without actually seeing it?

texts:
I'm currently working on a project that has sensitive data. I'm allowed to see the data on a single platform (company PC), but outside of that platform I cannot process it. I'd like to train a ML algorithm with the data but to do so I'll have to use a different PC. I cannot process the data on that PC though.
Is there anyway to translate and preserve the data in a format that can still be used for ML, but isn't in it's explicit original form?


texts:
I wanted to ensure that I understood the random forest algorithm right. I think it's fundamental to understand how something work before using it!

Am I right with this:

If not: what did I understand wrong?


texts: What does it mean when we say an algorithm/metric is agnostic

texts:
Problem

I have all kinds of machine learning terms that co-occur with the word "agnostic", including model-agnostic learning, model-agnostic metric. From the dictionary, it explains the word "agnostic" in the following way

  a person who holds the view that any ultimate reality (such as God) is unknown and probably unknowable.

which does not make those terms more understandable.

In some contexts, I find that "agnostic" refer to "generic" or "free of". For example, in the paper I am reading now, the authors define a threshold-agnostic metric, where they use score rather than hard 0/1 assignment for the task.

However, I am wondering if there is formal definition for the word "agnostic" in the machine learning community.


texts: CNN for subsets of a dataset - how to tune hyperparameters

texts:
I have a dataset and would like to train CNNs on subsets of different size of the dataset.
I already have a CNN, which classifies very well if I use the entire dataset.
Now the question arises if I should really try to additionally optimize the parameters of the CNN for the subsets, regardless of whether I do Data Augmentation or not?
Does it really make sense if I try to change the CNN model for the subsets by using RandomizedSearchCV or GridSearchCV to optimize the number of convolutional layers, different learning rates, etc....?

In other words, suppose I found the perfect CNN model for a dataset. Is this model also the perfect model for subsets of this dataset?

I hope someone can give me a hint. For any help, I thank you in advance.


texts: Compute specificity and sensitivity at certain thresholds

texts:
I have the following table with predictive probabilities and true class labels:

\begin{array}{|c|c|c|c|}
\hline
P(T=1) &0.54& 0.23 & 0.78 & 0.88 & 0.26 & 0.41 & 0.90 & 0.45&0.19&0.36 \\ \hline
 T&1&0 &0 &1 &0 &0& 1& 1& 0& 0\\ \hline
\end{array}

The question is to compute the specificity & sensitivity at the threshold of 0.5.

My attempt at answering this question:

Sensitivity = true positive rate[P(T=1) > 0.5]

= (0.54 + 0.88 + 0.9)/4 = 0.58

Specificity = 1-false positive rate[P(T=1) > 0.5]

= 1- [(0.78)/6]
 = 0.87

Not sure if my working above is correct. I would appreciate if someone can guide me to the correct solution. Thanks.


texts: Max depth for a decision tree in sklearn

texts:
I know there is a partial answer here but my question is slightly different.
I have implemented a decision tree in sklearn. Say I have $2^n$ different values for a feature, with just one feature. I was expecting that a good decision tree should be able to keep the depth at or below $n$. Let's say the $2^n$ values for the features are indexed ($1 \dots 2^n$), if I make a split in the middle, that is the most effective way, and I can get a leaf with just one element at the end of a a n-deep tree.

Of course, it could be less, if, for example, I only have two categories, and all the elements below $2^{n-1}$ are in one class, and the other elements in the other class. But, in the worst case, I thought I should have depth $n$. However, when I implemented a solution for just around 1 million examples with one feature that has about 100,000 values, I got a tree with depth 125! (and I was expecting about 17).

Why is that?


texts: Python: advice on which machine learning algorithm to use for a problem which involves lots of randomness

texts:
I'm new to machine learning so I'll summarize my problem with two examples without getting technical (because I can't).
The dog vs. cat classification example is solvable, in the sense that a human can tell you whether its a dog or a cat with certainty. Many machine learning algorithms are able to replicate the human performance and identify dog or cat with near certainty.
For my problem, there is no certainty, only a slight-better-than-random prediction. I am trying to predict whether a person who was recently released from incarceration will commit a crime within the next year. Lets assume the actual chances of re-offending are about 50/50. If I could use machine learning to make a modestly-better-than-random prediction, that would be a huge win for me. More specifically, if 50/50 is a random guess, then if I could achieve 55% to 60% success rate, that would be considered wildly successful.
I know this task is possible since I have used a dataset (with around 50 features and 100,000 observations) to make a "man-made" linear regression which achieves around 52% out of sample.
I have tried SKLearn's logistic regression and XGBoost but their performance has been lower than my "man made" attempt. I'm assuming that is because these algorithms aren't meant to deal with a prediction of an event that is mostly random.
Given that I am dealing with the prediction of an event which is mostly random and I am only looking to achieve slightly-better-than-random predictions, is there a machine learning algorithm / strategy you could recommend to best tackle this problem?


texts: Can I use an ANN to translate image output from one sensor to simulate output from another sensor?

texts:
Say, for instance, if I had image data from one high resolution digital camera and wanted to make it look like it was taken from another, lower resolution, digital camera?  Would training input/output pairs of overlapping images be a good way to do this?  What is this technique called?

For example, say I wanted to be able to count benches in parks in LOW resolution imagery.  Could I go through these sample images and create an appropriate dataset of high and low resolution pairs to train a network to learn what a low resolution bench looked like?  Would I be able to discern low resolution benches if my training set was incredibly diverse (image chips if entire city parks vs individual objects like fountains, trees and statues)?

I like this example because the images come from different sensors as well as being different resolutions. Some of my research has led me to super resolution, which is kind of the opposite of what I'm trying to do.

As for the amount of data, it would be painstaking but not technically difficult to get overlapping high and low resolution imagery.


texts:
I am studying the book "Elements of Statistical Learning". In chapter 12 it is given about the Generalized Linear Discriminant Analysis. In one of its section it is about Flexible Discriminant Analysis.

I am not able to understand the advantage of FDA and its algorithm. Also the complex mathematical equations given about FDA is difficult for me to understand. Can anyone explain me in simple language ,how do they come to the following equations?


texts:
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:

def kmeans(data, k, c=None):
    if c is not None:
        centroids = c
    else:
        centroids = []
        centroids = randomize_centroids(data, centroids, k)

    old_centroids = [[] for i in range(k)]

    iterations = 0
    while not (has_converged(centroids, old_centroids, iterations)):
        iterations += 1

        clusters = [[] for i in range(k)]

        # assign data points to clusters
        clusters = euclidean_dist(data, centroids, clusters)

        # recalculate centroids
        index = 0
        for cluster in clusters:
            old_centroids[index] = centroids[index]
            centroids[index] = np.mean(cluster, axis=0).tolist()
            index += 1

    print("The total number of data instances is: " + str(len(data)))

I have tested it for serial execution and it is OK. How to make it distributed in Hadoop? In other words, what should go to the reducer and what to the mapper?

Please note that if possible, I would like to follow the tutorial's style, since it's something I have understood.


texts: Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

texts:
I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave like that and would like to know why.

I have implemented the original word2vec in keras. I chose the variant with the dot product layer rather than the hierarchical softmax and trained the model on a Wikipedia dump that I split into 5-grams. For each word I construct 8 pairs with a binary target label as training items. I use the 4 context words with the label True and choose 4 random words that are not one of the context words with the label 0.

Intuitively this model should learns similar representations for words in similar context, because it modifies the representation of these words in a similar manner, as it optimizes them independently with similar context words.
So these similar words are not directly made similar but rather indirectly, as they are subject to similar nudges due to their similar contexts.

The model is this:

input_target = Input((1,))
input_context = Input((1,))

embedding = Embedding(vocab_size, vector_dim, input_length=1, name='embedding')

target = embedding(input_target)
target = Reshape((vector_dim, 1))(target)
context = embedding(input_context)
context = Reshape((vector_dim, 1))(context)

dot_product = Dot(axes=1)([target, context])
dot_product = Reshape((1,))(dot_product)

output = Dense(1, activation='sigmoid')(dot_product)

model = Model(inputs=[input_target, input_context], outputs=output)
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

However, what I observe from training this model is, that when I rank the most similar words given a certain word, that I get words that appear in the context of that word, rather than words that appear in similar context, as the most similar words.

For example:

words most similar to "plant":

rank |   word

0    |   amount
1    |   surface
2    |   electron
3    |   mass
4    |   plant # also: Why is plant not most similar to plant? How can that happen?
5    |   fluid
6    |   air
7    |   metal
8    |   molecule
9    |   cell
10   |   electric
11   |   per
12   |   oxygen
13   |   demonstrate
14   |   smooth

To me that looks a lot more like words that appear in the context of plant rather than words that appear in similar context as it.

The function to compute these is:

def get_most_similar(word_vector, embeddings, n=15):
    """
    find the `n` words that are most similar to `word_vector` in `embeddings`
    measured by their cosine similarity
    """
    v = word_vector
    m = embeddings
    cosines = (np.dot(v, m.T))/(np.linalg.norm(m.T, axis=0)*np.linalg.norm(v))
    ranked_by_similarity = np.argpartition(cosines, -n)[-n:]
    return reversed(ranked_by_similarity)

Is there a simple reason for that?

I have the following other parameters:

word vector size: 300
batch size: 128
vocabulary size: 169161 (distinct lemmas)
training sample count: 27793586 (5-grams, overlapping within sentences)

I trained the model for 1 epoch and did only observe marginal further convergence into the second epoch.


texts: Determining if time series follows a pattern

texts:
I was wondering if anyone had any idea how to solve this problem.

So basically I have a dataset where some person approximately comes at some regular interval and I don't know what that interval is. I need to determine if the person comes in at approximately regular intervals, not necessarily what is the specifically value of the interval. For example if a certain person comes to my house to deliver milk over some period on say

Week 1: Mon, Wed, Fri,

Week 2:Mon, Wed, Fri

Week 3:Mon, Thu, Fri

Week 4:Mon, Wed, Fri

Week 5:Mon, Wed, Fri

Week 6:Mon, Wed, Fri

Week 7:Mon, Wed, Fri

Week 8:Mon, Thu, Fri

So as we can see out of these 8 weeks only in 2 weeks the person didn't come on Wednesday and instead came on Thursday which can be attributed to maybe a holiday the day before. So the solution to this example is that the person does follow a regular pattern.

Similarly this is another example. Say the person came on -

Mon, Thu, Sun, Wed, Sat, Tue, Fri, Mon, Wed, Sun

where this person follows a regular pattern because except for the last Wednesday he comes every fourth day.

This is an example of where a person doesn't follow a pattern, say the person came on

Mon, Wed, Sat, Fri, Thu, Fri, Wed, Fri, Sun, Sun, Sat

I have to do this for hundreds of different people.

Another equivalent problem is if I know which days over a certain period of time (say a month) some person arrives I need to determine if they follow some pattern or not

I thought about trying to fit the data into a sinusoidal curve but I'm not sure if it will work when I have a person who comes every say Mon, Tue, Fri or the 7th of every month or the first and third monday of every month etc.

I'm open to any method as long as it has good accuracy. Also, depending on whichever algorithm you think is best, if possible could you share a link to some code which solves a similar problem so I can get a general idea of how I'm supposed to implement my algorithm. I'm pretty new to machine learning / data science. Thank You!


texts: AUC ROC Threshold Setting in heavy imbalance

texts:
I am doing binary logistic regression on a dataset with very heavy class imbalance. Class 1 is only 1% of data. When I train logistic regressor without class weights I get ROC AUC Score of 0.6269. Which is decent. However, when I see my confusion matrix I see that my model never predicted any 1's at all. So why is my AUC so high? I though AUC is meant to be a good measure for such a case.

Confusion matrix
 Predicted      0    All
True
0          32109  32109
1           1223   1223
All        33332  33332

I know Confusion matrix makes the probability threshold 0.5, so is score saying there is some threshold for which model will give higher recall? How can I get this threshold?

      Class  precision    recall  f1-score   support

       0       0.96      1.00      0.98     32109
       1       0.00      0.00      0.00      1223

I only care about precision and recall of class 1.


texts: How to avoid overfitting in random forest?

texts:
I want to avoid overfitting in random forest. In this regard, I intend to use mtry, nodesize, and maxnodes etc. Could you please help me choose values for these parameters? I am using R.
Also, if possible, please tell me how I can use k-fold cross validation for random forest (in R).


texts: How to avoid one hot encoding again and again on each run

texts:
I have a dataset which is categorical. I did one hot encoding in the train data which is taking much time for each run of my algorithm. I have also performed one hot encoding on the test data which I give at the run time to the algorithm. I want to know how can we avoid the one hot encoding of the dataset for each run, rather I want it to be done only on the new test data.

I am working on classification problem using Decision tree classifier.My dataset is categorical so I have done one hot encoding on the dataset as well as on the active user input given at the run time . It takes too much time while doing encoding in dataset. I wish , if once one hot encoding is done on training dataset, I should not do it again and again on each run and to perform encoding only the active test input i.e. on 'a'.

features = [0,1,2,4,5,6,7,8]
X = dataset.iloc[:,features]
y = dataset.iloc[:,3]
X_ordinal = X.values

encoder = OneHotEncoder(handle_unknown = 'ignore')
encoder.fit(X_ordinal)
X_one_hot = enc.transform(X_ordinal)
end4 = time.clock()

print("enter the inputs in space separated format:" )
start5 = time.clock()
a = [int(p) for p in input().split()]

values=array(a)
X_test_one_hot=encoder.transform(values.reshape(1, -1))


texts: Keras Bidirectional LSTM: low training and validation loss but very bad predictions

texts:
I'm training a Bidirectional LSTM using Keras.
My task is to predict the words order in a sentence, so, given a sentence, output of each timestep will be a real number: predicted real numbers of the sentence are ranked in order to obtain integer numbers, indicating the predicted position of the word in the sentence.

Example:

sentence -> "Nice I am"

predicted real numbers -> [0.2, 0.6, 0.4]

ranked real numbers -> [3,1,2]

Basically, I pad my sequences to 20, that is the max sequence length found in the dataset.

My model is the following:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=timesteps, return_sequences=True), input_shape=(timesteps, features)))
model.add(Dropout(0.2))
model.add(Dense(1, activation='linear'))

I mask all the padded vectors to avoid noise in learning.
My training loss and my validation loss, during the training, are both low and around 0.33.

In the prediction phase, however, I obtain very bad results.
I don't think my model is overfitting/underfitting looking at losses trend.

Is there something bad in my model architecture?

Thanks in advance.


texts: Forecasting/Predicition for a specific date

texts:
First of all data science is not my experties so apologies if this sounds stupid.

I have a data X that is either 1 or 0 and (possibly) dependent on some other variables. I have the values for these variables mesured at specific dates. Given a date, I need to predict whether X will be 1 or 0 on that day in the futire.

I checked logistic regression which seems to fit my definition (multiple independent variables, binary result). The problem is that none of the agorithms take date as a parameter. They all try to predict a given value which I dont want.

How do I forecast for any given date?

EDIT: Given

Traffic Volume (float),
Active Users (integer),
Throughput (float)

Congestion Flag (0 or 1) (we need to forecast this)


texts: how to apply MC dropout to an LSTM network keras

texts:
I have a simple LSTM network developped using keras:

model = Sequential()
model.add(LSTM(rnn_size,input_shape=(2,w),dropout = 0.25 , recurrent_dropout=0.25))
model.add(Dense(2))

I would like to apply the MC dropout method. How can I enable dropout in the test phase in order to compute the uncertainty?

Thanks.


texts: Which combination of 3 hyperparameters to combat overfitting of a convolutional neural network?

texts:
I have a small dataset with which I want to train a CNN by using Data Augmentation. Since the CNN is overfitting due to the small data set, I would like to optimize some hyperparameters.
However, since I would like to use GridSearchCV from Scikit-Learn for this and I would therefore like to optimize only 3 hyperparameters due to reducing computational time. Here the question arises which combination of hyperparameters should I use for grid search?

My current approach would be to optimize the learning rate, the dropout layer rate and the number of epochs.

I choose the learning rate because the book "Deep Learning" by Goodfellow recommends to always optimize the learning rate. But I'm not sure if my combination of hyperparameters for tuning is really good.

What combination would you recommend?
Many thank for every hint

My previous architectur is as follows:

model = Sequential()
model.add(Conv2D(32,(3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Conv2D(32,(3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2))
model.add(Dropout(0.2))

model.add(Conv2D(64, (3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Conv2D(64,(3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2))
model.add(Dropout(0.2))

model.add(Flatten())

model.add(Dense(512))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.4))
model.add(Dense(10))
model.add(Activation('softmax'))

As optimizer I use Adam.


texts: Is there an established methodology for human expert evaluation of machine learning predictions?

texts:
Assume a continuous dependent variable $Y$. Then, $\hat{Y}$ is the predictions made by a machine learning-based model using a set of independent variables $X$.

In this case, however, it is known that $Y$ is not the best possible values for this variable. That is, $Y$ is actually the sum of $Y^{true}$ and $\epsilon$.  $Y^{true}$ is the hidden true value of the variable that cannot be known to anyone. $\epsilon$ does not follow a known distribution or pattern.

In this case, any error measure $f(Y, \hat{Y}$) is not trusted. In principle, it is believed that human experts can judge the quality of predictions $\hat{Y}$ by examining the values of $X$. However, there are many ways where bias can sneak into this evaluation and it might be difficult to quantify the prediction quality.

Is there an established methodology (i.e., in terms of number of experts, the followed procedures, on which scale the prediction quality is judged, how conflicting expert opinions are accounted for, etc.) for evaluating the quality of such predictions?

If it makes it easy to understand or answer, the data and problem can be assumed to be in a business domain.


texts: Is it safe to use labels created from unsupervised model to train a supervised model using the same data?

texts:
I have a dataset where I have to detect anomalies. Now, I use a subset of the data(let's call that subset A) and apply the DBSCAN algorithm to detect anomalies on set A.Once the anomalies are detected, using the dbscan labels I create a label variable (anomaly:1, non-anomaly:0) in the dataset A. Now, I train a supervised algorithm on dataset A to predict the anomalies using the label as the dependent/target variable and finally use the trained supervised model to predict the anomalies on the rest of the data (A compliment).

While, this seems to be a fair approach to me, I am just wondering if there is any data leakage happening at any stage. Please note that I am using the same set of variables/features at both stage(unsupervised and supervised). Reason for posting is when I train the supervised model, I get very high roc-auc score, which is around 0.99XX and that is suspicious.

Note that, I can not use the DBSCAN algorithm for the entire data set because of computational constraints.
I can not use supervised model as I do not have labels.


texts: How to implement gradient descent for a tanh() activation function for a single layer perceptron?

texts:
I am required to implement a simple perceptron based neural network for an image classification task, with a binary output and a single layer, however I am having difficulties. I have a few problems:

I am required to use a tanh() axtivation function, which has the range [-1, 1], however, my training labels are 1 and 0. Should I scale the activation function, or simply change 0 labels to -1?
So the gradient descent rule dictates that I shift the weights according to the rule:

$$
\Delta \omega = -\eta \frac{\delta E}{\delta \omega}
$$

I am using the mean square error for my error:

$$
E = (output - label)^2
$$

Considering my output is $o = tanh(\omega .x)$, x is my input vector and $y_i$ is the corresponding label here.
$$
\frac{\delta E}{\delta \omega} = \frac{\delta (y_i - tanh(wx))^2}{\delta \omega} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) \frac{\delta wx}{\delta w} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) x \\= -2(y_i - o)(1 - o^2)x
$$

I implemented this is python, the dot product of the input vector with the weights turns out to be too large, which makes $tanh(x)=1$ and $1-o^2 = 0$, so I can't learn. How can I circumvent this problem?

Thanks for the replies.

The implementation:

def perc_nnet(X, y, iter = 10000, eta=0.0001):
    a, b, c = X.shape
    W_aug = np.random.normal(0, 0.01, a*b+1)
    errors = []

    for i in range(iter):
        selector = rd.randint(0,c-1)
        x_n = X[:,:,selector].ravel() #.append(1) #has the bias as well
        x_n = np.append(x_n, 1)
        v = x_n.dot(W_aug)
        o = np.tanh(v)
        y_i = y[:,selector] if y[:,selector]==1 else -1
        MSE = 0.5*(o - y_i)**2
        errors.append(MSE)
        delta = - eta * (o - y_i) * (1 - o**2) * x_n
        W_aug = W_aug + delta

    return W_aug, errors


texts: Supervised learning for image segmentation

texts:
Hello I am new to Applied ML and trying to solve a problem where I have given several images with few patches in it and masked images where these patches are classified/extracted as training data.Given that i need to use ML to train network such that It can generate those masked images for test data set.
I would like to know which technique will be best suitable in Both the cases where I want to use Python extended libraries like Scikit or Using just Numpy and Scipy.
Thanks in advance


texts: Observation window in Predictive modelling

texts: In any predictive modelling exercise we first start with defining observation window and perform window for the product/problem. Just wanted to know if the window is different for different predictors or it's same for all the predictor variables. for ex- variable A can be calculated in 3 year of observation window. Can another variable (Variable B) be calculated in 2 year of window ?


texts: Generating artificial data to extend learning set

texts: I have dataset containing 42 instances(X) and one final Y on which i want to perform LASSO regression.All are continuous and numerical. As the sample size  small, I wish to extend it. I am kind of aware of algorithms like SMOTE used for extending imbalanced dataset. Is there anything available for my case where there is no imbalance?


texts: Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?

texts:
During the equation deducing in policy gradient algorithm(e.g., REINFORCE), we are actually using an expectancy of total reward, which we try to maximize.

$$\overline{R_\theta}=E_{\tau\sim\pi_\theta}[R(\tau)]$$

Can't it be seen as an Actor-Critic Method since we are using V(s) as a Critic to guide the update of Actor π? (Here we've already introduced an approximation)
$$\nabla \overline{R_\theta} = \sum_{n=1}^N R(\tau^{(n)}) \nabla \log p(\tau)$$
If not, what's the clear definition of Actor and Critic defined in Actor-Critic Algorithm.


texts:
For some $8000$ customer profiles, in addition to a data-set, I have two kinds of scores available:

Type 1 Score ranges from $0$ to $1$ and gives the prediction probability of that profile belonging to a class. The distribution of this score is severely skewed, as shown

Type 2 Score ranges from $-5$ to $5$ and is a risk score for the profile obtained using the PRIDIT method on the actual data. The distribution of this score is fairly normal

The objective is to bin the profiles into some $N$ bins using the aforementioned information. The bins should represent various degrees of risk associated with that profile.

My first thought was to cluster the profiles based on the two scores available to us. However no meaningful clusters were observed.

The next idea was to sort the data on Type 1 Score and then cut on the points where there's a sudden change in slope, which seemed like a good idea at first but here's what the plot for Type 1 Score looks like ...

What I am thinking now is to transform Type 1 Score and Type 2 Score together to give some pseudo-normal type of distribution, hoping that some kind of discretization might give the correct bins.

My questions are:

How can I transform the two scores into a pseudo-normal distribution?
What would be a better way to approach the above problem?


texts: SGD learning gets stuck when using a max pooling layer (but it works fine with just conv + fc)

texts:
I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer.

  Please note that the whole thing was built from scratch following the
  neuralnetworksanddeeplearning book and other resources, so I'm aware there are ready-to-use libraries out there, as well as more optimized algorithms to perform certain operations (like the FFT convolution algorithm), but the point of this thing was to understand how a network works and to implement everything without using 3rd party libraries.

I'm quite sure the fully connected layers work fine, as I've tried training a network with the MNIST dataset and it quickly gets to over 96% in less than 60 epochs, just like the one in the book mentioned above.

I went on and implemented the convolutional layers, and they seem to be working fine too, in fact if I try to have a network like this one (sigmoid activation on all the layers):

Convolution: 28*28*1 inputs, 20 5*5*1 kernels
Fully Connected: 24*24*20 inputs, 100 outputs
Fully Connected: 100 inputs, 10 outputs, cross-entropy cost

And here's what I got:

As you can see, the network quickly reaches over 96% accuracy in just 3 epochs.

But, as soon as I try to put a 2*2 pooling layer between the convolutional and fully connected layer (just like Michael Nielsen did in the 6th chapter of his book), here's what I get:

The network reaches 30% and then just gets stuck there.

I've checked and everything seems to be working fine (but of course I can be wrong, and I probably am, since this thing is not working as it should), the forward pooling layer takes the maximum value in each 2*2 window across all the input images (doing so for each depth layer), and during the backpropagation through the pooling layer I just upscale the input delta using the previous outputs from the convolutional layer, so that each delta goes to the pixel that had the maximum value in the convolutional output, while all the other pixels get 0.

Where should I start looking for the error? I mean, clearly something happens with the pooling layer, but the code looks fine to me, both in theory and with the actual Unit tests I've added to the project. What's the right approach to take here to investigate the issue? Are there probable explanations for this issue?

Here's the code I'm using, if anyone has .NET Core 2.x installed and wants to try the library out (The code to download/parse the MNIST dataset is already in the library):

((float[,] X, float[,] Y) training, (float[,] X, float[,] Y) test) = DataParser.LoadDatasets();
INeuralNetwork network = NetworkTrainer.NewNetwork(
    NetworkLayers.Convolutional((28, 28, 1), (5, 5), 20, ActivationFunctionType.Identity),
    NetworkLayers.Pooling((24, 24, 20), ActivationFunctionType.Sigmoid),
    NetworkLayers.FullyConnected(12 * 12 * 20, 100, ActivationFunctionType.Sigmoid),
    NetworkLayers.FullyConnected(100, 10, ActivationFunctionType.Sigmoid, CostFunctionType.CrossEntropy));
await NetworkTrainer.TrainNetworkAsync(network, (training.X, training.Y), 60, 10, null,
    new TestParameters(test, new Progress<BackpropagationProgressEventArgs>(p =>
    {
        Printf($"Epoch {p.Iteration}, cost: {p.Cost}, accuracy: {p.Accuracy}");
    })));

Thanks for your help!


texts: How can I build an input pipeline for a large dataset in TensorFlow with eager execution?

texts: I've been struggling to get my dataset, a folder of RGB training samples and a folder of corresponding RGB ground truth for semantic segmentation, into TensorFlow. I use TensorFlow 1.13 with eager execution. The documentation recommends the TFRecords file format and demonstrates how this can be easily imported, but doesn't say a thing about how to create such a file. The structure also looks complicated. Alternatively, I suppose I could read the files into memory (using tf.io, PIL or OpenCV perhaps), but the whole dataset won't fit into memory. What is the easiest way to stream my dataset into TensorFlow with eager execution?


texts: how to create word2vec for phrases and then calculate cosine similarity

texts:
I have just started using word2vec and I have no idea how to create vectors (using word2vec) of two lists, each containing set of words and phrases and then how to calculate cosine similarity between these 2 lists.

For example :

list1 =['blogs', 'vmware', 'server', 'virtual', 'oracle update', 'virtualization', 'application','infrastructure', 'management']
list2 = ['microsoft visual studio','desktop virtualization',
'microsoft exchange server','cloud computing','windows server 2008']

Any help would be appreciated.


texts: Training Encoder-Decoder using Decoder Outputs

texts:
I am trying to build an encoder-decoder model for a text style transfer problem. The problem is I don't have parallel data between the two styles so I need to train the model in an unsupervised setting.

Some papers I have seen use an auto-encoder to train the encoder and decoder components separately. By setting the problem as an auto-encoder, they can train the decoder by passing the target sequence (equal to the input sequence) into the decoder. (Here are some examples, https://arxiv.org/pdf/1711.06861.pdf, https://arxiv.org/pdf/1804.04003.pdf)

Instead of an auto-encoder, I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. I would pass the generated output into a classifier to check the style and to obtain a training signal. Is this sensible and what are the pros / cons of doing so? Thanks.


texts: Out of Memory Error when Selecting Data from Redshift Table

texts:
I am selecting data from Amazon Redshift Table with 500 millions rows. I have 64bit python installed.

code

import psycopg2
from sqlalchemy import create_engine
import pandas as pd
engine = create_engine('postgresql://'username':pwd@host/dbname')
data_frame = pd.read_sql_query('SELECT * FROM table_name ;',
engine)

Everytime I run the code I get a "Out of Memory error". I have 16gb Ram.
I am not sure how to resolve this issue.

Would really appreciate any help on this!
Thanks


texts: How to use different classes of words in CountVectorizer()

texts:
Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example

  Text #1 : Comedy 10%, Horror 50%, Romance 1%

  Text #2 : Comedy 40%, Horror 3%, Romance 30%

We have given keywords in each class through which we make a comparison. Below is the code that explains this scenario better

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Comedy
keywords_1 = ['funny', 'amusing', 'humorous', 'hilarious', 'jolly']

# Horror
keywords_2 = ['horror', 'fear', 'shock', 'panic', 'scream']

# Romance
keywords_3 = ['romantic', 'intimate', 'passionate', 'love', 'fond']

text = ('funny hilarious fear passionate')

cv1 = CountVectorizer(vocabulary = keywords_1)
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv2 = CountVectorizer(vocabulary = keywords_2)
data = cv2.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

cv3 = CountVectorizer(vocabulary = keywords_3)
data = cv3.fit_transform([text]).toarray()
vec1 = np.array(data)
vec2 = np.array([[1, 1, 1, 1, 1]])
print(cosine_similarity(vec1, vec2))

The problem with this approach is that vocabulary in CountVectorizer() doesn't consider different word classes (Nouns, Verbs, Adjectives, Adverbs, plurals, etc.) of a word in a text. For example, let's say we have keywords list as below

  keywords_1 =  [(...),
  ('amusement', 'amusements', 'amuse', 'amuses', 'amused', 'amusing'),
  (...),
  ('hilarious', 'hilariously')
  (...)]

and we want to compute similarity as follows

cv1 = CountVectorizer(vocabulary = keywords_1)
data = cv1.fit_transform([text]).toarray()
vec1 = np.array(data) # [[f1, f2, f3, f4, f5]]) # fi is the count of number of keywords matched in a sublist
vec2 = np.array([[n1, n2, n3, n4, n5]]) # ni is the size of sublist
print(cosine_similarity(vec1, vec2))

How can we modify the above code to capture this scenario. Any advice is appreciated.