texts: Any way to implement sequential incremental stochastic gradient descent algorithm in logistic regression?

texts:
I am trying to program incremental stochastic gradient descent (ISGD) algorithm in logistic regression. Initially, I coded respective logistic regression' loss function and its gradient, also got some idea to proceed rest of workflow. But, I have no idea how to apply sequential operation in incremental stochastic gradient descent algorithm which can be used in the respective logistic regression. How can I implement the sequential operation in incremental SGD? Any way to make this happen in Python? How can I do that? Any idea?

Initial implementation

import numpy as np
import scipy as sp
import sklearn as sl
from scipy import special as ss
from  sklearn import datasets

## load input dataset
X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')
X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt',
                                          n_features=X_train.shape[1])

# logistic loss function
def lossFunc(x_i,y_i,w):
    w.resize((w.shape[0],1))
    y_i.resize((y_i.shape[0],1))

    lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y_i* np.dot(x_i,w,))))
    rslt=np.float(lossFnc)
    return rslt

#gradient function

def gradFnc(X,y,w):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    gradF1=-y*np.nan_to_num(ss.expm1(-y))
    gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))
    gradF3=gradF2.resize(gradF2.shape[0],)
    return gradF3

class ISGD:
    def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):
        self.w=None
        self.learnRate=learnRate
        self.verbose=verbose
        self.num_iter=num_iter

    def fitt(self, X,y):
        n,d=X.shape
        self.w=np.zeros(shape=(d,))

        for i in range(self.num_iter):
            print ("\n:", "Iteration:", i)

            grd=gradFnc(self.w, X,y)
            grd.resize((grd.shape[0],1))
            self.w=self.w-grd
            print "Loss:", lossFunc(self.w,X,y)

        return self

def predict(x_i, w):
        y_hat=w[0]
        for idx in range(len(x_i)-1):
            y_hat+=w[i+1]*x_i[idx]
        return 1.0/(1.0+np.nan_to_num(ss.expm1(-y_hat)))

def update_weights(x_i, y_i,w):
    lr=0.8
    yhat=predict(x_i, w)
    error=y_i-yhat
    return w+lr*(y_i-yhat)*x_i

How to proceed rest of workflow?

Here is blog about HogWild! for parallel machine learning. The particular interpretation of incremental SGD can be found here: hogwild! algorithm for logistic regression.

Now I have no idea how to apply sequential operation in incremental SGD which can be used in the respective logistic regression. How can I make this happen? Is there any efficient workaround to implement sequential incremental SGD algorithm for logistic regression? What is the efficient programming pipeline to accomplish the task that I stated above? Any more thoughts?


texts: scikitlearn MLPRegression model conversion to TensorFlow Dnnregressor conversion

texts:
I implement custom Neural Network (prediction part only) based on ScikitLearn MLPRegressor model weights with sklearn.preprocessing.StandardScaler normalization of input features

Can I convert it DNNRegressor of TensorFlow or any other way that would allow me to implement it in python and execute in C#/C++ .
I would like to include sklearn.preprocessing.StandardScaler in the model such that I would be able to serialize it to pb file that would be executed from C++

Thanks
Sasha


texts:
I got some question about the "standard" parameter from a random forest. Following I write my understanding about these parameters. I would be glad if I could confirm my understanding or correct it. :)
For your information, I'm using scikit-learn.

max_depth: Thats the depth of a tree. In case of execution-performance it is good to set a limit. And for the model-performance it is also good, because of overfitting.

max_features: With this parameter I specifie der number of features with which tree is build.
My question: If the parameter is set (for example 0.75), each tree includes 75% of all features, but every tree is build with different 75% feature?

min_samples_leaf: The minimal number of samples per leaf. Is this an important feature and why?

n_estimators: The number of trees. Is there a good default value for this parameter? Here are usually more trees better, right?

Are these the most important parameters or have I forgotten some?


texts: What is a good accuracy level for a chatbot in keras?

texts:
I have build the model and despite the level of accuracy at 0.87 the answers to my questions are not good enough. And somehow I am not getting the 'EOS' tag, making the answers go 20 words and its usually repetition. Is this because the accuracy is not good enough or is it simply a mistake in my code.

Below is the inference model:

context_model = Model(input_context, encoder_states)
# define the inputs for the decoder LSTM
target_h = Input(shape = (300, ))
target_c = Input(shape = (300, ))
decoder_states_inputs = [target_h, target_c]
# the decoder LSTM. Takes in the embedding of the initial word passed as input into the decoder model
(the 'BOS' tag),
# along with the final states of the encoder model, to output the corresponding sequences for 'BOS',
#and the new LSTM states.
target, h, c = LSTM(300, return_state = True, return_sequences = True)(input_tar_embed, initial_state
= decoder_states_inputs)
decoder_states=[h,c]
dec_output = Dense(vocab_size, activation = 'softmax')(target)
target_model = Model(
[input_target] + decoder_states_inputs,
[dec_output] + decoder_states)

 # pass in the question to the encoder LSTM, to get the final encoder states of the encoder LSTM
def str_to_tokens( sentence : str ):
    words = sentence.lower().split()
    tokens_list = list()
    for word in words:
        tokens_list.append(word_to_index[ word ] )
    return sequence.pad_sequences( [tokens_list] , maxlen=20 , padding='post')
# run the inference model
question='who are you'
for _ in range(1):
    states_values = context_model.predict( str_to_tokens(question) )
    empty_target_seq = np.zeros( ( 1 , 20 ) )
    empty_target_seq[0, 0] = word_to_index['BOS']
    stop_condition = False
    decoded_translation = ''
    while not stop_condition :
        dec_outputs , h , c = target_model.predict([ empty_target_seq ] + states_values )
        sampled_word_index = np.argmax( dec_outputs[0, -1, :] )
        sampled_word = None
        for word , index in word_to_index.items() :
            if sampled_word_index == index :
                decoded_translation += ' {}'.format( word )
                sampled_word = word
                print(sampled_word)
        if sampled_word == 'EOS' or len(decoded_translation.split()) > 20:
            stop_condition = True
        empty_target_seq = np.zeros( ( 1 , 20 ) )
        empty_target_seq[ 0 , 0 ] = sampled_word_index
        states_values = [ h , c ]
   print( decoded_translation )


texts: Autoencoder and Neural Network Overfitting in terms of parameter number?

texts:
I have 1100 sequences for 2 classes. Of them 400 are from one class 1 and 700 are from class 2. I have used an one hidden layer auto-encoder of 2 neurons to capture my features. My initial features are tri-grams for each sequences. So, for each sequence I have 6860 tri-grams. As a result most of my input vectors are sparse vectors.

Now, if I calculate the parameters for this network, I have

6860 * 2 = 13720 paramters (1st layer)
2 * 6860 = 13720 parameters (2nd layer)
-----------------------------------------
           27440 parameters (in total)

Now, that is way too many parameters in comparison to my number of data points. Hence, I have used a dropout value of 0.98, on layer 1->hidden layer as well as hidden layer->output layer which makes the number of parameters 13720 * 0.02 =  274 on each layer and in total 548 parameters.

Now, after training, I tried the encoder on my test data of 500 sequences, and extracted the hidden layer of 2 dimensional data. Then I use that data on another 5 neuron single hidden layer neural network to classify. My results are really good in that I am getting around 90% accuracy on my test data which is around the same ball park of the training data accuracy.

My question is am I overfitting in my autoencoder? Am I overfitting by using another neural network? I am worried about my low number of data points. Does my use of dropout seem sensible?

I am worried that I am calculating the parameter numbers wrong, and my network has more parameters than number of data samples even after using Dropout. Does my calculation seem correct?


texts: Are there any web services that can be used to analyse data in social networks with respect to a specific research question (e.g. mentioning of certain products in social media discussions)?


texts: Are SHAP values additive across examples?

texts:
I understand that SHAP values have a property called additivity that means that if you add the SHAP value of each explanatory variable of a particular example to the average prediction of the model on the dataset you obtain the prediction for that particular example.
I know that SHAP values are LOCAL explanations, therefore they do not exactly apply to multiple examples, so I assume the answer is no, they are not additive across examples.
The reason I am interested in this question is that I want to add up the SHAP values of several misclassified examples, does anyone has any insight on why the output may or may not be useful to explain what is the model doing with those examples?


texts: Find the column(s) name where the value of the variable matches a regex

texts:
I am trying to filter a data frame where I need to search for a regular expression within the data frame.

dim(df)
[1] 10299 561

I wanted to know if there is a way, I could find the name(s) of the column(s) where a particular regex may be available.

As an example, how do I find the name of the column where 'rsi' is present. I am referring to iris.


texts: How to interpret the Mean squared error value in a regression model?

texts:
I'm working on a simple linear regression model to predict 'Label' based on 'feature'. The two variables seems to be highly correlate corr=0.99. After splitting the data sample for to training and testing sets. I make predictions and evaluate the model.
metrics.mean_squared_error(Label_test,Label_Predicted) = 99.17777494521019
metrics.r2_score(Label_test,Label_Predicted) = 0.9909449021176512

Based on the r2_score my model is performing perfectly. 1 being the highest possible value. But when it comes to the mean squared error, I don't know if it shows that my model is performing well or not.

How can I interpret MSE here ?

If I had multiple algorithms and the same data sets, after computing MSE or RMSE for all models, how can I tell which one is better in describing the data ?

R2
score is 0.99, is this suspicious ?  Or expected since the label and
feature are highly correlated?
      Feature        Label
0   56171.757812    56180.234375
1   56352.500000    56363.476562
2   56312.539062    56310.859375
3   56432.539062    56437.460938
4   56190.859375    56199.882812
...     ...     ...
 24897  56476.484375    56470.742188
 24898  56432.148438    56432.968750
 24899  56410.312500    56428.437500
 24900  56541.093750    56541.015625
 24901  56491.289062    56499.843750


texts:
I am trying to understand the key difference between GAN and DCGAN.

I know that DCGAN uses a convolutional network. But:

What data is better to push into GAN and what data fits better to DCGAN?
Does DCGAN work better with small data dimensions?


texts: Problem with sort by and group by in pandas

texts:
I have a DataFrame with duplicate id's but have different dates.

For Example:

A          B
1    12.09.2015
1    14.08.2018
1    13.07.2012
2    01.09.2015
2    09.10.2011

I want to group them by the Id's and get each of their maximum and minimum dates and create a different dataframe with each line for one id like:

A  B(min)  B(max)


texts: How to find similar points to a positive set when you don't have any negative set?

texts:
The task I'm used to do is the following. A client comes to see me with a set of clients (called positive companies) and he wants me to find other similar prospects. Usually, he also gives me a set of negatives companies and I have a big set of potential companies (that I call the basket).

I perform this task by doing a Adaboost classifier that I train with the positives and negatives. I then run this classifier on the basket. Each company in the basket receives a score and the highest score shows the most promising prospects for the client.

Now, a new client doesn't have any set of negatives to give and I'm a little bit lost. I can not do a supervised learning anymore, obviously. I first thought of performing a k-nearest neighbours on each positive and I would receive a list of "close" prospects. The problem with that is that I don't have a score anymore.
Furthermore, with the k-nearest method, I should define a distance which I don't like because I don't want to give subjective weights to features. Indeed, the Adaboost classifier would learn some weights and would itself predict which features are important.

Could someone indicate me how I could tackle this problem?


texts: Correlation Matrix for non-numeric features

texts:
Currently, I have dataset with numeric as well non-numeric attributes. I am trying to remove the redundant features  in the dataset using R Programming Languages. Note: Non-numeric attributes cannot be turned into binary.

The Caret R package provides the findCorrelation which will analyze a correlation matrix of your data’s attributes report on attributes that can be removed. However, It only works numeric values of 'x'. I have been unable to find a package which does it for non-numeric attributes.

Is there a function in Caret R Package that does that for non-numeric attributes as well? If not, any method/package that would help me achieve the same?


texts:
Is there a way to retrieve the groups of features that, jointly, show a high loading in each LV. I'm aware that I can retrieve this by digging into x_loadings_, but given that covariance within and between groups is maximized, I'm wondering if there is a way to retrieve the combinations of features rather than individual features?

TL;DR: How to get strongly correlated features? Can we define these by their fitting coefficients?

This is how I'm performing the PLS-DA

from sklearn.cross_decomposition import PLSRegression
plsr=PLSRegression(n_components=2, scale=True)
plsr_fit=plsr.fit(input_data, y)

#x scores
plsr_fit.x_scores_

#x loadings
plrs_fit.x_loadings_

#x weights
plrs_fit.x_weights_


texts: Geometric and harmonic means in ensembling methods

texts: When using ensembling methods for regression, a common approach is to average (using the arithmetic mean) the outputs of the weak learners in order to obtain the output of the ensemble. Is there a reason why the average that people take is the arithmetic mean? Why not performing a geometric or harmonic mean?


texts: Understanding experiments in Continual Learning

texts:
Via paper Continual Learning Through Synaptic Intelligence, I see this figure for Split MNIST benchmark, but there is a point I can get.

Here there are 5 tasks, and finally we summarize the average accuracy over the 5 tasks.

Here, how the tasks are performed. Does they perform sequentially when first we learn how to categorize 0 and 1, then in the next task we expect that the model can also categorize 2 and 3, 4 and 5 and so on.

And here another question is, in the horizontal axes of each graph, there are 5 tasks, why we perform, for example Task 1 (0 and 1) on 5 tasks. Could someone clarify this point for me?


texts: Smaller test data set than training data set in machine learning

texts:
I would like to train different machine learning algorithms (SVM, Random Forest, CNN etc.) for the same data set (e.g. MNIST) und then compare their accuracies.
The goal would be to find out from which training data size which method is preferable to the others.
To do this I continuously reduce the original training data set (of 60000 samples) and train the models on these reduced trainings data sets.

If I then determine the accuracy using the original MNIST-test dataset (10000 samples), of course I will get overfitting, e.g. with a training data set of 1000 samples I get a training accuracy of 95% and test accuracy of 75%.

The smaller the training data set, the lower the test accuracy, while the training accuracy remains at about the same level.

Would it make sense also to reduce the test data set to restore the original 1:6 ratio of the test set : training set?
Personally, I think that does not make sense.
Or have I thought incorrectly about that?


texts: Matplotlib: plot indicating the precision of measurements

texts:
I have some data with some margin of error and am using numpy arrays to plot that data. Assuming the precision is, let's say, 0.05 or 0.03grams, then the data will be something like

[4.0 $\pm$ 0.05, 3.05 $\pm$ 0.03]

How can I show this on my plot (either on the x axis or the y axis. If it's both, even better)?

I use pyplot and my OS is macOS


texts: Autoregressive (AR) models constants - Time Series Analysis

texts:
I'm currently struggling with different Model like AR or MA.

If AR(1) is expressed as: $y_t = \beta + \beta_t \times y_{t-1} + \epsilon_t $

How do I know what the $\beta$ 's would be? What are the dependencies?

I think a simple example would help a lot.


texts: How to create pretrained word embedding text file with additional word features

texts: I've had an idea for using word features to improve the quality of neural machine translation. Now, I would like to create word embeddings with additional word features such as pos tag, named entity, lemma, etc. I used gensim to create pretrained word embeddings and pretrained tag embeddings. However, I still do not understand how to deal with it.  Are there any ways to implement this idea? Should I annotated pos tag on a training data file and train a new word embedding model with it?


texts: Where did this NY Times op-ed get his Google Search data?

texts:
I hope this is a question appropriate for SO.
The article in question: http://www.nytimes.com/2015/01/25/opinion/sunday/seth-stephens-davidowitz-searching-for-sex.html
As far as I can tell, the only publicly available data from Google Search is through their Trends API.  The help page states that

The numbers on the graph reflect how many searches have been done for a particular term, relative to the total number of searches done on Google over time. They don't represent absolute search volume numbers, because the data is normalized and presented on a scale from 0-100.

However in the article, the author reports (absolute) "average monthly searches".  The source is stated as:

All monthly search numbers are approximate and derived from anonymous and aggregate web activity.
Source: analysis of Google data by (author)

So, how did he get this "anonymous and aggregate web activity"?


texts:
The criterion for firing a perceptron is as follows

Why is it that when the function $w \cdot x + b = 0$ the output is zero as well. Why couldn't it have been set to 1?

If one were to simulate the behavior of the perceptron with a sigmoid function, $w \cdot x + b$ can be multiplied by any arbitrary constant $C>0$ (as $C$ tends to infinity) as long as $w \cdot x+b$ is not equal to 0. This is because at 0 condition, the firing always gives output as 1 instead of zero it should have giving. However, changing the firing rule overcomes this problem. Then, is there any reason why 0 should be the output when $w \cdot x+b= 0$?


texts: Masking numbers in plotly time series plot

texts:
I'm interested in masking all non-zero values in a plotly time series plot.

Is there a way to simply not include the zeroes when rendering the plot, and connect all data points greater than zero?

I tried limiting the range in the layout, but this did not change what was plotted:

import plotly.graph_objs as go
import plotly.plotly as py

data = [go.Scatter(
          x=data['timestamp'],
          y=data['units_sold'])]

layout = go.Layout(
    yaxis=dict(
        range=[2, 15]))

fig = go.Figure(data=data, layout=layout)

py.iplot(data)

Here is the general structure of my code as a Pandas dataframe:

import pandas as pd

d = {'timeseries' : [801, 802, 803, 804, 805],
         'units_sold' : [0, 0, 1, 3, 0]}

pd.DataFrame(d)


texts: How can you decide the window size on a pooling layer?

texts:
On the convolutional neural network, there used one or more pooling layers. As far as I know many tutorials instruct you to set it either 2 or 3 for the window size. For example, in this tutorial:

Pooling Layers
After some ReLU layers, programmers may choose to apply a pooling layer. It is also referred to as a downsampling layer. In this category, there are also several layer options, with maxpooling being the most popular. This basically takes a filter (normally of size 2x2) and a stride of the same length. It then applies it to the input volume and outputs the maximum number in every subregion that the filter convolves around.

I make the relevant part bold. But why is it usually set to 2 (or in some cases 3)? How can I know what size is appropriate? Should I just tweak the parameter via a trial and error like a brute-force way, or is there any tips on deciding the window size?


texts: How do the policy gradient's cost function and gradients work?

texts:
I am not a math expert but have a basic understanding of linear algebra, calculus and probability and I understand the math behind back propagation. Currently I am trying to learn about policy gradient algorithm, but I am having difficulty understanding the underlying math. The most commonly used cost functions in the neural network training includes a target and output. For example:
MSE:
$$E_{total} = \sum{\frac{1}{2}(target - output)^2}$$
Log Loss:
$$Error = Output(i) * (1 - Output(i)) * (Target(i) - Output(i))$$
The idea is to find parameters $\theta$ which reduces the distance between the target and the output.
But in policy gradient method the cost function is like this:
$$g = \mathbb E\Big[\sum R_t*\frac{(\partial)} {(\partial\theta)}ln\pi_\theta(a_t|s_t)\Big]$$
What is the target and output in policy gradient's cost function?
How does this cost function get minimized and how does it works?


texts: t-SNE plotting DBSCAN clustering results very scattered issue

texts:
We are trying a DBSCAN clustering model on our 30,000 samples with 15 features each. We tuned the epsilon parameter small enough to make sure the radius of the clustering circle is small while it does clustering. We expected to have small clusters with high density. After clustering and parameters tuning, we used t-SNE to plot the clustering results in 2 dimensional space, we found that we have small clusters like cluster 2,3,4,5 with high density as expected while large clusters like cluster 0,1 scattered loosely as unexpected. obviously, cluster 0, 1 looks very strange to us.

Is there something we are not aware of, is it something wrong with clustering algorithm?

Or it's something about t-SNE, when it plots the clustering results in lower dimension, it might look like cluster 0, 1 are very scattered, actually they are more close and tensed in higher dimensional space?

cluster 0: orange
cluster 1: blue
cluster 2: red
cluster 3: green
cluster 4: light yellow
cluster 5: purple

  30,000 samples, 6 clusters case:

  30,000 samples, 8 clusters case:


texts: How to put marker on time series training set

texts:
My input is this picture
And I would like to put markers on it and use time series with markers as a label

Two picture are not scaled. Main point is I would like to train RNN classifier and let it be able to mark the starting points and stopping points.
More generalized examples

Question
What is the best function to be marker encoding function?


texts: Are there any reasons for using Huber over the Pseudo Huber Loss?

texts:
The Huber Loss is:  $$ huber =
     \begin{cases}
       \frac{1}{2}  t^2 & \quad\text{if}\quad |t|\le \beta \\
       \beta |t| -\frac{\beta^2}{2} &\quad\text{else}
     \end{cases} $$
The pseudo huber is:
$$ pseudo = \delta^2\left(\sqrt{1+\left(\frac{t}{\delta}\right)^2}-1\right)$$

What are the pros and cons of using pseudo huber over huber? I don't really see much research using pseudo huber, so I wonder why?
For me, pseudo huber loss allows you to control the smoothness and therefore you can specifically decide how much you penalise outliers by, whereas huber loss is either MSE or MAE. Also, the huber loss does not have a continuous second derivative.
So, what exactly are the cons of pseudo, if any?


texts: SHAP for Deep Neural Network taking long time

texts: I have 60,000 samples with each having 1,800 features. I have made a multilayer perceptron in Keras and I want to use SHAP values to arrive at global feature importance. Is the matrix too big for SHAP? Is there any way to run SHAP in parallel or make it faster? because currently, it takes a very long time. What other methods do you recommend in case SHAP is impossible (LIME or Deepexplain or DeepLIFT?


texts: Statistical methods for Sequence learning

texts:
I am bit more traditional and I am looking for a statistical method that can help me also do inference or predictions.

I have some table that contains some transitions, where each row of the table correspond to a specific user. If you are into python this is it:

smallData=pd.DataFrame(np.array([[1,2,3,4,5],[4,5,6,7,8],[1,2,3,3,3],[1,2,2,3,3],[1,2,3,5,3]]),columns=range(0,5))
smallData
Out[266]:
   0  1  2  3  4
0  1  2  3  4  5
1  4  5  6  7  8
2  1  2  3  3  3
3  1  2  2  3  3
4  1  2  3  5  3

Now first row means that the following transitions happen
1-2-3-4-5

this kind of transitions can be of course be visualized as a tree or diagram. So far I have not found any method or library in python or R that can calculate the statistics of transitions. For example for the two transitions [1,2] three transition can happen

Transition 3 by 50%,
Transition 4 by 25% and
Transition 5 by 25%.

These were calculated from the table above.

Even harder will be to predict two steps forward.

I have seen the topic of sequence prediction to be discussed a lot in the machine learning community by using LSTM. I still wonder if in the statistics community are methods to handle such problems and if yes if there are in python, R libraries that can be used for such problem.


texts:
In RBF (radial basis functions)-Neural Networks, which method (Levenberg–Marquardt or Adam optimization) is more efficient for optimizing the parameters (centers, widths, and output weights) in PyTorch? why?
In what condition Levenberg–Marquardt works better?


texts: Quantifying feature importances using Auto-encoders

texts:
I have a set of features(mixture of numerical and categorical), each of size n. I am embedding them into a dense lower dimensionality space using an auto-encoder. I want to know if it is possible to get the feature importances.

One vague idea that I can think of this :
Assume the Weight matrix for the 1st layer(the layer immediately next to the input layer, of size p) is W(shape : n x p). This is the weight matrix used in the affine transformation dot(W.T, x) + b, x being the input. We can sum over the rows of W and then normalize it to get the feature importances of different features.

But again, this is quite ad-hoc. Can someone refer me to some paper where the authors have explored this, or maybe better some implementation of this. Thanks in advance.


texts: How important is it for each row of data to have the same number of features?

texts:
I'm using decision tree learning to try and classify a device based its components. Different devices have a different number of components and the location of these components within the device is important.

Device 1 might have components 9, 3, 8, 4, and 1 in that order.
Then device 2 might have components 4, 3, 2, 6, and 7 in that order.

The issue with this is I'm finding it difficult to convert these into features.

I would have the features of each row be the locations and the value be the name of the component in that location but with some devices having few (<10) components and some with many more (>60) this could lead to many blank values.

I'm worried that when training if most of the training data is on devices with for example < 40 components, when trying to classify a device with > 40 components I'm assuming it would go terribly wrong, but you also can't just use devices with more than 40 components because then it will never learn to work for simple devices with smaller numbers of components.

If there is a technique or way to get around this please let me know! Or if I've come at this problem the wrong way and need to rethink let me know that as well!


texts: Best classification technique for following kind of data set

texts:
I have a large table where each record or row represents a single salesperson, and there are 50 columns or dimensions where each column represents one of 50 products potentially sold by any given salesperson, with one final column representing their total compensation as a percentile of their salesperson peers.
The values within each column range from 0 to 100, reflective of the salesperson's percentile performance in sales for that product, and then in the final column, percentile in total compensation.

e.g.

prodA | prodB | prodC | prodD | ... | prodAW | prodAX | comp

0 | 93 | 0 | 87 | ... | 73 | 0 | 88

100 | 0 | 44 | 0 | ... | 99 | 63 | 67

88 | 91 | 0 | 89 | ... | 85 | 88 | 24

...

assume there are at least 50K records (i.e. 50K salespeople)

If the value in any of the 50 columns equals 0, it means the salesperson didn't sell any of the given product. Using the last row in the example table above
that salesperson may have been 88th percentile in terms of sales of product A, and 91st percentile in sales of product B, they are only 24th percentile in terms of total salesperson compensation (see last column).
What approach would be best to categorize and classify natural groupings of salespeople?  Would I use something like a k-means clustering?

TL;DR: What is the best approach for classifying/clustering/categorizing salespeople into appropriate groups?


texts: The Merits of Feature Reduction Routines

texts:
I am interested in learning what routine others use (if any) for Feature Reduction/Selection.
For example, If my data has several thousand features, I typically try {2,3,4} things right away depending on circumstances.

Zero variance/Near zero variance

Using R package caret, nzv
I find a v.small percentage is zero variance and a few more are near zero variance.
Then by using nzv$PercentUnique I may remove the bottom quartile of features depending on the range of PercentUnique's.

Correlation to find multicollinearity

I find the correlation matrix and remove values > 0.75 and remove.
I have seen others use correlations > 0.5 or 0.6, but don't have any references for it.

Boruta / Random Forest

Love Boruta package but it takes a while.
Then here again use Forward Feature Selection.

PCA

Depending on the nature of the data I will try PCA last.
If the model must be explainable then I skip this.
I may use several criteria: 80, 90, 95% error explained
Forward Feature selection, look for first ~3:10 orthogonal features

NOTE: I am not suggesting this is the best/worst routine but I'm opening the floor to civil debate. If you need a definition for Civil Debate see Wikipedia.


texts:
Yann LeCun mentioned in his AMA that he considers having a PhD very important in order to get a job at a top company.

I have a masters in statistics and my undergrad was in economics and applied math, but I am now looking into ML PhD programs. Most programs say there are no absolutely necessary CS courses; however I tend to think most accepted students have at least a very strong CS background. I am currently working as a data scientist/statistician but my company will pay for courses. Should I take some intro software engineering courses at my local University to make myself a stronger candidate? What other advice you have for someone applying to PhD programs from outside the CS field?

edit: I have taken a few MOOCs (Machine Learning, Recommender Systems, NLP) and code R/python on a daily basis. I have a lot of coding experience with statistical languages and implement ML algorithms daily. I am more concerned with things that I can put on applications.


texts: What are the major differences between Facebook's ELF RF framework and TensorFlows "Agents"?

texts:
I recently read a few papers, especially

ELF: An Extensive, Lightweight, and Flexible
Research Platform for Real-time Strategy Games
TensorFlow Agents:  Efficient Batched Reinforcement Learning in TensorFlow

As far as I can understand, ELF uses two separate environments (C++ and Python), because they say

  Most existing RL environments (OpenAI Gym, ...) provide Python interfaces which wrap only single game instances.  As a result, parallelism needs to be built in Python when applying modern RL methods. However, thread-level parallelism in Python can only poorly utilize multi-core processors, due to the Global Interpreter Lock (GIL). Process-level parallelism will also introduce extra data exchange overhead between processes and increase complexity to framework design.  In contrast, our parallelism is achieved with C++ threads for better scaling on multi-core CPUs.

The team from DeepMind (which developed the TensorFlow Agent lib) on the other hand explain it as such:

  Our implementation is completely defined within the TensorFlow graph.
  First, we parallelize the forward pass of the neural network by vectorizing the
  agent computation to produce a batch of actions from a batch of observations. [...] Using a batch size during inference allows us to leverage the internal
  thread pool of the TensorFlow session, or hardware accelerators such as GPUs and
  TPUs.

So, how are those two different? Sure, ELF uses C++, because they say doing it in Python is tedious and not efficient. TF agents uses "vectorization" and runs everything in the TensorFlow graph. I know, TF uses C underneath and a lot of Protobuffers to communicate data between Python & the underlying C code.

So did FB with their ELF just reinvent the wheel or did they do something else that is clever and the TF team didn't adopt?


texts: Is it possible to pass multiple features at once to Croston method?

texts:
I want to implement the Croston method for intermittent demand. I have a data frame that has 10 features and all those have many zeros in them. I want to pass the entire data frame to the Croston model but the model accepts a one-dimensional array. I'm not interested in looping through the model. Is there any way or any other methods that can give forecasts for intermittent demand?.
Thanks in Advance!!


texts:
This is a problem that I keep having when trying to do a boxplot in R. From what I can see the code I am using should work, not only that but when compared to my notes and other classmates it appears to be the same. What am I doing wrong?

result <- read.csv("keough.csv", header=TRUE, sep=";", dec= ",")
head(result)

 BIOFILM.SERP
1        SL,61
2       SL,113
3       SL,123
4        SL,75
5        SL,75
6        SL,83

attach(result)
boxplot(BIOFILM~SERP, dat=result)

  Error in eval(predvars, data, env) : object 'BIOFILM' not found


texts: Understanding the network structure of a multiple timeseries fusion model

texts:
Please don't mark this question as duplicated to
Can I create a layer with multiple rnn cell ? [question about a paper]
It has already been marked 2 times , I admit they do refer to a same paper, but are totally different questions , please look carefully ! Above question is also from me which already be solved by myself , why I need ask another same quesion I have solved ?

I want to implement this paper: https://dl.acm.org/doi/pdf/10.1145/3269206.3271794
It has structure graph :

But I can't understand how to generate Multiple Resolution Tensor R .
I understand all the steps except the detail of step 3:

we have 3 time series when α, β ∈ {day : 1,week : 7} ,  so dr = 2

we can generate dr*(dr +1)/2 unique time series with different configurations of {⟨α, β⟩|α ∈ α, β ∈ β, α ≥ β} to represent multiple periodic time series distributions, in which dr = min(|α |, |β|).

we need 3 gru/lstm to generate 3 hidden state with ds length

Generate Multiple Resolution Tensor R

Inspired by these recent advances, we propose a convolutional fusion
framework to summarize a multi-resolution tensor $ R ∈ R^{|α |× |β |×ds} $ into a conclusive representation. Here we use R to represent a
collection of time-evolving patterns generated from multiple time
resolutions. Namely, $ R_{i,j,1:ds} $ denotes a learned sequence
pattern representation $ h_t^{αi, βj} $ at time t w.r.t. temporal
resolution αi and interval resolution βj as described in Eq. 2. We
further apply a mirror padding along the diagonal to complete the
tensor. The generation process of R is shown in Figure 1.(b).

Apply CONV2D
Apply CONV2D
Apply FC

The ambiguous part is step 3 ! :

For me,  $ R ∈ R^{|α |× |β |×ds} $  shape is  [dr*(dr +1)/2, ds] , so I have to add one dim to [dr*(dr +1)/2, ds, 1] to make it can be pass to conv2d , is this right ?
But the Figure 1 write the ds (gru hidden state length) as channels ,  and it also draw the channels size has to be divide by 2  in every conv2d layer .

What is the correct way to generate Multiple Resolution Tensor R ?
PS
I wrote a dummy model, you could reuse this :
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

data = pd.DataFrame(np.random.uniform(size=(1000,3)), columns=['Sales', 'SalesDiff7', 'SalesAggMean7'])

multi_inputs = []
multi_outputs = []
window_size = k

for i in range(data.shape[1]):
    ti = keras.Input(shape=(window_size, 1), name=f't{i}')
    tlstm = layers.LSTM(32)(ti)
    multi_inputs.append(ti)
    multi_outputs.append(tlstm)

r = tf.stack(multi_outputs, axis=-2)
r = tf.expand_dims(r, -1)
conv1 = tf.keras.layers.Conv2D(filters=16,
                       kernel_size=(2, 2), #padding='same',
                       activation='relu')(r)
conv2 = tf.keras.layers.Conv2D(filters=8,
                       kernel_size=(2, 2), #padding='same',
                       activation='relu')(conv1)
fc = tf.keras.layers.Flatten()(conv1)
result = keras.layers.Dense(units=1)(fc)


texts: How to understand the performance of different machine learning models?

texts:
I have a dataset, which contains the processing conditions (i.e., 42 features) and the property (i.e., 1 target) of a class of material. To know the performance of different machine learning models, I tested five different machine learning models by considering different numbers of features in training. These models are linear regression (LR), Bayesian ridge (BR) , k-nearest neighbor (NN), random forest (RF), and support vector machines (SVM) regression. The coefficient of determination (R2) for the test dataset is used to represent the performance of trained machine learning models.

As we can see that the max. accuracy of these models are in order: RF>BR~LR~SVM>NN. The top 8 features are required to obtain good accuracy for RF and afterward the accuracy is almost independent of the number of top-ranking features. The performance of BR, LR and SVM continuously improve with an increasing number of top-ranking features until reaching the maximum with top 26 features. NN exhibits different trends from others. It already reaches the best performance with top 8-12 features but gets worse and worse with more features.
I am wondering what are the possible reasons for this results? Some directions or some hints? for example:

why the accuracy is in the order of RF>BR~LR~SVM>NN.
why RF model reaches a good accuracy with a few features then keeps almost constant with more features?
why linear-based models BR and LR have very similar performance as the SVM models.
why NN model reaches a good accuracy with a few features then the accuracy reduces with increasing numbers of features?

I understand the reason is from case to case, but what is the general explanation or directions for finding the answers?


texts: Is there any way to calculate a relevance score between a title and the content of a text?

texts: My question might sound a little bit stupid but I am trying to come up with a way to measure the relevance of the title of a text, let's say a piece of news headline, to the content. My idea would be to work with word2vec and have a go with cosine similarity. Are there any more efficient and proper ways to work with this task?


texts:
I have done linear and multivariate regression so I understand what probability, cost and gradient descent functions are. I do not understand what the last 3 formulas mean and how they relate to each other. I don't expect anyone to explain them to me but could someone please share a link that explains them in simple terms? I've tried looking online but the formulas used often look different from the ones below.

Sorry for posting a picture like this but I don't know how to format formulas.


texts: Adding dense layer to CNN causes to stop learning

texts:
I have 6 layer CNN model that (kind of) works with "GlobalAveragePooling2D()" in the end but instead if I flatten and add dense layers like below, it just does not learn. After 20 epochs loss and accuracy curves still doesn't change at all.

Do you think anything wrong the way I flatten or add dense layers in the end? I also printed out the shapes incase it helps.

model.add(Ativation('relu'))
model.add(BatchNormalization())
print("Conv6 after shape",model.output_shape)

model.add(Convolution2D(filters=64, kernel_size=(4,4),
                    strides=(2,2),
                    padding="same",
                    kernel_regularizer=l2(0.0001),
                    kernel_initializer="normal"))
model.add(BatchNormalization())
model.add(Activation('relu'))

print("before flat shape",model.output_shape)
model.add(Flatten())
model.add(Dropout(0.5))

print("after flat shape",model.output_shape)
model.add(Dense(200, activation='relu'))
print("after dense shape",model.output_shape)
model.add(Dense(2, activation='softmax'))

And here is output of full model:

0 shape (None, 1, 44100, 40)
1 shape (None, 1, 44100, 40)
2 shape (None, 1, 160, 40)
Conv3 after shape (None, 40, 160, 24)
Conv4 after shape (None, 40, 160, 24)
Conv5 after shape (None, 20, 80, 48)
Conv6 after shape (None, 10, 40, 48)
before flat shape (None, 5, 20, 64)
after flat shape (None, 6400)
after dense shape (None, 200)


texts:
This is a regression problem. I have a dataset of sales of various products overtime.  I have three kind of feature sets : Price features, Product features and Seasonality.
I want to build a customer estimator which is defined as follows : y = a*price_features + RandomForest(Product features + Seasonality features).
Moreover if there would be a possibility to switch RandomForest with some other non linear estimator it would be even better.
So the model is linear for price features and non linear for product and seasonality features. How can i do something like this in python ? I am interpreting this problem as GAMs because I express the model as a sum of Linear and Nonlinear components. I know that GAMs are fitted using backfitting but i dont know if I should take the pain of writing the whole backfitting algorithm. Also i am pretty sure there isnt a package which provides the functionality to use RandomForestas a GAM. I would really appreciate some insights. Thanks


texts: Predicting crowd density using real time streaming data

texts:
I would like to make predictions about how crowded a location (postal code mapped to lat/lng coords) would be. I would like to provide predictions to questions like: what’s the crowd density going to be like for the area surrounding postal code 12345 for 50metre next Monday between 6pm to 8pm?

For a start I’m working with historical transaction data (as a proxy for human activity) which contain time stamps and postal codes. I would also like to incorporate new transaction data into my model as they occur.

Does this make sense?

Filter the historical transactions based on requested time (between 6pm to 8pm).
Cluster the transactions based on the desired proximity to get a daily transaction totals for the area.
Use the daily totals as training data for my regression model.
Use the model to predict

Would I need to rerun the entire process to account for new data which arrives in real time (e.g every 5mins)? Are there any incremental ML models which could be suitable?


texts: Computation of kernel matrix using radial basis kernel in svm

texts:
I want to compute a kernel matrix using RBF on my own. The training data is multidimensional. My query is whether we will apply
$$e^{-\gamma(x-y)^2}$$
for each dimension and then sum the values across all dimension?


texts: How to apply multiple clustering algorithms to same dataset and make comparison?

texts: I've a dataset and I want to implement K-Means, Fuzzy C Means, Gaussian Mixture Model, Spectral Graph. After that, I want to see the clusters that I get from different methods. What is the proper way to do that? Or should I only stick one algorithm and try to maximize correctness of that clustering?


texts:
I've been asked to report the number of parameters to be learned in a PCA model. This answer implies that parameters do exist in PCA, but does not explain. Software packages often report the number of parameters, but do not document what those parameters are.
What are the parameters in a PCA model? Subsidiary question: how many parameters are there?
To be clear - I am not asking about model hyperparameters.


texts:
I have X values such as

[0.2, 0.1, 0.3, 0.5, 20, -0.2, -0.1, -0.6, -0.8, -30]

Now I know that if the abs value is below a certain threshold, let's say 2, the y-value (-1 or 1) is pretty much random. But when the x-value is above that threshold, there is a good linear/non-linear relationship between X and y (so high positive numbers such as 20 would more often map to y-value of 1 and negative numbers such as -30 in this case would more often map to -1).

What is best practice to account for that? I'm using Keras/Tensorflow, but I'm rather looking for a general answer.

I see two options:

Find out that threshold somehow and train/test the model only on values above that threshold (but this means I lose data)
Generate probabilities in the prediction (e.g. for x-values that are not significant to the output, the output layer generates values close to 0.5, but for more significant values, the output would be closer towards 0 or 1)

What is the best practice to approach this?


texts: Starting search from rightmost column and moving left pandas

texts:
I'm very new to pandas and I'm trying to implement the code to search for something on an excel spreadsheet.

Basically, the most recent price can be found on the furthest right column named "final cost", and that's what I need. However, there are multiple columns named "final cost" (as every invoice is posted on there).

What I currently have:

def find_most_recent(row_num: int, df) -> float:
    k = -1
    checker = False
    while not checker:
        if df.columns[k] == "final cost":
            if not pd.isnull(df.iloc[row_num, k]):
                checker = True
                return df.loc[row_num, df.columns[k]]
        k -= 1

where row_num is the current row that is being searched, and df is the current dataframe being searched.

However, this gets only the first encounter of "final cost". For example, on a dataset of:

I want the return to be one of the prices in column L, but I'm getting returns of values from column F instead.

Any suggestions of a method to use?


texts: Generating a set of different scenarios based on some initial observations

texts:
I have a in my hands 3 different time series which model 3 different scenarios (base, downside, upside). Every of this time-series depends on a set of 11 different attributes, which take values for different time intervals. Most of the different features of the input are highly correlated. There is also a (cdf) probability function which defines how probably every scenario is (every quintile), for every point in time.
In my case, I want to create more input data based on the current observations and create different time-series/ simulations. For example, let’s take the case of the base scenario. The first idea I had was to calculate the Covariance matrix and the mean value of the different data points through time and then just draw samples from a multivariate normal distribution. But this obviously is not correct because even if the correlation between the different attributes is preserved, some of the properties of the time series are not. If I just draw random points and assign them at the different time intervals, some of these attributes can become wiggle. And for example, if one attribute is kind of similar with let’s say ‘GDP’ then it does not make sense to fluctuate for short-periods of time.

At this point I have think couple of ways how to deal with this problem but I have not come up with a complete analytical solution. Someone else had attempted to take a look before. He just basically defined normal distributions for each point in time using the values of the attributes for the 3 scenarios and the cdf. For example if the value of an attribute is [1.25, 1.5, 2] for every scenario respectively and for some values of the cdf, these points can be used in order to create a distribution and then sample different points. But, with this way all the new times series have the form of the base one (same fluctuation)  and they are concentrated close to this series.

At this point I mostly interested in a simple approach on this problem that will give some reasonable results. It would be great if you could give some advice or any kind of reference to look for.


texts: Constructing the Confusion matrix from given metrics

texts:
I am given the following metrics for a certain classifier :
 -Total number of cases in the dataset = 110
 -Accuracy: 92.7%
 -Precision : 96.9%
 -Recall : 95%

Are this information enough to reconstruct the confusion matrix?


texts: sentence patterns/rules/formulas for imperative sentences …

texts:
Please let me ask if you know any ressources on imperative sentence patterns for natural language generation.

There’s probably some ressources online, but I did not really find any.

Imperative sentences, or command sentences, could be structured very simple, e.g.:

1) “Eat!” (intransitive action-verb + exclamation mark)

2) “Eat rice!” (transitive action-verb + noun (singular or plural in some cases) + exclamation mark)

3) “Eat healthy!” (intransitive action-verb + adverb)

… but the more words per sentence, the more complex it gets.

So maybe there’s some general patterns/formulas for creating different variations of such sentences in a grammatically correct sense.

EDIT, as it seems the question is unclear:

I want to generate imperative sentences, i.e. instructional sentences. Is there any pattern on how to create such sentences with NLG?


texts:
I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled datasets for this task.  So far the best that I've seen is the New York Times Dataset which I can't use due to licensing constraints. I've also seen the 20News Dataset but it only has twenty categories so it probably won't scale well to other domains.

Are there any other good datasets out there that I'm missing that can be used for topic modeling?  I'm happy to use a dataset that isn't explicitly meant for topic modeling; as long as it has some sentences/paragraphs that are tagged or labeled that should be fine.


texts: Every one knows data-driven modeling, but what is model-driven (or non data-driven) modeling?

texts: There are hundreds of data-driven machine learning models. It is easy to name a few: neural networks, linear regression, SVM, etc etc... but what is model-driven (or non data-driven) modelling and what are famous and useable examples for e.g. regression tasks?


texts:
I'm looking for a pre-trained net recognizing vehicles, something like the Inception network for images.

If that is impossible to find which vehicle detection algorithm would you suggest, possibly implemented in tensorflow?


texts: Keras Applications - using images larger than the default size

texts: I would like to use eg Xception network with default input size 299x299, but my images are 450x600. Are there any other options besides cropping and subsampling ?


texts: Which model for a content suggestion in which only some of the items are currently available

texts:
I am trying to understand the type of model that would be used in a content suggestion scenario where not all of the choices are available at a given time.  For example, when an online movie subscription service suggests a show it does not need to worry about if the content is available because it's unlimited.  A storefront renting DVD's however, does.  My problem is like the storefront where only a certain few of the content items is available at a given time.
For the environment, let there be 1000 pieces of content in total, 100 of which are available at a point in time.
I am considering the pros and cons of approaching this in two way:

use a model where I can ask for a probability for all 1000 pieces of content in one call for the current environment, then check which ones are available.

check which items are available, ask the model for the probability of each.

Both of these seem inefficient. Performance is certainly a consideration as this model will live in an environment where it needs to be very fast.
Are there any models which can account for some of the choices not being currently available?


texts: K-Fold Crossvalidation in Tensorflow when using flow_from_directory for image recognition

texts:
Disclaimer: I have very little experience with Tensorflow.

I have a custom dataset with 20 categories with 100+ images in each. I am doing 5-fold cross validation using InceptionV3 for transfer learning. The easiest way to load this dataset into Tensorflow that I was able to find was flow_from_directory. The method works for one fold, but not for 5 folds since you can't set the folds. How would I go about dividing up the generators into 5 folds? Should I use an alternative method of importing data instead of flow_from_directory? There was a similar question where the answer was seemingly just importing it in a different way.

from tensorflow.keras.preprocessing.image import ImageDataGenerator
datagen=ImageDataGenerator(preprocessing_function=preprocess_input,
                                 validation_split=0.2)
train_generator=datagen.flow_from_directory('/content/dataset',
                                                 target_size=(299,299),
                                                 color_mode='rgb',
                                                 batch_size=32,
                                                 class_mode='categorical',
                                                 shuffle=True,
                                                  subset='training')
val_generator = datagen.flow_from_directory('/content/dataset',
                                                 target_size=(299,299),
                                                 color_mode='rgb',
                                                 batch_size=32,
                                                 class_mode='categorical',
                                                 shuffle=True,
                                                subset='validation')


texts: Faced problem while applying OneHotEncoder

texts:
For classification, I was trying to convert categorical data into numeric by applying OneHotEncoder. But it shows error could not convert string to float

Here is the sample of my categorical data set and code of One Hot Encoding.

# TODO: create a OneHotEncoder object, and fit it to all of X
from sklearn.preprocessing import OneHotEncoder

# 1. INSTANTIATE
enc = OneHotEncoder()

# 2. FIT
enc.fit(train_obj)

# 3. Transform
train_ = enc.transform(train_obj)
train_.head()

And error message:

I couldn't understand what's the problem & how could I solve it.

If I apply get_dummies() method, is it similar of OneHotEncoder ?


texts: How to interpret linear trends in residuals?

texts:
I am trying to compare companies in the same industry and see how Profit and Number of employees correlated. My linear regression looks something like this:

Given the nature of the dataset, the model describes offices with low-profit better than the offices with a high profit. This is understandable since there are fewer samples for very profitable offices and a lot of samples for less profitable offices.
However, when I check for the residuals I get something like this:

The graphs show linear patterns for offices with lower profits. I am not certain how to interpret these trends.
Question:
What do linear trends in residuals indicate? How can the model be adjusted to handle them?


texts: Dealing with unseen data/categories in machine learning models for stream data

texts:
I want to build a machine learning model (xgb and lgbm) that has to handle streaming data on a weekly basis. The models are trained on a bi-weekly basis. The data includes order information and I want to predict the likelyhood that the order will be delivered. The orders are entered in the system and a week after one can say if the orders were indeed delivered or not.
For nominal data like supplier I use pd.get_dummies() for transformation. However, lets say I receive my order data for the orders that arrive next week. There is a new supplier that the trained model doesn't know yet as the column supplier_new_unkown_supplier does not exist in the saved model parameters.
Does anyone know how to deal with such cases?


texts: How to manage memory constraint and increase speed for 1 vs rest image similarity comparison for over 100k images for computer vision?

texts:
I'm looking for ideas on how to do things in a better way, efficiently when using Machine/Deep Learning.
I am working on a search improvement problem using Computer vision where I am thinking about comparing two images and see their similarity score so theoretically there are lots of things I can try such as: Structural Similarity, Template Matching, RMSE, PSNR etc... On the second thought, I can use Siamese Network. So in both cases, I have to search every image with every other image and get top-k scores which shows that k most similar images.
PROBLEM: I have a huge and by huge I mean around 20 Million+ images data. So in theory, let's suppose even if I use 100K images, it'll take huge amount of memory for nXm dimensional image matrix and on top of that, calculating scores with every other image for each image can't work on incoming live data as it'll take too much time. Impossible for Siamese Network as we have to insert all the images one by one.
Please tell me if there is a way to achieve this? I am using MongoDB. How can this be achieved? Someone said that ElasticSearch has a functionality to store image vectors. I have no clue what Elastic is and even if I read it, I don't know how to use that specially with Python?
Can someone suggest how it is achieved in IT industries and big companies because they have huge data and there must be a simpler way to do this.


texts: ffill missing data based on the sort order of another column

texts:
I have a column with missing data that I need to imput.  Column is called 'Bandwidth'.  There is a relationship between the Bandwidth column, and another column called 'Age'.  As Age increases, so does Bandwidth.
I want to ffill Bandwidth, but I have to do it based on the value in Age.  I have over 10k records, so there is too much data to create a dictionary. I have scoured the internet, and so far am not able to find a solution.
In the simplest form of what I want to do is this:
df.sort_values(['Age'])
df['Bandwidth'].fillna(method='ffill',inplace=True)
It did fill the bandwidth, based on the'sort' of bandwidth,  but not based on the Age sort.
The yellow ones were filled with the data from the green ones, but as you can see, the 'Age' is random.

But this is what I am needing it to do... (sort by age, then fill bandwidth from the above value.)


texts: How to pad the input of convolution layer with other values instead of zero

texts: In Keras, the area outside of the input is normally padded with zeros. Could we put other values instead of that? Because the characteristic of zeros to border is something we don't care about.


texts: Different Initial Q-Values in Q-Learning

texts: When working with Q-Learning, what is the difference between having a Q_0(a) with all values zero, random or optimistic?


texts: Trying to build a custom loss function in Keras/Tensorflow but get an error

texts:
I am using tensorflow 2.0 and I built a loss function that has two inputs, y_pred and y_true. I made sure to use only tensorflow operations throughout the entire function (it's long) and to return a final tensor of one floating number.

I ran the loss function with two tensors with the relevant sizes and it worked, no bugs, got a loss. But when I try to compile my model, it won't work and I get a dimensionality error, I'm not sure where it's coming from.

    inputs_shape_0=np.shape(np.array(pic))[0] #100
    inputs_shape_1=np.shape(np.array(pic))[1] #100
    last_layer_size=np.shape(np.array(x0))[1] #100*100

    model = tf.keras.Sequential()

    model.add(layers.Conv2D(64,
    kernel_size=2,padding='same',activation='linear',input_shape=
    (inputs_shape_0,inputs_shape_1,1)))
    model.add(layers.MaxPool2D(pool_size=(2, 2),padding='same'))

    model.add(layers.Conv2D(32,
    kernel_size=2,padding='same',activation='linear'))
    model.add(layers.Conv2D(1,kernel_size=2,padding='same'
    ,activation='linear'))
    model.add(layers.UpSampling2D(size=(2, 2)))

Summary of model:

    Model: "sequential_9"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d_14 (Conv2D)           (None, 100, 100, 64)      320
    _________________________________________________________________
    max_pooling2d_3 (MaxPooling2 (None, 50, 50, 64)        0
    _________________________________________________________________
    conv2d_15 (Conv2D)           (None, 50, 50, 32)        8224
    _________________________________________________________________
    conv2d_16 (Conv2D)           (None, 50, 50, 1)         129
    _________________________________________________________________
    up_sampling2d_3 (UpSampling2 (None, 100, 100, 1)       0
    =================================================================
    Total params: 8,673
    Trainable params: 8,673
    Non-trainable params: 0
    _____________________________

and I have a custom loss function:

   model.compile(optimizer='rmsprop',loss=dl_tf_loss, loss_weights=
   [None],metrics=
   ['accuracy'])

and I'm getting the following error:

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call
last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py
in
_create_c_op(graph, node_def, inputs, control_inputs)
   1550   try:
-> 1551     c_op = c_api.TF_FinishOperation(op_desc)
   1552   except errors.InvalidArgumentError as e:

InvalidArgumentError: Dimension size must be evenly divisible by 1000000
but is
10000 for 'loss_6/up_sampling2d_1_loss/Reshape' (op: 'Reshape') with input
shapes: [?,100,100,100], [2] and with input tensors computed as partial
shapes:
input[1] = [10000,1].

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call
last)
15 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py
in
_create_c_op(graph, node_def, inputs, control_inputs)
   1552   except errors.InvalidArgumentError as e:
   1553     # Convert to ValueError for backwards compatibility.
-> 1554     raise ValueError(str(e))
   1555
   1556   return c_op

ValueError: Dimension size must be evenly divisible by 1000000 but is
10000 for
'loss_6/up_sampling2d_1_loss/Reshape' (op: 'Reshape') with input shapes:
[?,100,100,100], [2] and with input tensors computed as partial shapes:
input[1]
= [10000,1].

I have no idea where it's getting the 1,000,000 from. In my custom loss function that doesn't exist but if you guys think that it's probably from there then I'll go back and look at it again.


texts:
This is my code to train a sigmoid neuron for Iris dataset .
convergence = np.Infinity

    temp_y_hat = np.zeros((80,1))
    alpha = 1
    epochs = 0

    for i in range(100):
        temp_y_hat = np.zeros((80,1))
        for i in range(X_train.shape[0]):

            print("W : ",W.shape)
            print("X_train :",X_train.iloc[[i]].shape)

            #res = np.dot(W,np.reshape(X_train.iloc[i].values,(-1,1))) + b
            res =np.dot(W,X_train.iloc[[i]].T) + b

            z = 1/(1 + np.exp(-res))

            if z >= 0.5:
                temp_y_hat[i] = 1
            elif z < 0.5:
                temp_y_hat[1] = 0

            if y_train[i]==temp_y_hat[i]:
                continue
            else:
                if y_train[i]==1 and temp_y_hat[i]==0:
                    #W = W + alpha*(np.reshape(X_train.iloc[i].values,(1,4)))
                    W = W + alpha*X_train.iloc[[i]]
                    b = b + alpha
                elif y_train[i]==0 and temp_y_hat[i]==1:
                    #W = W - alpha*(np.reshape(X_train.iloc[i].values,(1,4)))
                    W = W - alpha*X_train.iloc[[i]]
                    b = b - alpha

        convergence =  np.sum(np.square(np.subtract(y_train,temp_y_hat)))

        print("convergence ",convergence)
        epochs = epochs + 1

    print("epochs ",epochs)

    return W , b

When I run this code , I am getting the following error :
if z >= 0.5:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

When I print out the shape of W and X_train.iloc[[i]] in each iteration :
I get the following output which is fine in the first 4 iterations whereas in the 5th iteration , the shape of matrix W becomes wrong and hence I get the above error .
1st iteration : W :  (1, 4)
X_train : (1, 4)
2nd iteration : W :  (1, 4)
X_train : (1, 4)
3rd iteration : W :  (1, 4)
X_train : (1, 4)
4th iteration : W :  (1, 4)
X_train : (1, 4)
5th iteration : W :  (2, 4)  --- wrong shape
X_train : (1, 4)

I am suspecting the problem is in the segment where I'm updating the value of W .
 if y_train[i]==1 and temp_y_hat[i]==0:
                    #Commented W = W + alpha*(np.reshape(X_train.iloc[i].values,(1,4)))
                    W = W + alpha*X_train.iloc[[i]]
                    b = b + alpha
                elif y_train[i]==0 and temp_y_hat[i]==1:
                    # Commented W = W - alpha*(np.reshape(X_train.iloc[i].values,(1,4)))
                    W = W - alpha*X_train.iloc[[i]]
                    b = b - alpha

But , given that , W is (1,4) and X_train.iloc[[i]] is also(1,4) , and that I take transpose of X_train.iloc[[i]] in np.dot , Why am I getting this error ??
Can someone please help me understand the problem and get rid of this problem ?


texts: Understanding the softmax output in Youtube's recommender

texts:
This question has been asked before, but never (that I can see) satisfactorily answered.

I'm reading Youtube's paper on their recommender system. The system has two elements, the first of which is a DNN generating 100 "candidate" videos, which are then combined with candidates from other sources and ranked by a second DNN. In the paper they say that they treat the candidate generation problem as extreme multiclass classification with Softmax. That approach is understandable; the indices of the N highest values in the softmax output become the N predicted candidates - easy peasy.

Reading further in the paper however, I began to get confused about exactly what their network is doing. They give a figure of their network's structure:

And it's apparent from that image that there's a whole other step beyond the softmax layer, which is what I'm not understanding. There's also later in the paper the following quote:

  The softmax layer outputs a multinomial distribution over the same 1M video classes with a dimension of 256 (which can be thought of  as  a  separate  output  video  embedding).

and also that:

  Since calibrated likelihoods from the softmax output layer are not needed at serving time, the scoring problem reduces to a nearest neighbor search in the dot product space for which general purpose libraries can be use.

But I'm confused about what they're actually implementing here. The assertion that the softmax layer outputs...with a dimension of 256 implies to me that they literally have a final layer in the form Dense(256, activation='softmax') instead of Dense(n_classes..., however the presence of "class probabilities" in the figure makes that impossible, since you couldn't convert such an embedding to class probabilities.

So; I'm really confused; does anyone know how to interpret exactly what's being done here?


texts:
I have a sequence of event and would like to predict the next one. The training data looks like this:

Ev1,Ev2,Ev5,Ev6,Ev7
Ev1,Ev6,Ev99
Ev4,Ev3,Ev6

So, the idea is to get Ev7 given Ev1,Ev2,Ev5,Ev6

The problem is that the number of "final" events is very high (100K). I tried to look into neural network but it means the last layer would need to have size of 100K.

Can anybody point to any other ways how to do that or any examples?


texts: How to run AgglomerativeClustering on a big data in python?

texts:
I run AgglomerativeClustering on a sample of data and fit a model. then I decide to predict this fit for all of my data but I got MemoryError.

How can I run AgglomerativeClustering on a big dataset?
should I create a classification based on clusters label?!


texts: Embedding git commit into the resulting data

texts:
Our pipeline works something like that:

Collect bunch of raw data (10-100 GB) from microscope
Process data using MATLAB scripts
Change few parameters based on raw data, as well as add new features to the scripts
Commit the scripts with new features to git repo
Re-run processing, save results as figures and CSV data (1-10 MBs)

Are there tools available for seamlessly linking MATLAB figures or tabulated data with the code version (git commit hash) that was used to produce it?

I know that MATLAB is aware of git, as well as we can call system commands


texts: Split the data between the Training Data and Test Data using sklearn

texts:
Work to do

My job is to take the data and divide it between Training and Test using 30% of the data as Test where both should have the same ratio between positive and negative.

CSV File

age,Feature 2,Feature 3,Feature 4,income,Feature 6,Feature 7,Feature 8,Feature 9,Feature 10,Feature 11,Feature 12,Feature 13,Feature 14,Feature 15,Class
77,1,0,0,3,0,1,1,1,0,1,1,1,0,1,0
35,1,0,1,4,0,0,0,1,1,0,1,1,0,1,0
79,1,0,0,2,0,1,1,0,0,0,1,1,1,0,1
61,0,1,0,1,0,1,1,1,0,0,0,0,1,0,0
62,0,0,0,2,0,1,0,0,0,1,1,0,0,0,1
63,0,1,1,0,1,1,0,0,1,0,0,1,0,0,1
29,1,1,0,1,1,0,1,1,0,0,1,1,1,1,0
39,0,1,1,5,1,1,1,0,1,0,1,1,1,1,0
51,0,1,1,6,1,1,0,1,0,0,1,1,0,1,1

Code - Training Data and Test Data

#!/usr/bin/env python3

import numpy as np
import pandas as pd
import matplotlib.pyplot as plot
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import Imputer

# Reading CVS file
dataSet = pd.read_csv('./TestDataFile_ML.csv')
X = dataSet.iloc[:,:-1].values
y = dataSet.iloc[:,15].values

df = pd.DataFrame(dataSet)
print(df)

# Missing data
imputer = Imputer(missing_values="NaN" , strategy="mean" , axis=0)

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y , test_size = 0.30 , random_state = 0)

# Creating linear regression object
linearReg = LinearRegression()
linearReg.fit(xTrain , yTrain)

#Now, testing model
yPrediction = linearReg.predict(xTest)

print(X.shape)
print(y.shape)
print(xTrain , yTrain)

#Plotting the training set
plot.scatter(xTrain , yTrain , color="red")
plot.plot(xTrain , linearReg.predict(xTrain) , color="blue")
plot.title("title")
plot.xlabel("x label")
plot.ylabel("y label")
plot.show()

# Test set
plot.scatter(xTest, yTest, color = 'red')
plot.plot(xTrain, linearReg.predict(xTrain), color = 'blue')
plot.title("title")
plot.xlabel("x label")
plot.ylabel("y label")
plot.show()

Problems found

The following error is showing me:
ValueError: x and y must be the same size

This is because apparently in the part of

X = dataSet.iloc [:,: - 1] .values
y = dataSet.iloc [:, 15] .values

I'm not taking the correct values. If someone can help me to correct the error I will thank you

Updates

After addingstratify = y to:

xTrain , xTest , yTrain , yTest = train_test_split(X , y , test_size = 0.30, random_state = 0 , stratify = y)

It shows me the following graph, in which I presume that it is incorrect


texts: how to create sklearn pipeline object using predtrained standardscalar object

texts:
I am having pretrained Sklearn model and pre-trained Standard scalar object saved as pickle . And now I want to create Sklearn pipeline using both of it.
I need sklearn pipeline to convert it into ONNX format.
I couldnt do it as pipeline takes standard scalar class and then we need fit pipeline using data but in my  case models and scalar both are already fitted.


texts: Adding additional information in content-based recommendations

texts:
I have a book dataset where 100 users have rated the books as like/dislike. Each observation with features
Table1 : ['user_id','book_name', 'book_genre','author','date_published','like/dislike']
These users were asked to list down their favorite authors and genres, they are stored in a different table as
Table2: ['user_id', 'authors_liked', 'authors_disliked', 'genre_liked', 'genre_disliked'].
where the fields are lists of authors and genres liked/disliked.
Then there is the books database with information on all the available books(100,000 entries)
Table3: ['book_id','book_name', 'book_genre','author','date_published']
Question: Given a 'book_name' and 'user_id', predict if the user likes or dislikes it.
Currently, I have trained a classifier for Table1.
I have applied cosine similarity in Table2 and Table3.
I'm predicting the book as 'likes' if the output of the classifier is 'like' and the similarity is >0.5
Is there a better/standard way to incorporate Table2 information into the recommendation system?


texts: Getting the positive impacting features using SHAP

texts:
I'm attempting to use SHAP to automatically extract feature names that have a positive impact on my regression models. On inspection of the code I see that the bar plot, for example, determines these by taking the mean absolute SHAP values for a feature. Being an absolute value, it obviously takes the absolute impact but I want to only consider positive impacting values.

Is my intuition that I can just take the mean instead of the mean of the absolute values correct?
(highly) Negative SHAP values should give a negative mean value.

Is this a good approach or am I missing some better way to do this?

EDIT:
I am specifically interested in features that raise the predicted value. ie. if feature_1 lifts the predicted value by 100 and feature_2 by 1000, I want this information to be extracted as feature_2 has and higher impact on the output value.


texts: How to determine the number of forward and backward passes in deep learning (CNN)?

texts: Is there a way to determine the number of forward and backward passes in the training of a neural network using python?


texts: Classification accuracy of a Random Multi-label Classifier

texts:
What is the exact accuracy of a random classifier which has n labels (say 1000) where k labels (say 50) are true?
Can I say the accuracy of a random classifier has an upper bound of k/n?

-Edit-

I am interested in a numerical figure or an upper bound for a random classifier (for example the accuracy of a classifier randomly picking 50 classes out of 1000 where some 50 classes are correct).
Sorry if my previous question is not fully understood.


texts:
I have 2 vectors $1000 \times 1$, lets call them $y_{1}$ and $y_{2}$. Each vector represents a normal distribution with certain mean and variance.

I plot the contour plot using the following R code:

x <- y1
y <- y2
s <- subplot(
  plot_ly(x = x, type = "histogram", showlegend=FALSE),
  plotly_empty(),
  plot_ly(x = x, y = y, type = "histogram2dcontour", showlegend=FALSE),
  plot_ly(y = y, type = "histogram", showlegend=FALSE),
  nrows = 2, heights = c(0.2, 0.8), widths = c(0.8, 0.2),
  shareX = TRUE, shareY = TRUE, titleX = FALSE, titleY = FALSE
)
layout(s)

and i get the following plot

What I would like to do now, is to plot a 3D contour plot (so that I can actually see the "mountain" that is created after plotting the histogram of $y_{1}$ against the histogram of $y_{2}$). So in the z-axis I would like to have the frequencies of the values. Any suggestions ?


texts: How to develop a good model for predicting electricity usage?

texts:
My goal is to come up with a set of research questions for which to drive my self study into my data.

I have Smart Meter data that is taken every minute, on my home's power usage.

How could I possibly use the time series data on my household energy consumption that would help me extract practical insights from it?


texts: Predict_proba on a binary classification problem

texts:
I have a binary classification task on my hands, i have a bunch of people that i need to classify as being ones or zeros and then use predict_proba to estimate how confident my prediction was on the samples used for inference.
My understanding is that predict_proba for most classification algorithms isn't accurate and needs to be calibrated. Is there a common approach to get objectively accurate class probabilities ?
Algorithms names , techniques and some code if possible. Thanks!

Note : my classes are imbalanced 80/20.


texts:
I would like to capture the following pattern using python
   anyprefix-emp-<employee id>_id-<designation id>_sc-<scale id>

Example data

strings = ["humanresourc-emp-001_id-01_sc-01","itoperation-emp-002_id-02_sc-12","Generalsection-emp-003_id-03_sc-10"]

Expected Output:

[('emp-001', 'id-01', 'sc-01'), ('emp-002', 'id-02', 'sc-12'), ('emp-003', 'id-03', 'sc-10')]

How can i do it using python.


texts: Pre-trained Alexnet on the Places205 dataset gave me bad accuracy

texts:
I need a pre-trained Alexnet model on the images dataset Places205 (details on this dataset are [here] http://places.csail.mit.edu/index.html)
As i couldn't find any pre-trained model on pytorch/keras...etc, I am re-training it myself and it take 10 days to get 500 epochsbut still the best top-1/top-5 accuracy i could achieve is : 46%/78%. It's not really bad but i would be happy if i can get something like top-1 accuracy of 50% (like most papers online).
Now i have some theories about the matter but i could be wrong :
1-It's because i am using a cropping of picture instead of resize which is taking some needed details off.
2-Maybe be because i am using 30% of my data as validation/test which means less data for traininig (i am not using k-fold. i hope it's not cretical).
3-maybe because the learning rate is constant (no scheduler).
I want to try those but it takes lots of time to get the result. so, a little experience might help me here.
Also, if you have any idea where i can find a pretrained model of this and hopefully import it to pytorch, it would be really helpfull.
Thank you in advance


texts: Plotting libraries for Scala on Zeppelin

texts:
My main question is it looks like Zeppelin limit the display of the results to on 1000, I know that I can change this number but when I change it Zeppelin become slow. And it looks like the default plotting tool of Zeppelin also plot the first 1000 results.
Is there a configuration or a way to make the plotting tool plot all the data?

If no, is there any equivalent for Matlibplot to Scala on Zeppelin?


texts: Output prediction from a sequence of data

texts:
Given the machine learning libraries available for many different languages, it's possible to utilise algorithms where you don't need to have a detailed understanding of their application or workings or datascience in general.

I have tried to find a list of algorithms that are suitable for sequence prediction from a window of previous input values, which I could then apply in such a library e.g. Accord for C# but have failed.

Each observation I have is an array of 2-dimensonal data in the following form: {MODE1/2/3, int 1-98}. That is, multiple inputs produce multiple outputs. I'd like to examine preceding sets of values from either t-1 or t-x where x could be a variable amount of preceding value sets.

So, I'd like to ask what algorithms excel at prediction from a window of sequence data, and what are their strengths / weaknesses.


texts: How does a CBoW model convert a word to a vector?

texts:
A CBOW model actually takes multiple words as inputs and a targeted central word as the output.
So, the trained model actually maps several words to a single one, I mean it takes context words and outputs the central word.
But what we expected to get is model mapping a word to its vector representation. It seems the output is coincident but not the input and the mapping.

So like in genism, how it really works to map a word to its vector representation?
Does it just save all final model's outputs as the central words' vector representation? But the final model's outputs would close to the ground truth's one-hot embedding rather than a vector with context information.

For short, my question is:
How does a CBoW model convert one word to its vector representation?


texts: What Is The Difference Between Additive Natural Cubic Splines and Tensor Product Natural Cubic Splines?

texts:
Good day. Taking into account the picture shown, using tensor product is computationally expensive considering the fact that it has higher dimensions. I am just thinking why it is compared both additive and tensor products here (having the same training error, test error, bayes error values)?

Another thing I want to clarify is how did the formula for degrees of freedom for additive arrive as total df = 1 + (4-1) + (4-1) = 7? (Tensor product is a simple multiplication of 4x4).


texts: reinforcement learning with both dynamic and static states

texts:
I am wondering is there any algorithm accommodate both dynamic and static states?

In my problem, some states change over time depending on the actions taken. Other states are static. I am aware that contextual bandit is appropriate for static states.

But is traditional reinforcement learning algorithm, like Q learning, appropriate for a problem where states contain both dynamic ones and static ones.


texts: Using VLOOKUP to match each value of cell (cell has multiple values separated by comma) in table A with values in table B

texts:
Problem Statement:
Table 1: It has multiple values in each cell of Column B.
Table 2: Is the reference table Employees and their category is mentioned

Output Required:
Table 3: This is what the output should look like, i.e. each column (C to F) shows the total no. of Sr, Jr, Cont and Int in that row


texts:
My hypothesis h depends on multiple categorical variables (a,b,c) each with their corresponding set of possible values (A,B,C). Now each of my data point exist in this space where I have no control over the values (observational data).

  For e.g. Hypothesis to predict user shopping probability say depends
  on (Age, Country, Gender, Devicetype etc.)

How could I sample the above data set so that it would give me a good representation. The techniques I have learned from the books very well apply to one dimension but that is a rare case in practice. If I sample across one dimension my other dimensions will be heavily skewed towards some values. Is there any standard algorithm to give good sampling?


texts: What does backpropagating through labels mean?

texts:
In tensorflow's documentation for tf.nn.softmax_cross_entropy_with_logits_v2 it is written:

  Backpropagation will happen into both logits and labels. To disallow
  backpropagation into labels, pass label tensors through
  tf.stop_gradient before feeding it to this function.

I do not understand what backpropagation into labels mean. How do we calculate the loss of the labels?


texts:
I have an excel file containing a long text in column A. I am looking for the words starting by "popul" such as popular and populate . I can find these cells by the formula:

    =SEARCH("popul",A1,1)

I want a function that returns the whole words starting by popul such as popular and populate.


texts: Where the following propagation model arise in practice

texts: This is in continuation to a previos question. I would like to know where in practice will the following M-ary tree-structured social network propagation model arise. I am looking for some concrete examples in social media (twitter and youtube etc.). Eventually, I want to do some statistical analysis on such social networks.


texts: find most dense neighborhood of points in high dimensional space

texts:
I'm working on a project where I have many high-dimensional points and I want to find the most dense neighborhood of them. Ideally, out of my ~500 points that are each a 4x300 matrix (300 ms time series of four variables) I want to find the ~30 points that are the most similar. I've looked into k nearest neighbor methods but those are all finding the smallest neighborhoods of a certain point, I want to be able to specify the size N of the neighborhood (how many data points) and get the subset N points that are the closest together.

I think(?) that this is a clustering problem so I've looked at soft subspace clustering which clusters using the more relevant dimensions and also I've looked into biclustering which was developed specifically for time series data.

Any help would be greatly appreciated! I have a problem that I'm sure someone else has thought of but I don't think I know exactly how to word it and everything I've looked at so far has been close but not exactly what I need. So any papers/code/tutorials/etc that give me any information on how I might solve this would be fantastic :D

Update: I've looked into using Frechet distance to use as a distance between time series. Would that be an appropriate metric to use for this type of data?


texts: Should I accept a model with good validation loss & accuracy but bad training one?

texts:
I am training a model for 3D CT scan classification (is there a nodule or not in the image).
I'm using a 3D CNN and I obtaint the following curve;

(blue curve is the validation data (10% of dataset))
as you can see, I obtain good result on both validation loss (categorical cross entropy, up to 0.16 for val) and metric (categorical accuracy) (up to 1.0 for val) but not on the training data.
I'm using a grid deformation on the training data and several dropout layer since my dataset is very small (120 samples), so it could explain the lower result on training set.
Still, I dont know if I can accept this type of result ?


texts: Machine Learning Best Practices for Big Dataset

texts:
I am about to graduate from my Master and had learnt about machine learning as well as performed research projects with it. I wonder about the best practices in the industry when performing machine learning tasks with Big Datasets (like 100s GB or TB). Appreciate if fellow data scientists can share their experience. Here are my questions:

Obviously, very large datasets take longer time to train (can be days or weeks). Many times we need to train various models (SVM, Neural Network, etc.) to compare and find better performance model. I suspect, in industry projects, we want the results as quick as possible but produce the best performance. Are there any tips for reducing the training & testing time? If you recommend subsetting the dataset, I will be interested to learn how best to subset the dataset to cover all or majority of scenarios from the dataset.
We know that performing cross validation is better as it may reduce over-fitting. However, cross validation also takes time to train and the model trained with cross validation may not be implemented straight (speaking from python sklearn experience: I need to train the model with dataset again after cross validation testing for it to be implemented). Do you normally do cross validation in your big data projects or getting by with the train-test split?

Appreciate the feedback.


texts:
The problem I want to solve is as follows:
I have data about how many teaching hours different students spend with their teacher in order to pass their exams. The teaching varies from 25-55 hours. In the school there are approximately 15 teachers and given a set of 10 student-teacher features that describe personality characteristics such as, age, sex, education, nationality, etc., I want to develop a student-teacher match model in order to minimize the teaching hours a student needs to pass the exams.

Any idea of what sort of algorithms I should use, or any examples I can look at?


texts: What it's the function of batch size in Keras LSTM?

texts:
I'm trying to understand the structure and input shape of data of the LSTM networks. If I have this diagram

in which there are 4 input features and I wish predict value of only one feature
Q1: what's the function of the batch size or samples if features and time steps are contemplated yet?

Q2: input shape it should be (1, 5, 4) if I want a batch size = 1?


texts: Why is MLP working similar to RNN for text generation

texts:
I was trying to perform text generation using only a character level feed-forward neural network after having followed this tutorial which uses LSTM. I one-hot encoded the characters of my corpus which gave a vector of length 45. Then I concatenated every 20 characters and fed this 20*45 length vector as input to an MLP with the 21st character's one hot as the output.

Thus my X (input data) shape is -> (144304, 900)
and my Y (output data) shape is -> (144304, 45)

Here's the output from my code:

  alice was beginning very about a grible thing was and bet she with a
  great come and fill feel at and beck to the darcht repeat allice
  waited it, put this was not an encir to the white knew the mock turtle
  with a sigh. ‘i only took the regular cours was be crosd it to fits
  some to see it was and getting she dodn as the endge of the evence,
  and went on to love that you were no alway not--ohe f whow to the
  gryphon is, who denight to goover and even stried to the dormouse, and
  repeated her question. ‘why did they live at the bottom of a well?’

  ‘tabl the without once it it howling it the duchess to herself it as
  eng, longing one of the door and wasting for the homend of the taits.’

  ‘sthing i cancus croquet with the queen to-day?’

  ‘i should like it very much,’ said the dryphon, ‘you first form into a
  line along-the sea-shore--’

  ‘the right!’ cried the queen nother frowing tone. any the this her for
  some thing is and at like the look of it at all,’ said the king:
  ‘however, it may kiss my hand if it likes.’

  ‘i’d really feeer that in a few this for some whele wish to get thing
  to eager to think thcapered twice, and shook note bill herself in a
  lell as expectant, and thowedd all have come fuconfuse it the march
  hare: she thought it must be the right way of speaking to a mouse: she
  had never done such a thing before, but she remembered having seen in
  her brother’s latin grammar, ‘a mouse--of a mouse--to a mouse--a
  mouse--o mister once to hin fent on the words with her fane with ale
  three king the said, and diffich a vage and so mane alice this cime.

My Question is why is MLP working similar to an RNN/LSTM. What's the advantage of using RNN/LSTM for such tasks over MLP?


texts:
I am aware there are time series cross validation methods however I don't think there is one yet which solves my task. I want to know if my idea seems reasonable or if there are better solutions out there.

Lets say I have a one time event each year for each observation in my data. Each observation experiences this one time event every year.

I also have 365 daily time series data for each observation.

That is, say each observation is a person who releases their progress report (randomly throughout the year)
They mostly just copy and paste their progress report from year to year - making small changes
I can track the individuals work performance over the calendar year after they release their progress report.

I want to build a classifier based on performance ("good", "bad", "neutral") and then predict worker performance.

For exampe:

---- Say we are in 2007

---- I collect the reports for all workers from Jan 2005 - Dec 2005

---- Run a model to process the text and put into machine readable format for 2005 (Doc2Vec or some other word embedding model) - therefore each document is represented in some high dimensional space (each worker can submit their document at any stage throughout 2005 which is why I choose dec 2005 to construct this part of the model).

---- I calculate the "net" performance - such as some productivity score from Jan 2006 - Dec 2006 - therefore each worker will have a single score for their performance in Dec 2006.

---- I build my classification model in Jan 2007 to classify which workers are going to be "good", "bad" or "neutral" throughout 2007.

---- So I use the reports from 2005 to build the document embeddings from a Doc2Vec model. I then feed these document vectors to a machine learning classifier on the training sample of 2006 (i.e. the model gets to see the dec 2006 scores to train on). The out of sample testing occurs in Jan 2007 where the model is not able to see the end result (dec 2007).

Problem:

Say I train the model on 5000 reports in 2005, put these 5000 reports into a ML classifier in 2006 to learn from the true performance score in that year, finally I test in 2007 on the out of sample data. Since these document vectors are in a high dimensional space the model will simply just classify the documents in 2007 as it saw in the results in 2006 - therefore the results are biased....

To over come this issue...

Say I randomly sample 4000 reports to build the document vectors on in 2005 , I pass these 4000 documents to the ML classifier as training data in 2006 to obtain my prediction model based on these workers 2006 scores. Then in Jan 2007 I pass the remaining 1000 observations documents in 2006 (not 2005) as my testing data to the ML classifier - i.e. reports and scores the model has never seen before.

I thought that this method will not bias the results, have no look ahead bias, but I am only able to "forecast" a random sample of the workers - which isn't a problem.

I just want to know your opinions on this method. I do not want any biases at any stage of the process.


texts:
I don't understand the following statement:

  The choice of learning rate m does not matter (for Perceptron) because
  it just changes the scaling of w (weights). The site with this statement that wikipedia cites.

The update rule of perceptron is $w \pm x$ for mistaken cases. If the update to the weight vector is addition or subtraction, how can it only scale the weights?


texts:
I have completely followed the machine learning course on coursera Machine Learning by professor Andrew Ng

Now I want to put my knowledge to action. Some ideas that I have include :

-Voice synthesis
 -Voice recognition

But since the course did not focus specifically on application of machine learning in these domains, could some one point me to some other course or books that can get me started.


texts: how to make the object detection detect only full object not part of it?

texts:
I have trained an object detection model to detect different kinds of cards and crop them and save them, the model detect the object very well and I got a good accuracy, now I want to add some post-processing to not saving the object if there are missing part of it, for example the model detect correctly the object if with only 50% of the object in the image or even less with high confidence, I don't want to save the object when there is obstacle over the object or when there is only part of the object, how can I solve this ?
example is this image:


texts: What parameters to use when normalising training, validation, and testing data?

texts:
I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data.
If we split up the data as follows:

'Training' (call this set 'A' for now) and testing data
Split the 'training' into training (call this set 'B' for now) and validation sets

what parameters should be used when normalising the datasets?
Am I correct in thinking that:

We normalise dataset 'B' and then extract the means and standard deviations on it
We then normalise the validation set using those parameters obtained from set 'B'
Once we have used the validation set to find my hyper-parameters with cross-validation, then we normalise set 'A' and extract its parameters
Use the parameters from set 'A' to normalise the testing set

Is this correct, or have I misunderstood something? I know this is basic, but I can't seem to find a straightforward answer to this anywhere?


texts:
I am trying to assume that my y variable has a Poisson distribution and I am using TensorFlow to do so. The model I have is as follows:
def my_dist_Poisson(params):
    return tfd.Poisson(rate=1.0)

model = tf.keras.Sequential([
  tf.keras.layers.Dense(1),
  tfp.layers.DistributionLambda(my_dist_Poisson),
])

model.compile(optimizer=tf.optimizers.Adam(learning_rate=0.05), loss=NLL)
model.fit(x_train, y_train, epochs=20, verbose=True)

The variable I am trying to predict is the 'cnt' variable in the following dataset (for regression): https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset
However, I get the following error and don't seem to know why this shows up.
ValueError: No gradients provided for any variable: ['dense_24/kernel:0', 'dense_24/bias:0'].