texts: CNN with Multi channel input or CNN with Multi instance learning?

texts:
I have 500 Dicom images of medical scan of patients. These are 3 dimension scans , shape = [300 x 300 x 3]. From these I have extracted Front and side views. So, for each patient I have 2 images of shape [300 x 300].

In order to build a classifier,
Should stack these 2 views and train a CNN {[300 x 300 x 2] x 500} ->  Multi channel input,

Or should i pass each view as a new data altogether {[300 x 300 x 1000]} -> Multi instance learning?


texts: sklearn.feature_selection vs xgboost feature_importances?

texts:
sklearn.feature_selection vs xgboost feature_importances

Can somebody explain in-detailed differences between sklearn.feature_selection and xgboost feature_importances? And how the algorithms work under the hood?

Which module gives the best results?


texts: Scalable way to calculate betweenness centrality for a graph in spark

texts: I have a use-case to calculate betweenness centrality of nodes. I have tried graphx with spark-betweenness but it is a very long running job. Has anyone successfully calculated betweenness centrality of a large network with around 10 million vertices and 100 million edges?


texts: Is R2 score a reasonable regression measure on huge datasets?

texts:
I'm running a regression model on a pretty large data set and getting a fairly woeful $R^2$ score of ~0.2 (see plot below), despite the plot looking like the model is generally pointing in the right direction.

My question is, when you have over a million data points, how high can you realistically expect the $R^2$ to go in real world data with a decent amount of noise?

What prompts by scepticism of such traditional measures are articles such as this that discuss how quantity of data can degrade statistical tests.

Let me know what you think and any regression examples using $R^2$ score as a quality metric.


texts: Where can I get labels for small ImageNet?

texts:
I have recently downloaded the small image net dataset:

http://image-net.org/small/download.php

The archives contain a lot of images, but no other files. How do I know which label the images have?


texts: What does 'contextual' mean in 'contextual bandits'?

texts: I recently read a lot about the n-armed bandit problem and its solution with various algorithms, for example for webscale content optimization. Some discussions were referring to 'contextual bandits', I couldn't find a clear definition what the word 'contextual' should mean here. Does anyone know what is meant by that, in contrast to 'usual' bandits?


texts: Machine Learning algorithm to predict an outcome where output is not known

texts:
I have a very unique problem that I would like to solve using machine learning. I have a set of 90 or so unique options. Each of these options has a unique set of features (5 to be specific) that vary over time and that can be represented mathematically as either scalars or booleans. I want to train a model to tell me what the best option at that moment is given the complete set of inputs.

At first, I figured a simple feedforward neural network would do the trick but I don't have a ton of experience with ML models so I ran into the following issue. I know my problem isn't really a classification problem since I am trying to pick the best option. I figured it could be easily modelled as a regression problem (as some mathematical combination of the inputs) which is fine but then I don't know what the expected outcome would be. For example, if the expected output is the weighted average of the inputs, I don't know what the final value should be. All I know is that option B should be suggested over option A.

I started reading about random forests and they seem like they might help my situation but I wanted to get other opinions first.


texts: memory error in matrix cosine_similarity

texts:
I have (20905040, 7)
of a dataset to recommend 10 different product to the user
it could be larger than that but anyway I got memory error when processing the

cosine_sim = cosine_similarity(normalized_df,normalized_df)

  --------------------------------------------------------------------------- MemoryError                               Traceback (most recent call
  last)  in
        1 get_ipython().run_line_magic('time', '')
  ----> 2 cosine_sim = cosine_similarity(normalized_df,normalized_df)

  ~/venv/lib/python3.6/site-packages/sklearn/metrics/pairwise.py in
  cosine_similarity(X, Y, dense_output)    1034     1035     K =
  safe_sparse_dot(X_normalized, Y_normalized.T,
  -> 1036                         dense_output=dense_output)    1037     1038     return K

  ~/venv/lib/python3.6/site-packages/sklearn/utils/extmath.py in
  safe_sparse_dot(a, b, dense_output)
      140         return ret
      141     else:
  --> 142         return np.dot(a, b)
      143
      144

  MemoryError:

questions
1. when I have too much rows how do I apply cosine similarity?
2. do they talking about ram memory? or what memory error ?
3. is there way to use gpu for cosine similarity training?
4. any good idea?


texts:
I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works.

Let's say that our context size is 1024 tokens, the embedding size is 768 (so that each token and its subsequent hidden states are represented by vectors of size 768), and we are using 12 multi-attention heads. So in the diagram above, there are 1024 r's and each r has dimensionality 768.
For a given layer in the transformer, how many normalization statistics (sample mean and stdev) are computed? Do we do one normalization per token, for 12x1024 normalizations so that the feature values within each token have mean 0 and std 1? Or do we normalize the values for each feature across tokens, for 12x768 normalizations? Or do we normalize all the feature values for all the tokens together, for 12 normalizations? Do we compute separate normalizations for each context in the minibatch?
I'm also keen to understand intuitively why this normalization is desirable. Assuming that the scheme is to normalize the feature values within each token: let's say one of our tokens is a bland word like "ok" whereas another token is the word "hatred". I would expect that the representation of "hatred" would be spikier, with higher variance among the different feature values. Why is it useful to throw away this information and force the representation for "ok" to be just as spiky? On the other hand, if the normalization scheme is to normalize across feature values, so that if you take feature 1 from all of the tokens in our context, they will have zero mean and stdev 1, doesn't this throw away information when all of the words in our context are very negative, for example in the context "war violence hate fear"?
Separately, with layer normalization it seems like it is optional to re-scale the normalized values through learned bias and gain parameters. Does GPT-2 do this, or does it keep the values normalized to mean 0 and std 1?


texts:
I am looking at density trees

The intuition about the y-axis is clear: the tree indicates the modes which then merge at merge height:

$$m_p(x,y) = \sup{t: \exists C \in \textit{C} \quad s.t. \quad x,y \in C }$$

In terms of execution, everything goes through the use of a density estimator, as clearly described in the paper.

I am however not sure about what goes in the x-axis.

The fact that the x-axis ranges between 0 and 1 makes me suspect that the x-axis is somehow related with the distribution of the probability; but I am missing the details.

I have also checked the paper which is the original source of the picture above. There, the 1-dimensional case is somehow covered:
. But I am still missing the details for n-dimensional cases, such as the Yingyang data.


texts: Why fitting/training a model can be considered as learning?

texts:
I was looking around and I can't find a good answer, I just want to know why it can be considered as learning and is not just "calibration" or "parametrization".
I feel the word "learning" is overqualified for the things the models do.
Thanks in advance.


texts: How many pairs of image needed for training Siamese Network? And how to augment them?

texts:
Hi Deep Learning researchers and engineers,

Does anyone have experience in Siamese Network regarding the training data size? How many pairs of image do I need to train a Siamese Network?

And what kinds of data augmentation should I do?

Thank you.


texts: Where can I get mapped ground-truth of Pascal VOC2012 in COCO classes?

texts:
I'm trying to make use VOC2012/2007 dataset into COCO2014 training set ?
As far as I know that most of class (20) in VOC is also in COCO (80).
I think I can write code to map from VOC class to COCO class but I think I'm not the first one thinking like this, so anyone know where I can get such a thing please help to reduce time.
I tried to search but GG gave wrong results.


texts: Multidimensional collaborative filtering model

texts:
I have a dataset that is approximately structured in the following way:
500 users, 500 products, 100 countries, 2 seasons, 300,000 ratings. Meaning that I have 300,000 rows containing unique combinations of user, product, country, season, each with an assigned rating for the product from the user in a given year and location.
E.g. User 257 rated Product 105 a score of 4.1 in Country 29 in Season 1
From this, I would like to create a model to predict the mean rating between any given combination of user and product, i.e. an estimate of what the mean rating would be if ratings across all countries and seasons were known.
My understanding is that collaborative filtering typically uses a matrix that has a structure where users are rows and products are columns. I'm not sure that is applicable in my case due to the countries and seasons giving the matrix two extra dimensions. The model would ideally learn a mapping from four dimensions (n_users x n_products x n_loc x n_year) -> two dimensions (n_users x n_products) where the first tensor is sparsely populated with known ratings and the second tensor would be populated by predicted ratings.
Using the 300,000 known ratings, we can estimate the mean ratings of roughly 8,000 user-product combinations by averaging across all their known countries and seasons and then treat these mean ratings as labels for supervised learning, i.e. 8,000 of the 250,000 (500 x 500) entries could be filled.
My questions are:

Which type of model would you recommend using?

If we use the 8,000 estimated labels, can features be engineered to capture the information contained in the joint rating distribution of user ID, product ID, country and season?

Many thanks for any help or suggestions provided!


texts: How many layers should I replace in transfer learning CNN

texts:
I am designing a convolutional neural network that I believe requires transfer learning to function in practice. The network will be a character level CNN for text classification, more specifically, authorship identification of an author given unknown texts. The initial model will be trained on millions of texts from thousands of authors. In practice, if I want to be able to determine the authorship of a new given author/class not trained upon originally, I need to use transfer learning.
The structure of the network involves 6 convolutional layers and 3 fully connected layers. Given that the amount of data of the new author/class will be minimal in most cases, which layers should I replace and retrain for the new class for it to be the most effective? Or are there other methods I could consider to solve this problem?


texts: Reducing features to 2 by PCA confusion Matlab

texts:
I am trying to reduce $500$ features to $2$ as an assignment. I wrote the following code and I am deeply concerned if it is true as when I plot it on the graph it does not look good. It should look good as I have to apply cluster algorithm to is as a next step. Can you help me pls?
X = data;

X = bsxfun(@minus,X,mean(X));  %%standardize

CM = transpose(X)*X;  %%coveriance matrix is calculated

[V,D] = eig(CM);
eigenvalues = zeros(1,306);

for k = 1:306
    eigenvalues(1,k) = D(k,k);
end

sortedEigenValues = sort(eigenvalues,'descend');  %%eigenvalues are sorted
sortedEigenVectors = zeros(306,306);

for k=1:306  %%eigenvectors are sorted according to eigenvalues
    for l=1:306
        if sortedEigenValues(1,k) == D(l,l)
            sortedEigenVectors(:,k) = V(:,l);
        end
    end
end
projectionMatrix = sortedEigenVectors(:,1:2); %% first to columns of sortedEigenVectors matrix are taken to create projection matrix
projectedMatrix = CM*projectionMatrix; %%projectedMatrix is what I am intending to use.

figure
stem(projectedMatrix(:,1),projectedMatrix(:,2)); %%plotting altogether

I have found eigenvalues and sort them and also sort eigenvectors and I take first $2$ eigenvectors and multiply it with the coveriance matrix and hope that this new matrix is what I am supposed to cluster, but I am very confused. Can you give some feedback?


texts:
I have recently started learning deep learning. In machine learning using sklearn library with n_jobs = -1 all my cpu cores are used and this speeds the grid search. Now I am trying to fit an rnn model on training data, which is taking a lot of time. Is there a way I can speed up the training?

# Initialising the RNN
regressor = Sequential()

# Adding the first LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 7)))
regressor.add(Dropout(0.2))

# Adding a second LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a third LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

# Adding a fourth LSTM layer and some Dropout regularisation
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))

# Adding the output layer
regressor.add(Dense(units = 1))

# Compiling the RNN
regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

# Fitting the RNN to the Training set
regressor.fit(X_train, y_train, epochs = 100, batch_size = 32,shuffle=False)


texts: How word embedding work for word similarity?

texts:
I am trying to apply word2vec/doc2vec to find similar sentences. First consider word2vec for word similarity. What I understand is, CBOW can be used to find most suitable word given a context, whereas Skip-gram is used to find the context given some word, so in both cases, I am getting words that co-occur frequently. But how does it work to find similar words? My intuition is, since similar words tend to occur in similar contexts, the words similarity is actually measured from the similarity among contextual/co-occuring words. In the neural net, when the vector representation for some word at the hidden layer is passed through to the output layer, it outputs probabilities of co-occuring words. So, the co-occuring words influence the vectors of some words, and since similar words have similar set of co-occuring words, their vector representations are also similar. To find the similarity, we need to extract the hidden layer weights (or vectors) for each word and measure their similarities. Do I understand it correctly?

Finally, what is a good way to find tweet text (full sentence) similarity using word2vec/doc2vec?


texts: Neural network training with known erroneous data

texts:
I am trying to make a neural network to predict some values but I know my training data contains lots of "bad" expected outputs. That is I know some of the data would train it in the wrong direction.

Would it make sense to train the network once, then remove data that the network produces a big error for (supposing that that's the "bad" data) and then retrain it with the rest? Is there some other better way to approach this?

I'm sorry I'm only starting out dealing with NNs and I don't know much of the related jargon to make the question clearer.


texts: How to extract important phrases (which may contain company name) from resume?

texts:
I have thousands of CV / resumes with me. We want to build a parser which can extract company names from resume.

So far we have tried

Maintained a list of common words present in companies (Eg. Org, Ltd, Limited, Technologies etc.) and use them to identify probable companies. But this list is limited and many times many companies don't get extracted.
Using HTML of CV we have tried to give more score to probable companies which have a certain feature (like Bold, Italics)

Since CV is not only text and we always have some structural information along with it. There should be better ways to extract information. Maybe training some model which could predict companies mentioned in the resume. We are open to any better approaches/suggestions we could incorporate into our system for better accuracy. The precision so far is really bad (less than 45%).

We have already done the segmentation of work experience in CV. So we are able to extract the segment containing work experience with very high precision.

We also have a comprehensive list of companies (millions). Although it contains duplicates and needs significant amount of cleaning. But yes we have a lot of data

Edit

Other approaches we are trying -
We try to predict the important phrases inside text using N-Grams and then mark them as probable companies. Then we do a search on companies corpus with us to find any match. How useful is this technique ? Any better approaches ?


texts: Predicting a Continuous output in a dataset with categories

texts:
Let's say this:
I'm working on a machine learning project and I'm working on a dataset with a 4250,13 as shape and it is already grouped in 7 categories!
Note that those categories can't be considered as predictor.
Here is how my data is grouped in the categories feature 'FAC':

FSEG       32.852598 % of the dataset
FSTA       19.151644 % of the dataset
FD       19.003181 % of the dataset
FM      16.076352 % of the dataset
FT      5.132556 % of the dataset
PPSE       4.814422 % of the dataset
FSDC    2.969247 % of the dataset

I have a continuous output that I want to predict 'CGPA', so the task is a regression, my goal is to predict it in each category, and the final decision will be the category where the predicted output is maximized.

My approach to deal with this problem is to sample my dataset into 7 sub-datasets and train the model in all those 7 datasets. And for a new input, predict the output in each category, and the final category will be where the predicted output is maximal.

Now I want to know, is there any way to do it in one dataset and automatically predict the category where my output is max? With a single model instead of 7?

PS: I'm using python and scikit learn

Sound like random-forest but not sure that is it ...
can someone help? Any help will be appreciated.

Here is my dataset with samples collected in each category. My final output is called CGPA and the category label is 'FAC'.


texts:
In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters simulate the Light Curve the best.

At the moment I do it with the Cross-Correlation function from numpy. But I am not sure if that is the best option, due to the fact that the Light Curve with the highest Cross-Correlation Coefficient not always looks like the best fit/simulation compared to other simulations with a lower CC-Coefficient.
Is there a another way to measure for similarity? I read something about the Chi-Square Statistics, but I am not sure how that works and how this could be applied to my problem.

The observation data I use is not evenly binned, so I used the interpolation function of Scipy. Should I also smooth the observation data or would I lose true features of my data. I thought about using the savitzky golay smoothing.

At the moment I am using a brute force method to try out all possible parameters and simulated the corresponding light curve. The problem is this takes a lot of time with 20 parameters. The parameters are more or less dependent on each other. So I cant use a least-square fit method, because there are multiple possible minimas. Is there a simple method that I overlooked. Or is a restricted brute force fit my best option?

In the picture below you'll see one plot with the simulation and the observaton data.

Thanks for all suggestions.

![Red: Simulation Curve, Blue: Observed, interpolated Data


texts:
I want to train an ngram language model

Let's say I have the following corpus:

The sliding cat is not able to dance
He is only able to slide
Because obviously he is the sliding cat

I am planning to use tf.data.Dataset to feed my model, which is fine

But I don't know if it is better to use a sliding window to iterate through my copus or simply feed my corpus n words at a time

Using a sliding window, my model (assuming a bigram) will see:

The sliding
sliding cat
cat is
is not
...

Going n word at a time:

The sliding
cat is
not able
...

I'd appreciate any recommandation, thanks


texts:
Suppose I have 1-D data which has some outliers, I want to normalize the data to be in the range [0,1].
I tried calculating the maximum value and the minimum value as follows:
q1,q2,q3 = quartiles of the data
max = q3 + (q3-q1)*1.5
min = q1 - (q3-q1)*1.5

I used the above approach because I have read that data above maximum or data less than the minimum (as calculated above is noise).
My question is: whatever I am doing, is it correct or is there any other way to achieve good results?
Thank you for helping.


texts: What should I do with a test and train set not having same classes?

texts:
Assume we have a train dataset with 5 classes a,b,c,d,e but the test dataset have only d and e with extra class f not present in train set. If i want to do machine learning with this datset in weka or python what should I do with this dataset? should I transform them and how should I do it? Should I remove classes form train set which are not in test set?
(it's a multi-class classification, and the letters a-f are the target classes)


texts: Why does the generator produce output in a different scale than the training sample?

texts:
I am currently trying to train a GAN (based on proGAN) to produce images in a Vaporwave-style which is quite distinct.

The results so far have been underwhelming, which I suspect might be due to a very small training sample (only 909 images).

However in trying to optimize the model and my code I stumbled upon the following problem:

The output of the generator model has a different scale than my training samples!

My input data are image arrays scaled from 0-1, however the generator produces arrays scaled from -1 to 2!

Why does this happen? And is it a problem at all?

My code

You can find my code and the output so far here:

Vaporgan - Kaggle Notebook


texts: Keras ANN Trained Model's Accuracy change on prediction

texts:
I have trained an ANN Binary classifier using Keras. It gives 90% accuracy.
After testing when I predict same data again but pass only one class then accuracy decreases to 40%.
I have figured out that if I pass mixed classes while predicting then it will give me around 90%  accuracy and if I pass data points of only one class then accuracy decreases .As I increase the data points of other class as well then accuracy increases.
Long in short.
CASE 1:
100 samples from class 0,
100 sample from class 1,
on predicting using trained model Accuracy = 90%

CASE 2:
Same 100 samples from class 0 passes to same trained Models give me 40% accuracy.
Why accuracy changes ?

EDIT: I'm performing Standardization before predicting every time which effects the predictions. How to handle this case?
Any suggestions would be much appreciated. Thanks


texts: Resources for text classification algorithms

texts:
I'm mining raw Facebook comments (irrelevant) and i am looking for an algorithm that can classify their context as negative/positive/neutral. So you can think of the output in the form of two columns. First column the comment (exists), and second column its classification. If possible i'd like for the algorithm to be able to process the sentence as is (e.g. not having to remove stop words etc). Can someone recommend any resources that i can refer to? I have no problem doing this in either Python or R.
Thanks


texts: Neural network breaks if hidden layers are increased

texts:
I am trying to make my neural network as dynamic as possible. For example, I want to be able to increase hidden_layers number. It works so far with one hidden layer, but if I increase it to two, it instantly breaks. There is a problem when the backpropagation is reaching the layer between Inputs <- 1st_hidden. Here is my code. I am trying to solve XOR problem by providing this to my NN:

inputs  = np.array([[1, 0], [0, 1], [1, 1], [0, 0]])
targets = np.array([[1], [1], [0], [0]])

Can anyone help me? I can not seem to solve it.

nn = NeuralNetwork(input_nodes = 2, hidden_layers = 1, hidden_nodes = 3, output_nodes = 1)

I initialize my neural network:

class NeuralNetwork:
def __init__(self, input_nodes, hidden_layers, hidden_nodes, output_nodes):

    self.input_nodes   = input_nodes
    self.hidden_layers = hidden_layers
    self.hidden_nodes  = hidden_nodes
    self.output_nodes  = output_nodes
    self.layers_count  = self.hidden_layers + 1

    self.weights       = []
    self.bias          = []

    min_value = -2
    max_value =  2

    # Initialize weights and bias matrices and place them in a list
    # Matrix shape within lists
    # [(2,2), (2,2), (1,2)]

    for i in range(self.layers_count):
        if i == 0:
            self.weights.append(np.random.uniform(min_value, max_value, (self.hidden_nodes, self.input_nodes)))
            self.bias.append(np.random.uniform(min_value, max_value, (self.hidden_nodes,)))
        elif i == self.hidden_layers:
            self.weights.append(np.random.uniform(min_value, max_value, (self.output_nodes, self.hidden_nodes)))
            self.bias.append(np.random.uniform(min_value, max_value, (self.output_nodes,)))
        else:
            self.weights.append(np.random.uniform(min_value, max_value, (self.hidden_nodes, self.hidden_nodes)))
            self.bias.append(np.random.uniform(min_value, max_value, (self.hidden_nodes,)))

    self.learning_rate = 0.1

def train(self, input_array, targets):

    inputs = np.transpose(input_array)

    applied_activation = nn.feedforward(inputs)

    output_errors = np.subtract(targets, applied_activation[-1])

    # Backpropogation
    for i in reversed(range(self.layers_count)):
        delta    = np.ndarray((self.input_nodes,))
        gradient = np.ndarray((self.hidden_layers,))

        if i == self.hidden_layers:
            errors = output_errors
            delta, gradient = calculate_weights_deltas(applied_activation[i], output_errors, applied_activation[i-1])
        elif i == 0:
            # IT BREAKS HERE
            errors          = calculate_errors(self.weights[i+1], output_errors)
            delta, gradient = calculate_weights_deltas(applied_activation[i], errors, inputs)
        else:
            errors          = calculate_errors(self.weights[i+1], output_errors)
            delta, gradient = calculate_weights_deltas(applied_activation[i], errors, applied_activation[i-1])

        delta, gradient = calculate_weights_deltas(applied_activation[i], errors, applied_activation[i-1])
        # Adjust weights by deltas
        self.weights[i] = np.add(self.weights[i], delta)
        # Adjust the bias by its deltas
        self.bias[i] = np.add(self.bias[i], gradient)

The error I get is the following:

            elif i == 0:
                errors          = calculate_errors(self.weights[i+1], output_errors)

            ValueError: shapes (3,3) and (1,) not aligned: 3 (dim 1) != 1 (dim 0)


texts: What are the best ways to tune multiple parameters?

texts:
When building a model in Machine Learning, it's more than common to have several "parameters" (I'm thinking of real parameter like the step of gradient descent, or things like features) to tune. We validate these parameters on a validating set.

My question is: what is the best way of tuning these multiple parameters? For example, let say we have 3 parameters A, B and C that take 3 values each:

A = [ A1, A2, A3 ]
B = [ B1, B2, B3 ]
C = [ C1, C2, C3 ]

Two methods come to my mind.

Method 1:

Vary all the parameters at the same time and test different combinations randomly, such as:

Test1 = [A1,B1,C1]
Test2 = [A2,B2,C2]
Test3 = [A3,B3,C3]
Test4 = [A1,B2,C3]
Test5= [A3,B1,C2]
etc..

Method 2:

Fix all the parameters except one:
 - TestA1 = [A1,B1,C1]
 - TestA2 = [A2,B1,C1]
 - TestA3 = [A3,B1,C1]
In that way, we can find the best value for parameter A, then we fix this value and use it to find the best value for B, and finally the best for C.

It seems more logical to me to use the method 2 which seems more organized. But may be we will miss a combination which can be found only in method 1 that doesn’t appear in method 2, such as [A1,B2,C3] for example.

Which method is the best? Is there another method more accurate for tuning multiple parameters?

Thanks in advance.

Regards.


texts:
I am trying to build a regressor for a dataset which gives info about students' school performance and the probability of getting admitted in the University of their choice.

The first 5 observations look like this :

    GRE_Score  TOEFL_Score  Uni_Rating  LOR   CGPA  Research Chance_of_admit
0      337         118           4      4.5   9.65     1          0.92
1      324         107           4      4.5   8.87     1          0.76
2      316         104           3      3.5   8.00     1          0.72
3      322         110           3      2.5   8.67     1          0.80
4      314         103           2      3.0   8.21     0          0.65

I have build the following regressors so far : linear regressor, knn regressor and a recurrent neural network. ( I will try a few more later. )

So far, in order to chose among my models, I used the "score" method on the test set for the first two regressors ( it returns the $R^2$ for each one of them ) and the "evaluate" method on the test set for the network ( it returns the MeanSquaredError ).

So, keeping in mind that $R^2$ and MeanSquaredError have different formulas, how can I compare my network with the other two models ??

Any help is much appreciated.


texts: Unbiased Predictions for all Distinct Training Subsets

texts:
Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ exist. For example, if I'm trying to use blood pressure ($X$) to predict resting heart rate ($y$), I might have the dataset $\left((X_i, y_i) = (120, 60), (130, 58), (118, 72), ... \right)$. Furthermore, using assumptions about the heart rate and blood pressure of people, I can say $\chi = [50, 90] $ and $\zeta = [95, 180]$ where "[]" is a closed interval in $\mathbb{R}$. In otherwords, I know that nobody has a resting heart rate lower than 50 and higher than 90, and I also know nobody has a resting blood pressure lower than 95 and greater than 180.
Here's the question. Suppose I partition my set $\zeta$ into $k$ distinct subsets $\zeta_1, \zeta_2, ... \zeta_k$. These subsets, for example, might be $\zeta_1 = [95, 130], \zeta_2 = [130,160], \zeta_3 = [160,180]$ (i.e. different groups of people based on blood pressure: low, medium and high blood pressure). Suppose for each subset of labels, I want my machine learning model to make unbiased predictions. I should note that a model that makes unbiased predictions on $\zeta$ as a whole (trained with a regular RMSE loss function) can still make biased predictions on each $\zeta_i$ (it might overpredict the blood pressure of people with low blood pressure and underpredict the blood pressure of people with high blood pressure, for example). What is the best loss function to use to prevent this from happening, and make unbiased predictions in each label region $\zeta_i$?

I've tried training with the loss function $L = L_0 + \alpha \sum_{j=1}^k \frac{1}{|\zeta_j|}\sum_{y_i \in \zeta_j} (\hat{y}_i - y_i)$ where $L_0$ is the MSE loss function and $\hat{y}$ are the predictions, and while this does a decent job, it can often lead to unstable training conditions. In addition, it requires using many instances in a single gradient descent step (note the summation over instances).


texts: Using BERT for co-reference resolving, what's the loss function?

texts:
I'm working my way around using BERT for co-reference resolving. I'm following this highly-cited paper BERT for Coreference Resolution: Baselines and Analysis (https://arxiv.org/pdf/1908.09091.pdf). I have following questions, the details can't be found easily from the paper, hope you guys help me out.
What’s the input? is it antecedents + parapraph?
What’s the output? clusters <mention, antecedent> ?
More importantly What’s the loss function?
For comparison, in another highly-cited paper by [Clark .et al] using Reinforcement Learning, it's very clear about what reward function is. https://cs.stanford.edu/people/kevclark/resources/clark-manning-emnlp2016-deep.pdf


texts: Detecting punch type using CoreML Activity classifier

texts:
I’m trying to train an activity classifier (made by Apple) to detect with kind of punch is thrown during boxing training.
Accelerations are taken directly from an Arduino Nano 33 using Bluetooth low energy. I have acceleration in 3 axes and gyroscope acceleration in 3 axes at a 100Hz sample rate.
A punch depending on the experience of the boxer lasts from 0.3 to 0.7 seconds
To get accelerations I have a special configuration on my application that produces CSV data files based on the specific program that must be followed. No activity, jab, left hook, left uppercut. The application tells the user which punch to throw and creates the relative file.
So far I’ve managed to get some data and test some results, results are not bad, but I have some concerns about how to improve the training.

At the moment I’m using raw acceleration values, will it be better to
filter them?
This is something that really makes me crazy. In a WWDC
video an engineer shows that she trained a machine to understand which kind of a frisbee launch had been thrown. In the brief explanation
she tells that in each file, used for training, she has at least 3
launches for a specific type of lunch and of course more files for each. Wouldn't be better to register just one launch and repeat that procedure for more files. Since core ml training is based on the directory structure is like she is saying that 3 launches of that kind are associated with a specific activity when it must be detected it’s just one throw. Which approach is better? More files with just one activity registered repeated overtime or more files with the same activity repeated more time in one file and then the same over time?


texts: What are best activation and regularization method for LSTM?

texts:
In Keras there are:

activation: Activation function to use (see activations). Default: hyperbolic tangent (tanh). If you pass None, no activation is applied (ie. "linear" activation: a(x) = x).
recurrent_activation: Activation function to use for the recurrent step (see activations). Default: hard sigmoid (hard_sigmoid). If you pass None, no activation is applied (ie. "linear" activation: a(x) = x).

For activation functions and

kernel_regularizer: Regularizer function applied to the kernel weights matrix (see regularizer).

recurrent_regularizer: Regularizer function applied to the recurrent_kernel weights matrix (see regularizer).

bias_regularizer: Regularizer function applied to the bias vector (see regularizer).

activity_regularizer: Regularizer function applied to the output of the layer (its "activation").

For regularization methods that we can choose from:

keras.regularizers.l1(0.)
keras.regularizers.l2(0.)
keras.regularizers.l1_l2(l1=0.01, l2=0.01)

Or define our regularization method. It also is possible to add dropout() layer after our LSTM layers:

keras.layers.Dropout(rate, noise_shape=None, seed=None)

And maybe the other methods that I don't know. But which activation and regularizaion is more suitable for LSTM? Is it a good idea to add regularization to the LSTM or it hast regularization property inside it's cell by default?


texts: How handle the add of a new feature to the dataset?

texts:
Let's say that you trained a model (eg. Random Forest) on a dataset with ten features (or columns).

Now you add one or more features to the dataset. You need the information brought from these new features so you want to enrich your model.

What happens to the data collected till now? How can I use it? Those data have just ten features. How can I use them to train the new model?


texts: How are ANN's, RNN's related to logistic regression and CRF's?

texts:
This question is about placing the classes of neural networks in perspective to other models.

In "An Introduction to Conditional Random Fields" by Sutton and McCallum, the following figure is presented:

It shows that Naive Bayes and Logistic Regression form a generative/discriminative pair and that linear-chain CRFs are a natural extension of logistic regression to sequences.

My question: is it possible to extend this figure to also contain (certain kinds) of neural networks? For example, a plain feedforward neural network can be seen as multiple stacked layers of logistic regressions with activation functions. Can we then say that linear-chain CRF's in this class are a specific kind of Recurrent Neural Networks (RNN's)?


texts:
I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.

In essence, we will be consuming real-time coordinates from vehicles and users and performing analytics and machine learning on this incoming data. This processing can be rather intensive as we try predict the ETAs of this entities matched to historical data and static paths.

The approach they want to take is using the latest and most powerful technology stack, that being Hadoop, Storm etc to process these coordinates. Problem is that no-one in the team has implemented such a system and only has had the last month or so to skill up on it.

My belief is that a safer approach would be to use NoSQL storage such as "Azure Table Storage" in an event based system to achieve the same result in less time. To me it's the agile approach, as this is a system that we are familiar with. Then if the demand warrants it, we can look at implementing Hadoop in the future.

I haven't done a significant amount of research in this field, so would appreciate your input.

Questions:

How many tracking entities (sending coordinates every 10 seconds) would warrant Hadoop?
Would it be easy to initially start off with a simpler approach such as "Azure Table Storage" then onto Hadoop at a later point?
If you had to estimate, how long would you say a team of 3 developers would take to implement a basic Hadoop/Storm system?
Is Hadoop necessary to invest from the get go as we will quickly incur major costs?

I know these are vague questions, but I want to make sure we aren't going to invest unnecessary resources with a deadline coming up.


texts: Do you know of any machine learning add-ins that I could use within Excel? For example I would like to be able to select a range of data and use that for training purposes and then use another sheet for getting the results of different learning algorithms.


texts:
I'm trying to build a 5 input-5 output model using LSTM, where all the outputs are the same features as the inputs, predicted in the future.

My question is: is it better to build 5 models, each with the same 5 inputs, but predicting just 1 of the 5 sequences at a time, or is it the same as building 1 model predicting all 5 sequences?
In other words, is the accuracy per predicted sequence going to be higher with 5 separate models or will it be the same as 1 model with 5 outputs.

The reason for my confusion is that, in the case of the multiple output model, the hidden layer will be the same; so how would the algorithm go about optimizing the weights so as to minimize error for all output sequences?


texts: What does n means in neural network neuron output?

texts:
I've found this equation that explains the output of a neuron in a MLP network:
$y(n) = f(\mathbf{w}^T \mathbf{x}(n) + b)$
I can understand the general context, but since i have no background with mathematical notation, i don't understand what the $(n)$ parameter means (e.g. $y(n)$, $x(n)$). Is it sort of a temporal or sample index? I've seen this notation in other machine learning subjects, but didn't find an explanation.


texts: Finding number of common elements between different columns of a DataFrame

texts:
I have a dataframe which has almost 70-80 columns. Each column consists of 100-150 rows in which values are stored as strings.

I would like to find, for each column, what is the number of common elements present in the rest of the columns of the DataFrame.

For example: say I have a dataframe like:

       0      1      2        3
0    cat     ox   bull    horse
1   lion    dog    cat    zebra
2   bull     ox  horse    tiger
3  horse  eagle   bull  giraffe

Starting with first column, I need answer as 2,3 as 2 seems to have most number of common elements with the column 0 after that column 3 as it has comparatively less number of common elements.


texts:
Here, I have the single hidden layer multi-layer perceptron (Input: 4 units, hidden unit: 3 units, output: 1 unit).

I want to do pruning before a training process with Keras.
Concretely, I just want to remove the connection between the input units and hidden units. (so I don't remove the hidden units.)

Would you tell me how to implement this code?
Now I am thinking that the skip connection with merge and concat is one of the solution, however, it is not smart way when the number of the units and layers are increased.


texts: Time series feature extraction from raw sensor data for classification?

texts:
I have a tabular raw data from sensors with associated label and i want to extract the time series features like mean,max,min and std from the data all the sensor data and form another table or export to csv file so that i can do classification task on that data.

Data table


texts: How many people can use a single Hadoop cluster at one time?

texts:
I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a class of 12 to 24 students.

Also, I am wondering if anyone could recommend whether the specifications each for the nodes on a high-end educational-level cluster should be the same as for any other production-level cluster ( ie. 64G-128G RAM, 24TB hard drive space, 8 cores, etc.). I believe that the dataset sizes that students would be using will range between 20MB minimum to about 0.500TB maximum, I imagine that we will ultimately be working on real problems and datasets, even if they are not exactly considered to be big data.


texts: Differences between class_weight and scale_pos weight in LightGBM

texts:
I have a very imbalanced dataset with the ratio of the positive samples to the negative samples being 1:496. The scoring metric is the f1 score and my desired model is LightGBM. I am using the sklearn implementation of LightGBM.

I have read the docs on the class_weight parameter in LightGBM:

  class_weight : dict, 'balanced' or None, optional (default=None) Weights associated with classes in the form {class_label: weight}. Use this parameter only for multi-class classification task; for binary classification task you may use is_unbalance or scale_pos_weight parameters. The 'balanced' mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)). If None, all classes are supposed to have weight one. Note that these weights will be multiplied with sample_weight (passed through the fit method) if sample_weight is specified.

On using the class_weight parameter on my dataset, which is a binary classification problem, I got a much better score (0.7899) than when I used the recommended scale_pos_weight parameter (0.2388). Should I use the class_weight parameter or the scale_pos_weight parameter to balance the classes?


texts: Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series')

texts:
Several terms are used to describe the structure of instances of a dataset. For example, an instance can be 'univariate' if it depends on only one variable or 'multivariate' if it depends on 2 or more variables. I wonder if there are technical terms to distinguish the structure of a, let's call it a 'tuple', and a 'time series' or 'sequence'.
Here are two examples that I am trying to describe or distinguish structurally:

The left example is a time series, where all 11 samples belong to one instance. The right example is a collection of 'tuples' (if that is the correct term), where each row represents a separate instance.
I always refer to the term 'tuple' here, because the concept seems similar to me with the record in a database or the representation of points from mathematics. I am not sure if 'time series' and 'tuple' are the correct terms to describe how many samples an instance consists of. The terms should be analogous to uni/multivariate to describe the number of features in an instance.
Hence my question:
If univariate/multivariate describes an instance in the horizontal, what terms are used to describe an instance vertically?


texts: XGBoost parameters that will overweight errors for a specific decile

texts:
My goal is to make predictions about the bottom decile of my dataset. If I can predict the bottom decile values well (ie. minimize the errors there) then I am not worried about making accurate predictions of the rest of the dataset.
I have tried a classification model, but the problem is that I can't punish big errors (where it predicts bottom decile, but actual value is top decile). This type of error would be very costly for me. Therefore I am assuming I must do a regression instead of a  classifcation?
When I switch to a regression, is there a way of changing parameters such as objective function or evaluation metric to overweight the importance of the predictions in the bottom decile?


texts: Training a floor detection model: use full room images or only the cropped floor?

texts:
I'm trying to build a floor type image classification model.There's an open dataset called OpenSurfaces containing images segmented by the material type of every item appearing on a room.

Something like this:

I thought that using this dataset to train a floor detection model would be a good thing, so I wrote a script to extract the materials I'd like to detect (wood, tile, carpet, marble, stone, ...). These are some examples of the images I've got as a result of the script:

Wood material:

Tile:

Carpet:

Then I trained a CNN but I've only got something like 70% of accuracy and I don't really now if I'm going along the right path. Is it better to train a model with the pictures I extracted or it'd be better to train it with the full room image, not the segmented part?

I'm quite lost, so any guidance will be greatly appreciated.


texts: Is there a universal convergence rate when stacking models/experts?

texts: It's fairly common to see people stacking different models when chasing marginal gains in contexts such as Kaggle competitions or the Netflix challenge. I would like to know about the mathematics behind it. When stacking experts, under the assumptions that they are independent and each one brings a new knowledge/information/signal to the stack, what is the expected convergence rate in terms of KL-divergence, or MSE, or binary cross-entropy? Are there universal behaviours or convergence rates known in this domain?


texts: Generating a set of Rules that defines a Group of users ( Multivariate Analysis )

texts:
I am working on a Multivariate Analysis on my dataset, I have a large dataset of 50k users let's call it (Loyal Users) with various numerical attributes +13 tracking the user's activity and events within a timeframe.
I want to come up with Set of rules to define this group of users so that i can come up with a common pattern.
This is an example of the data that i am working on

username        Like     Share           Select     Send  Verify  Register

Alex            524      329             203        787    1       1
Kristina        197      186             286        120    0       1

Here is an example of rules that i want to make :
If Share > 23 and Verify =1 and Like >120  then Loyal User

would that be possible after doing a principal components analysis (PCA) and linear discriminant analysis (LDA) ? And how can i do that ?

Any ideas would be great.


texts: Extracting vectors of FastText own model to use it on a NN

texts:
I have trained my own model of fasttext using the pretrained model of English available on their website with the next code:

from gensim.models.fasttext import load_facebook_model

mod = load_facebook_model('fasttext/cc.en.300.bin')
mod.build_vocab(sentences=list(df_train.text), update = True)
mod.train(sentences=list(df_train.tex), total_examples=len(df_train.text), epochs=10)

Now I will like to extract the vectors of this embedding to train a LSTM neural network with it. Any tip on how to do so?

Thanks in advance.


texts: Should images with multiple objects of the same class be used as training sample for multi-classes object detection models?

texts: Let's say the model try to detect all the grapes on a branch of grapes. Can I use images of a grape branch with all the grapes labeled as a training sample? Will it affect the quality of the RPN ? Is it better to train on images which a single or a few grapes in it, since the classification is done separately on each slide of the image?


texts: Different results by running single model and from the VotingClassifier?

texts:
I have a situation where my model predictions differ when I use the prediction from a single classifier (e.g. SVM) and the sklearn.ensemble.VotingClassifier with only option the same model.
vot = VotingClassifier(estimators=[ ('SVC', model_SVC)], voting='soft')
probas = [c.fit(X, y).predict_proba(X) for c in (model_SVC, vot)]

Then if I return the results
for i in range(2):
    print(probas[i][0], unique_types[np.argmax(prb)])

I get for example (array of probabilities for a 7-class problem):
SVM: [0.01703915 0.00290338 0.00528683 0.00578544 0.89824208 0.01244273
 0.0583004 ] Class_4
Vot: [0.01604109 0.00273581 0.00497365 0.00547579 0.90292339 0.00859484
 0.05925544] Class_4
...
...
SVM: [0.27039153 0.03815588 0.30307504 0.02600352 0.03358154 0.26581178
 0.06298071] Class_2
Vot: [0.23473323 0.03981095 0.25290641 0.02860963 0.03515882 0.34675459
 0.06202637] Class_5

Why do I get slightly different values? In the first case the differences are small and they do not affect the end classification (although I wouldn't expect that much of a difference!) but on the second case the classification result changes...


texts:
Suppose I have a dataset of size(10000, 45). One of the features in the dataset is activity_type in which the values vary from 1 to 15 as shown below:
df = pd.read_csv('actTrain.csv')
df['activity_type'].head()

The output of the above code is as:
0    1
1    1
2    2
3    1
4    3
Name: activity_type, dtype: int64

Will encoding the activity_type in the above code using OneHotEncoder in sklearn improve the model in anyway? Is it necessary to encode that feature? And if yes, which one should I choose : LabelEncoder or OneHotEnocder?


texts: Can someone direct me to research papers that have tried to understand word embeddings?


texts: Where can I get Insurance claim data for practicing NLP(Natural Language) processing?

texts:
I am looking for specifically Insurance dataset for practicing Machine Learning & NLP, but unable to find much in kaggle, udemy or other websites. Is there a way to get that dataset or any website that stores this.

I am mainly looking for NLP practice for automatic claim generation or other insurance activities, please suggest the approach


texts: Binary Classification of Numeric Sequences with Keras and LSTMs


texts: Predict if user will come the next day or not

texts:
im novice in data science field,
i have a problem statement: to predict which users will come online the next day.

i have two datasets of the user:

1st dataset of the user ( master table ) which has the overal information of the user:

uid totlike totview totshare last_social_activity   last_service_activity   last_active_date    most_active_day   is_active( True if(last_active_day - present date )< 30 )
1   200     500     115      like                   see_news_feed           11/12/2017           Monday             TRUE
2   300     600     2223     share                  see_dashboard           11/13/2017           Monday             TRUE
4   500     237     311      comment                see_news_feed           11/14/2016           Wednesday          FALSE

totlike - total like
totview - total view
totshare- total share

2nd data set of the user ( everyday activity ): Historical Data of user activity with time stamp.

date       uid  total_like  total_view  total_share last_social_activity    last_service_activity
11/4/2016   1     10        20         1            like                        see_news_feed
11/4/2016   2      9        25         2            share                       see_dashboard
11/4/2016   4      1        23         3            comment                     see_news_feed
11/4/2016   3      3        14         4            like                        see_dashboard
11/8/2016   1      4        14         5            comment                     see_news_feed

what is the right way to predict if the user will come tomrrow?


texts:
is it a good idea to use k-fold cross-validation in the recurrent neural network (RNN) to alleviate overfitting?

A potential solution could be L2 / Dropout Regularization but it might kill RNN performance as discussed here. This solution can affect the ability of RNNs to learn and retain information for longer time.
My dataset is strictly based on time series i.e auto-correlated with time and depends on the order of events. With standard k-fold cross-validation, it leaves out some part of the data, trains the model on the rest while deteriorating the time-series order. What can be an alternate solution?


texts:
In Orange3 while only using its widgets, without writing Python code, I’ve implemented the following typical machine learning processes

Train a training set, (1 file)
Validating a validation set (e.g. using Naiive Bayes and kNN), and
Testing the model with a test set.(another file)

I keep getting:

  test and train datasets have different target variables

error message. This is despite using exact same formats and names (with different values of course). What did I do wrong?


texts: Practical problems in anomaly detection where the number of normal data is extremely high compared to abnormal data

texts:
If the ratio of abnormal data is about 1 to 10,000 normal data, even if the true negative rate is 99%, there will be 100 false positive data, and the precision( TP/(TP+FP) ) will be low.
If this kind of anomaly detection is to be put to practical use, I think it is necessary to create a model with a fairly high prediction accuracy.
How do the actual examples of anomaly detection in the world deal with this problem?
Is it difficult to put anomaly detection to practical use for difficult problems that do not allow for high prediction accuracy, or for problems where the number of anomalies is too small?


texts: Training loss stuck in the starting epochs but then starts decreasing. What could be the reason for it?

texts: I am training a model where I found a unique problem that for starting 4 epochs, my loss did not change with the epochs but after that, it started changing. Could it be because of the high learning rate, local minima or something else like some regularisation parameter is high??


texts:
I'm learning the linear regression now. I used R to build linear model upon a set of train model and try to predict() data based on the test data.

My question: I understand how train data gets collected. But what's with the test data? How did it get collected? Is the test data built, or is it collected, or is it predicted or what?

PS: I'm learning data science by method of self-learning, so I lack the structural in my knowledge. I might know something on one place while lack the knowledge at another place. Please forgive and guide. Thanks. :)


texts: Possible for batch size of neural network to be too small?

texts: We know that when a batch size is too large, the model might not be able to converge. But what is the drawback of having a batch size be too small (say batch_size = 1) other than taking a long time to train? A batch size of 32 is commonly used and referred to as "small," but why don't we use a smaller batch size of 1 to guarantee convergence?


texts:
Question

Why lmplot show the line but regplot does not?

lmplot

import seaborn as sns
import matplotlib.pyplot as plt

sns.lmplot(x="Value", y="dollar_price", data=merged_df, height=8, aspect=1.5)

regplot

# seaborn.regplot() returns matplotlib.Axes object
plt.rcParams['figure.figsize'] = (15,10)
ax = sns.regplot(x="Value", y="dollar_price", data=merged_df, fit_reg=False)
ax.set_xlabel("GDP per capita (constant 2000 US$) 2017")
ax.set_ylabel("BigMac index (US$)")

# Label the country code for those who demonstrate extreme BigMac index
for row in merged_df.itertuples():
    ax.text(row.Value,row.dollar_price+0.1,row.country)

Documentation

seaborn.regplot documentation page shows the line. So I suppose there is something missing?

>>> import seaborn as sns; sns.set(color_codes=True)
>>> tips = sns.load_dataset("tips")
>>> ax = sns.regplot(x="total_bill", y="tip", data=tips)


texts: Why is there a big drop off in performance in my GBM?

texts:
I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 observations that I use for a final validation. All 10,200 observations were obtained in similar fashion.

I carried out a grid search with 5-fold cross-validation in order to obtain a suitable set of hyper parameters. The results for my test set are good and very stable. However, there is a big drop off in performance when use my final validation data.

Results for the test set

->  Precision: 0.836 / Recall: 0.629 / Accuracy 0.874

Results for the final validation set

->  Precision: 0.149 / Recall: 0.725 / Accuracy 0.484

At first I thought this could be the caused by data leakage, but even after removing "suspicious" features, there is still a big drop off when comparing the test results with the final validation results.

Surely I'm doing something wrong, but I'm at a loss as to what exactly. Here are the relevant lines of code (nothing fancy):

> X = pd.read_csv('train_test.csv')
> y = X.pop('Target')
> X_final = pd.read_csv('final_validation.csv')
> y_final = X_final.pop('Target')

> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

> gb = GradientBoostingClassifier(n_estimators=300, max_depth=5, learning_rate=0.2)
> gb_model = gb.fit(X_train, y_train)

> # test set
> y_pred = gb_model.predict(X_test)
> precision, recall, fscore, support = score(y_test, y_pred, average='binary')

> # final validation set
> y_hat = gb_model.predict(X_final)
> precision, recall, fscore, support = score(y_final, y_hat, average='binary')

Any thoughts?


texts: remove special character in a List or String

texts:
Input_String is Text_Corpus of Jane Austen Book

output Should be :
['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

But getting this Output :
['to', 'be,', 'or', 'not', 'to', 'be:', 'that', 'is', 'the', 'question!']


texts: Can the attention mask hold values between 0 and 1?

texts:
I am new to attention-based models and wanted to understand more about the attention mask in NLP models.

attention_mask: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
input sequence length in the current batch. It's the mask that we typically use for attention when
a batch has varying length sentences.

So a normal attention mask is supposed to look like this, for a particular sequence of length 5 (with last 2 tokens padded) --> [1,1,1,0,0].
But can we have attention mask like this --> [1, 0.8, 0.6, 0, 0] where values would be between (0 and 1) to indicate that we want to pay attention to those tokens, but it's result wouldn't be completely effective on the model's result due to it's lower attention weights (kinda of like dealing with class imbalance where we weight out certain classes to deal with imbalance).
Is this approach possible? is there some other way to have the model not use the information presented by some tokens completely?


texts:
Assume that "1,2,3" are the ids of users, active means that person visited the stackoverflow in last one month (0=passive, 1=active), and there are positive and negative votes.

id  question       votes                 active
 1     1        -1, +1, -1, -1, -1         0
 1     2        -1, +1, -1, -1, +1         0
 2     1        +1, +1, -1, -1             0
 3     1        +1, +1, +1, -1, +1         1
 3     2        +1, +1, -1, +1, +1, +1     1
 3     3        -1, +1                     1

I want to know what makes the users stop using stackoverflow. Think that, I have already calculate the how many times did they get negative votes, total vote, average vote for each question...

I wonder what kind of information could I get from these sequences. I want to find something like this: these users who are passive have two negative votes sequentially. For example, one positive vote after two negative votes in the second question of user 1, doesn't prevent the user churn. User 3 doesn't have any 2 negative votes sequentially in any of his questions. Hence he is still active.

I'm looking for something like PrefixSpan Algorithm but order is important for me. I mean, I can't write the sequences like

<(-1 +1 -1 -1 -1) (-1 +1 -1 -1 +1 )>

or

<(-1) (+1) (-1) (-1) (-1) (-1) (+1) (-1) (-1) (+1 )>.

Because the first one loses the order, and the second one jumbled the questions together. Is there any algorithm to find these sequences which is common for churners?


texts:
I'm actually reading this tutorial about deepLearning and in particular about Logistic Regression.

I don't get why it first says to optimize logistic regression taking the max Probability and after using the Log loss function ?

Sorry you can explain me the point where him will use the Argmax and where Will use the Loss ?

you would not need only 1 of this 2 ?


texts: Working as data scientist for a nonprofit company

texts:
Hello I work as data scientist for a private company.
I am interested in working for a nonprofit company, such as a research institute (public or private) or a company that takes care of  issues such as  environment, public health,  social improvements. Even an internet company like Wikipedia can be interesting.
Does anybody know if nonprofit companies hire data scientists?


texts: how to evaluate the performance of a recommender system with single recommendation

texts:
Say we have a recommender system in production which recommends 1 our of N items according to some internal algorithm f given inputs Xi for each user i, let's assume f is a black box model. We have historical data recording user's reaction yi (click, dismiss, conversion, could be delayed), to the recommendation.
How should I evaluate the performance of this recommender system offline given the data available?


texts: Detection of a very large object (vehicle) in a single static camera

texts:
I'm trying to make a deep learning model for an automated detection and classification of vehicles (i.e cars, bus, trucks, etc) in a paid parking lot, but since the cameras are too up close for a large vehicle to fit in a single frame, it is creating an issue.
I thought of extracting the moving vehicle(s) from the video and classify it in real-time using image stitching technique but I'm kinda new to this image stitching thing so any help will be appreciated.
Input will be a video file from a single static camera and vehicles may or may not fit in single frame. (Example: the cameras you see in a toll booth or a parking lot)


texts: How to code a simple forward propagation of recurrent neural networks?

texts:
I know the theory behind recurrent neural networks or RNN but I am confused about its implementation. This is an rnn equation I got from the web,

I tried to code the forward propagation alone in python's numpy

import numpy as np

outputs = 5
inputs = 3

# Input value
# (batch_size,seq_len, vocab_len)
X = np.ones((10,3,3))

# Initializing rnn weights and hidden states
Wxh = np.random.rand(outputs,inputs)
Whh = np.random.rand(outputs,inputs)
Why = np.random.rand(outputs,inputs)
h = np.zeros((1,inputs))

# Forward propagation
def rnn(x,h):
    h = np.tanh(np.dot(Whh,h.T) + np.dot(Wxh,x.T))
    y = np.dot(Why,h.T)
    return y,h

for i in X:
    _,h = rnn(i,h)

But I get a broadcasting error. How do we implement the forward propogation of rnn?


texts: First perform data augmentation or normalization?

texts: Should I first perform data augmentation or normalization in deep learning? I am mainly interested in 2D and 3D input data. In tutorials that I have seen so far the data augmentation always comes first. Is there a (mathematical) reason for that? Would it also work vice versa?


texts:
I'm trying to train a model in keras and I'm using ModelCheckpoint to save the best model according to a monitored validation metric (in my case the Jaccard index).

While I can see the model improving in tensorboard, when I try to load the weights and evaluate the model it isn't working at all. Furthermore, by the timestamp on the file where the weights are supposed to be stored, I can tell that they are not being saved at all. The timestamp corresponds roughly to the time I started training.

Has anyone encountered such a problem before?


texts: How to download BAIR action free robot pushing dataset?

texts:
I'm trying to download BAIR action free robot pushing dataset. I tried downloading from here. In browser, it shows its size is 30GB, but downloads some data and then fails. I tried multiple attempts with no success. Then I tried to download using wget

wget http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar

Even with this, it shows total size is 30GB, but after downloading some 199MB, it ended saying download is complete

wget http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar
--2019-05-16 12:30:50--  http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar
Resolving rail.eecs.berkeley.edu (rail.eecs.berkeley.edu)... 128.32.189.73
Connecting to rail.eecs.berkeley.edu (rail.eecs.berkeley.edu)|128.32.189.73|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32274964480 (30G) [application/x-tar]
Saving to: ‘bair_robot_pushing_dataset_v0.tar’

bair_robot_pushing_dataset_v0.tar                    0%[                                                                                ] 189.95M   456KB/s    in 10m 59s

2019-05-16 12:41:50 (295 KB/s) - Connection closed at byte 199172826. Retrying.

--2019-05-16 12:41:51--  (try: 2)  http://rail.eecs.berkeley.edu/datasets/bair_robot_pushing_dataset_v0.tar
Connecting to rail.eecs.berkeley.edu (rail.eecs.berkeley.edu)|128.32.189.73|:80... connected.
HTTP request sent, awaiting response... 416 Requested range not satisfiable

    The file is already fully retrieved; nothing to do.

Also, I found a script that downloads BAIR dataset here. But I encountered the same problem here as well.

I'm confused now. Is the dataset so small or am I doing something wrong?


texts: How do I deploy a model when using Stratified K fold?

texts:
I have used Stratified K fold for learning the model . Below is the python code:
>def stratified_cv_v1(X, y, clf, shuffle=True, n=10,):
>    stratified_k_fold = StratifiedKFold(n_splits=n,shuffle=shuffle)
>    y_pred_v1 = y.copy()
>    for ii, jj in stratified_k_fold.split(X,y):
>        X_train, X_test = X[ii], X[jj]
>        y_train = y[ii]
>        clf_v2 = clf()
>        clf_v2.fit(X_train,y_train)
>        y_pred[jj] = clf.predict(X_test)
>    return y_pred_v1

>print(classification_report(y, stratified_cv_v1(X, y, GradientBoostingClassifier)))

Now how do I use the model to deploy on a new data set where I need to predict ?


texts:
I'm using a basic RNN as in the figure below (say for translation).
The model has the following structure:

\begin{aligned} s_t &= \tanh(Ux_t + Ws_{t-1}) \\ o_t &= \mathrm{softmax}(Vs_t) \end{aligned}

Assume m is the vocabulary size and n that of the hidden layer.
If $x_{t}=\{0,1\}^{m}$ and U is a $ n \times m$  matrix then W is a $ n \times n $ matrix.
If $o_{t}$ is $\mathbb{R}^{k}$ and $s_{t}$ is $\mathbb{R}^{n}$then V is a $ k \times n $ matrix.

What's the # parameters for this RNN model?


texts: Help improving time series prediction with LSTM on PyTorch

texts:
So, I am trying to use a LSTM model to forecast temperature data on PyTorch. I am relatively new to both PyTorch and the use of recurrent networks so I took a model I found on the internet to start. Unfortunately, I am working with missing data and I am assigning the value 0 to it. The whole project is on github if you need more details.
From the templates, I split it into two approaches.
One train2.py takes a tuple $x=(x_0,...,x_{n-1})$ as input and uses $y=(x_1,...,x_n)$ as target. From there, I recursively call the model $N$ times to forecast $N$ times into the future. Here's the part of the code that matters:
class LSTM(nn.Module):
    def __init__(self, input_size=1, hidden_layer_size=5, output_size=1):
        super().__init__()
        self.input_size = input_size

        self.hidden_layer_size = hidden_layer_size

        self.lstm = nn.LSTM(input_size, hidden_layer_size)

        self.lstm2 = nn.LSTM(hidden_layer_size, input_size)

        self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size).double(),
                            torch.zeros(1,1,self.hidden_layer_size).double())

        self.hidden_cell_2 = (torch.zeros(1,1,self.input_size).double(),
                            torch.zeros(1,1,self.input_size).double())

    def forward(self, input_seq):
        inpt = input_seq.view(len(input_seq) ,1, -1).double()
        lstm_out, self.hidden_cell = self.lstm(inpt, self.hidden_cell)
        lstm_out2, self.hidden_cell_2 = self.lstm2(lstm_out,self.hidden_cell_2)
        predictions = lstm_out2.view(len(input_seq), -1)
        return predictions.view(-1)

Here's how the output looks like (predict2.html)

It kind of gets the oscillatory behavior but the amplitude is way off.
The second one, train.py takes a tuple $x=(x_0,...,x_{\frac{n}{2}-1})$ as input and $y=(x_{\frac{n}{2}},...,x_n)$ as output. For predictions in this one, I make a single call to the model and I can only look at $N<\frac{n}{2}$ points into the future. Here's the code:
class LSTM(nn.Module):
    def __init__(self, input_size=1, hidden_layer_size=5, output_size=1, window=10):
        super().__init__()
        self.window = window

        self.hidden_layer_size = hidden_layer_size

        self.lstm = nn.LSTM(input_size, hidden_layer_size)

        self.linear = nn.Linear(hidden_layer_size, output_size)

        self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size).double(),
                            torch.zeros(1,1,self.hidden_layer_size).double())
        self.sigmoid = nn.Sigmoid()

    def forward(self, input_seq):
        inpt = input_seq.view(len(input_seq) ,1, -1).double()
        lstm_out, self.hidden_cell = self.lstm(inpt, self.hidden_cell)
        predictions = self.linear(lstm_out.view(len(input_seq), -1))
        return self.sigmoid(predictions[-self.window:].view(-1))

The output looks like this:

It fails to keep continuity most of the time, probably because it doesn't have a reference starting point in the output?
With all of that said my question is, how to improve the prediction? Should I add more data to it? Am I using too many hidden layers? Is the use of sigmoid flattening the fluctuations in the data too much? Is the model just completely wrong? I need guidance here.
Thank you for your time.


texts: Difference between AlphaGo's policy network and value network

texts:
I was reading a high level summary about Google's AlphaGo, and I came across the terms "policy network" and "value network". At a high level, I understand that the policy network is used to suggest moves and the value network is used to, "Reduce the depth of the search tree [and estimate] the winner in each position in place of searching all the way to the end of the game."
These two networks seem redundant to me. What is the policy network doing if it's not using the value network to prune its policies? It seems pretty clear that the value network is a deep learning neural network; is the policy network just a theoretical abstraction and not an actual neural network? The target variable for the value network seems to be win/loss. Is there a target variable for the policy network; if so, what is it? What is the policy network trying to optimize?
The full pdf of Google's paper, published in Nature, can be found here.


texts: Does it make sense to randomly select features as a baseline?

texts:
In my paper, I am saying that the accuracy of classification is $x\%$ when using the top N features.

My supervisor thinks that we should capture the classification accuracy when using N randomly selected features to show that the initial feature selection technique makes an actual difference.

Does this make sense?

I've argued that no one cares about randomly selected features so this addition doesn't make sense. It's quite obvious that randomly selecting features will provide a worse classification accuracy so there's no need to show that using any sort of feature ranking metric will be superior.


texts: Rewards are converged but with a lot of variations

texts:
I am training a reinforcement learning agent on an episodic task of fixed episode length. I am tracking the training process by plotting the cumulative rewards over an episode. I am using tensorboard for plotting the rewards. I have trained my agent for 20M steps. So I believe the agent has been given enough time to train. The cumulative rewards for an episode can range from +132 to around -60.
My plot with a smoothing of 0.999

Over the episodes, I can see that my rewards have converged. But if I see the plot with smoothing of 0

There is a huge variation in the rewards. So should I consider that the agent has converged or not? Also I don't understand why is there such a huge variation in rewards even after so much of training?

Thanks.


texts:
I'm trying to understand the Bayes by Backprop algorithm from the paper Weight Uncertainty in Neural Networks, the idea is to make a NN in which each weight has it's own probability distribution. I get the theory, but I don't undertsand how to update the mean and variance in the learning part. I found a code in Pytorch which simply does:
class BayesianLinear(nn.Module):
def __init__(self, in_features, out_features):
    (...)
    # Weight parameters
    self.weight_mu = nn.Parameter(torch.Tensor(out_features, in_features).uniform_(-0.2, 0.2))
    self.weight_rho = nn.Parameter(torch.Tensor(out_features, in_features).uniform_(-5,-4))
    self.weight = Gaussian(self.weight_mu, self.weight_rho)

How does the optimizer knows how to update mu (the mean) and rho (the variance)? It is the Parameter function which allows it? Or it is happening somewhere in the code? (it looks like a normal neural net from there!).


texts: How much can the AUC improve comparing the raw dataset and the feature engineered dataset?

texts:
Let's say I put the following two datasets in the best possible model (same model for both):

A raw dataset, the variables as they came just from the query.
A feature-engineered dataset, with hundreds of created variables, which came from the same raw dataset I just mentioned.

Could the difference between both AUCs be high? How much?


texts: How to build recommendation model based on resume and job description?

texts:
How to build a model which will result in better recommendation of resumes based on the job description given?
I am familiar with bow or tfidf (n-grams) approach and then take a cosine similarity but I'm looking for a deep learning approach. I don't have any labelled data to evaluate.
Anything suggestions will be really appreciated.


texts: Split Dataset in two Explainable Subsets

texts:
is there a general approach for splitting a dataset into two or more subsets so that a classification problem can be solved with higher accuracy for the resulting separate subsets?

I have a very inconsistent dataset which I can only classify with a 50% accuracy. I spit the set in two subsets based on an assumption of what predictor should correlate with a output variable. In my first subset, I only included samples from the dataset were the predictor correlates with the output variable. The remaining samples go in the second subset. Afterwards, both subsets (1k samples/2k samples) performed better, from 50% accuracy to 95%/65%.
In this specific example, I have a strong suspicion that the class labels used for tagging were interpreted differently by different persons which led to conflicting tags. Manually "enforcing" one interpretation helped separating this. This is supported by the fact that correlations between this output variable and other output variables drastically swap signs between one and the other subset.

I was wondering if there is a general approach on how this could be achieved for my other output variables. I.e., find a separation of my dataset so that both sets separately can be classified accurately with some machine learning algorithm.


texts:
Looking for a dataset that meets the following criteria:

Multi-class, 3+ labels, preferably not binary.
Tabular data, not images.
Getting above 65-85% prediction accuracy is a challenge. In general, high prediction accuracy would be seen as a struggle.

I am hoping to prove some potentially novel classification methods. To date, I have been working with the classic Iris, Abalone, Sonar, and Kaggle Zoo datasets - while scouring Kaggle for more.

Anyone know of a toughie? Even if it is your own data.


texts: Dealing with noisy training labels in text classification using deep learning

texts:
I have a dataset that comprises of sentences and corresponding multi-labels (e.g. a sentence can belong to multiple labels). Using a combination of Convolutional Neural Networks and Recurrent Neural Nets on language models (Word2Vec) I'm able to achieve a good accuracy. However, it's /too/ good at modelling the output, in the sense that a lot of labels are arguably wrong and thus the output too. This means that the evaluation (even with regularization and dropout) gives a wrong impression, since I have no ground truth. Cleaning up the labels would be prohibitively expensive. So I'm left to explore "denoising" the labels somehow. I've looked at things like "Learning from Massive Noisy Labeled Data for Image Classification", however they assume to learn some sort of noise covariace matrix on the outputs, which I'm not sure how to do in Keras.

Has anyone dealt with the problem of noisy labels in a mutli-label text classification setting before (ideally using Keras or similar) and has good ideas on how to learn a robust model with noisy labels?


texts: Recall score for each sample in multilabel classification

texts:
Does it make sense to calculate the recall for each sample in a multilabel classification problem?
Suppose I have 3 data samples, each having its own true set of labels and predicted set of labels.

I want to see the match between the true set of labels and the predicted set of labels. I do not care for the true negatives or false positives in each prediction, so this translates to recall score for me. Programmatically, I would do an AND operation between y_predicted and y_true to get the number of true positives and divide it by the total number of true labels for each sample. (in other words, true positives/(true positives+false negatives))
My question is -

Is calculating recall per sample (not per label), usually done?
Is my thought process correct?

I've seen articles where a single recall is calculated for the whole matrix of y_true and y_predicted or recall is calculated for a single label.


texts: review: gradient descent, epochs, validation in neural network training

texts:
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.

During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.

If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?


texts: Make a binary classification RNN to only focus on "malicious" words to make the model more robust? (Non negative model)

texts:
There are some works that make a model robust against attacks, by making the model "focus" on only malicious features, therefore addition of benign features will not affect the outcome of model and only addition of malicious content can change its prediction.
more on this :
https://arxiv.org/pdf/1806.06108
https://towardsdatascience.com/evading-machine-learning-malware-classifiers-ce52dabdb713  (read Non-Negative MalConv)

Non-Negative MalConv was constrained during training to have
non-negative weight matrices. The point of doing this is to prevent
trivial attacks like those created against MalConv. When done
properly, the non-negative weights make binary classifiers monotonic;
meaning that the addition of new content can only increase the
malicious score. This would make evading the model very difficult,
because most evasion attacks do require adding content to the file.
Fortunately for me, this implementation of Non-Negative MalConv has a
subtle but critical flaw.

my question is, how can i implement this in RNN/LSTM using pytorch?
my model currently takes a sequence of words and predicts whether the sentence is malicious or not, and right now addition of a lot of benign words (by beaning i mean words that appear in a lot of benign sentences) will evade the model.
basically i want my model to learn to predict based on only words that are malicious, meaning mostly appear in malicious samples, and addition of benign words that appear in many benign sentences will not change the prediction of the sentence (so adding many benign words will not affect the prediction)
how can i implement this? is this possible?


texts:
I have some doubts regarding encoding (i am not familiar with tasks like these) categorical variables in order to use them as parameters in a model like logistic regression or SVM.
My dataset looks like the following
Text                                  Symbol    Note    Account    Age   Label
There is a red car                      !        red      John    24   1
My bag was very expensive               ?       orange    Luke    36  0
Where are my keys?                      @        red      Red     58  1
I promise: I will never let you go!    ...       green    Aoife   28  0

In text there are stored comments from users in a community; symbol includes the most used symbol by an user; Note represents its level: green is more experienced, red is a new joiner; Account is the user name. Label gives me information about user’s trustworthy: if 0 the user is not fake; if one the user might be a possible bot.
I would like to classify new users based on the current information (see columns above). My dataset includes more than 1000 rows and 400 users.
Since to use classifiers I need to encode categorical fields, I have tried to do as follows, by using MultiColumnLabelEncoder in sklearn:
MultiColumnLabelEncoder(columns = ['Text', 'Symbol', 'Note', 'Account']).fit_transform(df)

where df is my dataframe. However, I understood that also onehotencoder should be preferable. I also included Account as there might be more comments from the same account, so if I classified an account as fake and I receive a new comment from the same account, then this account could be easily detected as fake.
The aim, as earlier i mentioned, would be to classify, with a certain accuracy, new elements from a test set, based on the information given(symbol, note , age, texts), i.e. looking for a possible correlations among these variables which can allow me to say that a new account is fake (1) or not (0).
The problem, as you can see, is related to classifiers where parameters are not only numerical, but also categorical.
For data preprocessing (removing stopwords and cleaning data), I have used  Python packages of nltk; regarding features extraction ( this should be a key point as it is linked to the next step, i.e. using a classifier to predict class - 1 or 0), I have found difficulties in understanding what output I should expect from the encoding in order to be able to use information above as inputs in my model (where target is called label and it is a binary value).
I am using as classifier logistic regression, but also
SVM.
My expected output, in case of user X with age 16, symbol #, note Wonderful, and note red (new joiner) would be a classification as fake with a certain percentage.
I would appreciate if someone could explain me, step by step, the way to transform my dataset in a dataset whose variables I can use  within a logistic regression in order to determine the label (fake or not fake) of new users.


texts: Good chromosome representation in a VRPTW genetic algorithm

texts:
I have a genetic algorithm for a vehicle routing problem with time windows and I need to implement certain modifications. I am not sure what would be the best chromosome representations.
I have tasks which can be divided into 3 sub-tasks with certain ordered time windows, they have to processed in order and all 3 (they represent collecting certain goods in a storage, delivering them and returning packaging to another storage). In the algorithm crossover part these tasks are combined together and evaluated. They have to processed in order according to their task number, i.e. combination "A1, B1, B2, A2, B3, A3" is correct, but "A1, B2, A2, B3, A3, B1" or "A1, A2, B1" is not.
The problem is, I don't know how to assure the order of events will be kept. How can I represent this demand in chromosomes? Or, where in the algorithm should I keep this demand?
In the previous version of the algorithm we used only the whole task A or B without distinguishing the subtasks, which is now not sufficient. I am relatively new to genetic algorithms, so pardon me if it's something obvious.


texts: Can we look just at the other features when we have a missing vaue?

texts:
For my classification task i have 3 features a,b,c.
Feature c is missing for some datas.
I can have already a good score for my classifier training with the two other features a,b and even better if I don't look at the missing data and I train with a,b,c.

Is it possible to train my classifier with features a,b,c on all the data which don't have missing values, and in my prediction phase, if the feature c is missing, to predict the class like if I had train my classifier with a and b?


texts: How to run a python script on GCP Compute Engine?

texts:
I want to run some machine learning algorithms such as PCA and KNN with a relatively large dataset of images (>2000 rgb images) in order to classify these images.

My source code is the following:

import cv2
import numpy as np
import os
from glob import glob
from sklearn.decomposition import PCA
from sklearn import neighbors
from sklearn import preprocessing

data = []

# Read images from file
for filename in glob('Images/*.jpg'):

    img = cv2.imread(filename)
    height, width = img.shape[:2]
    img = np.array(img)

    # Check that all my images are of the same resolution
    if height == 529 and width == 940:

        # Reshape each image so that it is stored in one line
        img = np.concatenate(img, axis=0)
        img = np.concatenate(img, axis=0)
        data.append(img)

# Normalise data
data = np.array(data)
Norm = preprocessing.Normalizer()
Norm.fit(data)
data = Norm.transform(data)

# PCA model
pca = PCA(0.95)
pca.fit(data)
data = pca.transform(data)

# K-Nearest neighbours
knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)
distances, indices = knn.kneighbors(data)

print(indices)

However, my laptop is not sufficient for this task as it needs many hours in order to process more than 700 rgb images. So I need to use the computational resources of an online platform (e.g. like the ones provided by GCP).

Can I simply make a call from Pycharm to Compute Engine API (after a I have created a virtual machine in it) to run my python script?

Or is it possible either to install PyCharm in the virtual machine and run the python script in it or to write my source code in a docker container?

In all, how can I simply run a python script on GCP Compute Engine without wasting time in needless things?


texts: Can we do autoregressive using pad_packed_sequence?

texts:
I’m curious about if we want to do the autoregressive manner. Is it possible to do with implemented using pad_packed_sequence and pack_padded_sequence input to some recurrent network? Because we need to wait for the output of timestep t. Given this scenario we cannot gather all prediction and packed them. Then what should I do with this. Is there any way to do this without looping to each time step? Also If i want to do it in batch processing is it possible?
Thank you


texts: Why would one crossvalidate the random state number?

texts:
Still learning about machine learning, I've stumbled across a kaggle (link), which I cannot understand.

Here are lines 72 and 73:

parameters = {'solver': ['lbfgs'],
              'max_iter': [1000,1100,1200,1300,1400,1500,1600,1700,1800,1900,2000 ],
              'alpha': 10.0 ** -np.arange(1, 10),
              'hidden_layer_sizes':np.arange(10, 15),
              'random_state':[0,1,2,3,4,5,6,7,8,9]}
clf = GridSearchCV(MLPClassifier(), parameters, n_jobs=-1)

As you can see, the random_state parameter is been tested across 10 values.

What is the point of doing this?

If one model perform better with some random_state, does it make any sense to use this particular parameter on other models?


texts: Algorithms that would benefit from variable transformations?

texts:
1- Which algorithms would benefit from data that has been transformed, so that distributions of continuous variables resemble that of a normal distribution ?

2- What would be the benefits of transforming variables in such a way ?


texts: Are there any unsupervised learning algorithms for time sequenced data?

texts:
Each observation in my data was collected with a difference of 0.1 seconds. I don't call it a time series because it don't have a date and time stamp. In the examples of clustering algorithms (I found online) and PCA the sample data have 1 observation per case and are not timed. But my data have hundreds of observations collected every 0.1 seconds per vehicle and there are many vehicles.

Note: I have asked this question on quora as well.


texts:
I am building a regression model. Each sample/object in my dataset has some numerical and categorical features. Let's call those $f_1, f_2, f_3$ for now. The output that I'm trying to learn is a $2D$ grid of numerical values. Kind of like an $3\times 3$ image with values in it. The output variables are correlated.

For my first attempt at the problem, I thought of using $f_1, f_2, f_3$ as input features and then flatten my image into 9 output features $out_1,...,out_9$. There are multiple techniques that can optimize for those variables independantly/jointly so this is fine.

The other option that I thought of was to use the output grid coordinates as input features. So I would have $5$ input features: $f_1, f_2, f_3$ (as before) and $f_4, f_5$ are my $x, y$ coordinates for the target variable. In this case I would have more samples and my output would consist of only $1$ output target variable.

My problem is that I'm not sure whether any of the options is fundamentally wrong or just bad practice. Is there anything I should watch out for in training and testing if I proceed with either approach?


texts: How discriminator knows if the image is real or fake at the initial phase?

texts: If the discriminator and generator in GAN learn together, How the discriminator knows whether the image is real or fake in the initial phase of training? Does the discriminator need to get trained earlier?


texts: Algorithmically extract seasonality in time series data

texts:
Suppose you are trying to measure if the seasonality of a particular event stream is consistent i.e. the events in a time series happen more or less in pattern like fashion. How can you algorithmically measure and extract seasonality?

Currently, I am using autocorrelation to view the clearly seasonal (monthly) data events. I am struggling to figure out how to write an algorithm that could test whether or not something is seasonal by month.
The picture below shows the results of the pacf function within the statsmodels.tsa.stattools library. From the graph it is clear, how can I use the array to determine seasonality? Is there a library that would be useful for this?

This question is similar, but did not get any great answers:
https://stats.stackexchange.com/questions/225003/test-for-trend-and-seasonality-in-time-series


texts: Can we use Binary Cross Entropy for Multiclass Classification?

texts: In this link, the author has implemented a CNN which classifies 15 classes and has used Binary Cross Entropy as the loss function. But since it's multiclass classification, is it valid to use Binary Cross Entropy? Or should we use Categorical Cross Entropy instead?


texts:
I'm using Keras, and I am struggling to know how many parameters Resnet-50 has. Keras documentation says around 25M, while if I use model.param_count() when loading a Resnet-50 model, it says 234M.
Which one is correct? I'm confused.

model.summary prints this:

Total params: 234,355,586
Trainable params: 210,767,874
Non-trainable params: 23,587,712


texts: How to create user and item profile in an item to item collaborative filtering? (Non-rating case)

texts:
I want to build a recommender system for a coupons website which should do the following: Given the past purchase behaviour of a user, recommend coupons which the user is likely to buy. The data does not have any ratings for coupons by the user. It tells if a user bought a certain coupon or not, which category did it belong to, what is the gender and location of the user who bought these coupons etc.? I am implementing an item to item collaborative filtering. Based on what I have learnt through videos available on this topic, the following is my approach:

Build an item profile of all the coupons with attributes such as category of the coupon, location the coupon is available in and the days on which the coupon can be redeemed.
Build a user profile of all the registered users with attributes such as age, gender, and location.

Now I am making an item and user matrix, where I will have Coupons on the rows and Users in the columns. I will fill 0 and 1 depending on if a coupon C1 was bought by a user U1 in the past or not. In the same dataset, for the coupons for which I want to predict, I will calculate a score based on cosine similarity between items.

My question is if I am using just 0s and 1s (coupon bought or not) to recommend coupons, what is the use of the item and user profile? How I can incorporate the intelligence in the recommendation system that if a user has bought more coupons from X category in the past than recommend him new Y coupons from the same category?  I read online about using random forests etc. to calculate "weights" of the attributes in recommendation systems, where do they fit in this scenario?

P.S. I am doing everything in R.

Many thanks in advance for any advice/suggestions.