texts: How to initialize a new word2vec model with pre-trained model weights?

texts:
I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset pretrained model). I have been struggling with it for a couple of weeks. Now, I just found out that in gesim there is a function that can help me initialize the weights of my model with pre-trained model weights.
That is mentioned below:
reset_from(other_model)

    Borrow shareable pre-built structures (like vocab) from the other_model. Useful if testing multiple models in parallel on the same corpus.

I do not know this function can do the same thing or not. Please help!


texts: Genetic algorithms: what connection to support vector machine / naive bayes

texts:
I found the following list of seven classifiers:

Linear Classifiers: Logistic Regression, Naive Bayes Classifier
Nearest Neighbor
Support Vector Machines
Decision Trees
Boosted Trees
Random Forest
Neural Network

What are genetic algorithms, and why aren't they considered as part of the seven classifiers?


texts:
I am new to ML and have been reading online about training bottlenecks when there are frequent updates to data.

Let's say I have a built a model based on a dataset of 10M records.

Now, in another 2 months, I might receive another 1M records which we would like to feed into our model as well.

Similarly this goes on for every 2 months. We would like to update/train our model with latest data as and when it's available

1) But for example, let's say the training takes 1 week for every new data update

2) Any suggestions on how we can minimize the training time (when we train every 2 months?)?

3) Should we select a representative sample from 1 Million datapoints? Is that good enough

4) I understand it's all about tradeoff but I am curious to know whether I am missing any known approaches to save training time? I am thinking representative sample can reduce the sample size and help us fasten the training process

Can you share your suggestions on this?


texts: Hierarchical classification with multi-class predictor for every parent node

texts:
Edit: It turned out that I had an error in my function to compute the combined probabilities (a typo that changed the behavior of my function quite a bit without giving me an error message). Without that, my classification is much more satisfactory. Also, I don't use softmax anymore because my probabilities are normalized anyway. So the question is basically solved. If anyone has a good idea how to improve my hierarchical classification algorithm, I'm happy to hear it, though!

I have a data set with tf-idf/bm25 weighted words (I'm trying both approaches) with several classes that I try to predict. The classes have a hierarchical structure with 2 levels. A part of the hierarchy is shown in the graph. In total, there are 20 classes on the first level and 111 leaf nodes. Some parent nodes don't have any child nodes.

I would like to fit a multi-class classifier for every parent node (so considering only the part of the hierarchy shown above, that would be 3 classifiers, one for the first level and two for the parent nodes that have children), which is called the “Local classifier per parent node approach” according to Silla & Freitas, 2011. How I understand it, the classifier for the first level is trained using the whole training set, while the classifiers for the second level are trained only using examples with labels that belong in these classes (so the classifier for the children of Catering is trained with only training examples that are labeled as Catering).

I planned on using Multinomial Naive Bayes and Support Vector Machines from scikit-learn as my classification algorithms. After training, I obtain log probabilities for the test set.

What I'm unsure about is how I should combine the probabilities that I obtain with my classifiers. I thought that I could just multiply the probability of the first level with the probabilities in the second level (or add the log probabilities) to get leaf node probabilities.

My reasoning behind that is that if a mistake is made on the first level, so that the true level 1 class has a relatively low probability, then it can be still compensated by a high probability in a child node.

After obtaining all probabilities, I would have normalized the leaf node probabilities so that they add up to 1 with softmax. However, when doing that, I get very similar probabilities for every leaf node, so that the highest leaf node probability is only a tiny bit higher than the probabilities of the other leaf nodes.

That makes sense in a way because I force every second level classifier to give me probabilities for only the children of that classifier and those add up to 1. So even if my test example does not belong to the parent node in question, one of the children classes is going to get a relatively high probability.

How can I avoid the problem of leaf node probabilities that don't discriminate enough? Or is there some other error in my reasoning that led to this result?


texts: Why does Multinomial Naivebayes fails/poor results for continous data like iris dataset?

texts: i'm confused as to why MultinomialNB will give poor accuracy as we're using the same formula for calculating the probability i.e finding of count of x for particular class divided by count of all words in that particular class.


texts: In which cases are non-linear learning methods preferred than logistic regression in classification problems?

texts:
We know that neural networks and other learning methods can have better performance relative to logistic regression in some non-linear classification problems. But, it is known too that logistic regression can separate classes with a line that can be curvy if only we add more predictors that are a square, cube, etc of the given predictors (still considered a linear decision boundary).

So my question is, can we theoretically solve any classification problem through logistic regression, or are there limitations that I do not realize that make other non-linear learning methods mandatory to certain classification problems?


texts:
I am trying to implement some deep learning models with large amount of data around 10gigabyte. Although, my Laptop and Collab-free crashes when it tries to load them. Do you think it worths to buy collab-pro? Do you suggest any other solutions? But my worries are mostly about buying collab-pro is only for US and Canada while I am from Europe.
Thanks in advance.


texts: What's the good index to choose number of clusters so that obtained clusters are homogeneous?

texts:
I perform a clustering on one-dimensional dataset and I need a way to automatically decide what's the optimal number of clusters from $k \in \{2, 3, 4, 5, 6\}$. The number of observations to cluster is low (usually around 10-13). I think I'd need to check optimising for one of two goals (or both at the same time) and see what works best:

to achieve partitioning with the lowest within-cluster variances. Intuitively, I would go for something like average within-cluster variance, but I'm actually ok with the situation when some clusters would be formed out of single observation (it's actually desirable for outliers and that's why I check for relatively high number of clusters). And average within-cluster variance would always favour lower number of clusters.
to achieve partitioning with the most similar distances between pairs of observations within a cluster. For example, if I have objects $a, b, c, d$ in my cluster, I'd like to have $d(a, b) \approx d(b, c) \approx d(c, d)$ where $d$ is euclidean distance and $a, b, c, d$ are sorted.

I have studied scikit-learn options and none of them seems appropriate to my case.


texts:
Suppose if I delete all the objects from current session:

rm(list=ls())

Is there any way in base R or using a function from a package which lets me restore the deleted objects from the current session?


texts: Measures for choosing the best Decision Tree split?

texts:
I'm searching for a paper or book that list several possible measures for deciding which variable to split a Decision Tree at.

The basic ones are well known and very well documented in several sources:

Entropy (or Information Gain)
Gini impurity
Classification Error
Variance reduction

But I suppose people come up with all kinds of other split criteria all the time, don't they?
Has someone ever collected a list?
Preferably with some explanations about the use case and particular advantages (and disadvantages) of the new split criterions.


texts: XGBOOST missing_value feature degrades my performance?

texts:
I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction means is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

the cross-validated performance on the train population stays the same.
BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?


texts: Is it cheating to stratified sample the whole dataset based on a previous evaluation result?

texts:
I trained a model using a small mri dataset(57 patients).
The model's performance was so low(Train set 0.7, Val set 0.7, Test set 0.45).

I found the model segment tumor in upper part of brain well, couldn't segment tumor which is in middle part of brain.

So i stratified sampled the whole dataset with a position stratum(upper, not upper), the model performance was improved.

In this case, is it cheating to do that?


texts: How to use glob() output as input to os.listdir()

texts:
I'm trying to use the output of glob.glob() as the input to os.listdir() in order to get the number of files in the directory. The output of glob() gives the following:

f = glob.glob(ct)
print(f)

['C:\\Users\\tennant\\Desktop\\RF WAVEFORMS\\SPRING 2018\\RF\\1L22\\2018_05_02\\133258.2']

which, if I try to use as input to listdir() gives the following error

test = os.listdir(str(f))

Eventually through enough trial and error, I was able to find a solution.

f = glob.glob(ct)
fnew = str(f).strip('[]')
test = os.listdir(fnew.replace('\'',""))
print(fnew.replace('\'',""), len(test))

C:\\Users\\tennant\\Desktop\\RF WAVEFORMS\\SPRING 2018\\RF\\1L22\\2018_05_02\\133258.2 7

However it's messy and I'm clearly not understanding something more fundamental about the output of glob() or strings in general. Anything that could clean this code up and help my understanding would be greatly appreciated!


texts:
Given a pair of Variable i and j and a Tensor A, how to get a new Tensor which is A with ith element and jth element swapped? How about N pairs of i and j and we need to swap them all? How to do that in batch training?

I found swapping one pair is easy:

if i < j:
    return tf.concat((A[:i], A[j], A[i+1:j], A[i], A[j+1:]), 0)
else:
    return tf.concat((A[:j], A[i], A[j+1:i], A[j], A[i+1:]), 0)

Swapping multiple pairs would require explicit loop / reducing:

for i, j in ijpairs:
    A = swap_one_pair(A, i, j)
return A

Swapping multiple samples in a batch would require explicit mapping:

return tf.map_fn(swap_one_pair, (As, is, js))

I found if I separate the swapping into "read" and "write", reading can be easily vectorized via gather_nd:

indexes = tf.stack((tf.tile(tf.range(batch_size)[:,None], (1,read_size)), all_i), 2)
index_results = tf.gather_nd(A, indexes)

Is there a "write" version of gather_nd? Or is there any more elegant way of implementing swapping utilizing vectorization/broadcasting?

EDIT

I found a "write" version scatter_nd_update. Unfortunately it only works for Variable. I tried casting Tensor to Variable using tf.Variable(myTensor) but don't work. I currently use:

def scatter_nd_update(ref, indices, updates):
    original_part = ref * (1 - tf.scatter_nd(indices, tf.ones(updates.shape), ref.shape))
    update_part   = tf.scatter_nd(index, update, ref.shape)
    return original_part + update_part

This method uses a lot of meaningless computation. Any idea of better one?


texts: I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:

texts:
Column Name: "Genres" //This will contain the genres of movies
User genre: "Action Adventure"
I want to perform cosine similarity on this data, compare user genre with genres of different movies, sort and display the names of movies having similar genre. How should I do it?


texts: How to identify word in a sentence representing the song genre?

texts: I am training a model to identify a word that represents a song genre given a sentence.  For example, the model is given a sentence "Beethoven songs are part of the classical genre."  The model will categorize the word, "classical" as "song genre".  Can I use an RNN classifier to do this or is there another algorithm that can accomplish this task?


texts: Should bias updates be porportional to overfitting?

texts:
According to questions on the internet, the bias is a learnable parameter,
 and there are different solutions to updating it, but I failed to find a concise methodology of
 correctly updating biases during training.

When I tried to overfit a small network, it failed when the bias was introduced into the training:

When I tried to scale down bias updates it produeced similar, but delayed results:

Next I tried to make bias updates porportional to the train set error, again only delaying the trend:

Next I tried to make bias updated inversely porportional to the error in the training set, but I suppose this would not have any shown benefits without a validation set. Alas the effect was the same:

According to @Noah Weber, Bias is something that would help in reducing overfitting during training, which is actually consistent with my previous experiments.

Based on this I would suppose the more
overfitting occurs, the more the bias term should be updated. This can clearly be measured by the differences in the error and test set. Should bias be updated according to that?


texts: Feature has a pattern in relation to class but does not enhance classifier to predict class

texts: I have a feature x that when I plot it again my class variable y it shows some sort of pattern, i.e. it is obvious that x has a relation with y, but when I add the x to my logistic model it reduce the model accuracy. I want to know the reason, I thought that maybe there is a different distribution in my training and testing data, that lead to this, and checked this hypothesis, the result was that distributions are different. In general case any idea is appreciated.


texts:
I am a newbie to machine learning and I am trying to apply the SVD on the movielens dataset for movie recommendation. I have a movie-user matrix where the row is the user id, the column is the movie id and the value is the rating.

Now, I would like to perform normalization on the movie-user matrix (subtract the data by users ratings mean). Then pass the normalized matrix to Scipy.sparse svds as follow:

from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k = 50)

Now, I have the 2 method to do it:

Method 1.) Fill all the missing rating with 0 first, then calculate the user rating mean for normalization.

The predicted rating dataframe for method 1 by using svd is:

Method 2.) Calculate the user rating mean first and do the normalization, then replace the missing rating with 0.

The predicted rating dataframe for method 2 by using svd is:

I would like to know which method is better or there are other methods to do it. So far as I can observe from method 2, the predicted ratings for a user are quite similar. For example, user A may get 4.XX ratings for all movies. Meanwhile in method 1, there are more variation. I would like to know if there are something wrong.


texts: StackOverflow Tags Predictor...Suggest an Machine Learning Approach please?

texts:
I am trying to predict tags for stackoverflow questions and I am not able to decide which Machine Learning algorithm will be a correct approach for this.

Input: As a dataset I have mined stackoverflow questions, I have tokenized the data set and removed stopwords and punctuation from this data.

Things i have tried:

TF-IDF
Trained Naive Bayes on the dataset and then gave user defined input to predict tags, but its not working correctly
Linear SVM

Which ML algorithm I should use Supervised or Unsupervised?
If possible please, suggest a correct ML approach from the scratch.
PS: I have the list of all tags present on StackOverflow so, will this help in anyway?
Thanks


texts: Train object detection to detect floor type

texts: What is the best approach to train object detection to efficiently detect floor type in an image (e.g., hardwood floor, tile floor, etc.)? I don't need a full mask or bounding box, I just need to highlight a single pixel on the floor and tell the floor type (e.g., hardwood floor). I hope to have a method with the least annotation effort that works.


texts:
I cannot wrap my head around this simple concept.

Suppose we have a linear regression, and there is a single parameter theta to be optimized (for simplicity purposes):

$h(x) = \theta \cdot x$

The error cost function could be defined as $J(\theta) = \frac1m \cdot \sum (h(x) - y(x)) ^ 2$, for each $x$.

Then, theta would be updated as:

$\theta = \theta - \alpha\cdot \frac1m \cdot \sum (h(x) - y(x)) \cdot  x$, for each $x$.

From my understanding the multiplier after the alpha term is the derivative of the error cost function $J$. This term tells us the direction to head in, in order to arrive at the minimum making a small step at a time. I understand the concept of "hill climbing" correctly, at least I think.

Here is where I don't seem to wrap my head around:

If the form of the error function is known (like in our case: we could visually plot the function if we take enough values of theta and plug them in the model), why can't we take the first derivative and set it to zero (partial derivative if the function has multiple thetas). This way we would have all the minimums of the function. Then with the second derivative, we could determine whether it's a min or a max.

I've seen this done in calculus for simple functions like $y = x^2 + 5x + 2$ (may years ago, maybe I am wrong), so what is stopping us from doing the same thing here?

Sorry for asking such a silly question.

Thank you.


texts: Storing Audio features to a csv file using pyAudioAnalysis

texts:
I'm trying to store MFCC feartures of an audio file to a csv file. I'm following the wiki on github for Feature Extraction using pyAudioAnalysis.
The suggested command is:

python3 audioAnalysis.py featureExtractionFile -i test.wav -mw 1.0 -ms 1.0 -sw 0.050 -ss 0.050 -o data/speech_music_sample.wav

I'm using the command with minor changing i.e., I'm using python as I have only python-3 installed on my PC.
I'm getting an Error:

C:\Users\myusername\AppData\Local\Programs\Python\Python37\python.exe: can't open file 'audioAnalysis.py': [Errno 2] No such file or directory

Please Help!


texts: Multiple input single output supervised learning ANN problem

texts:
I have a dataset of 120 tuples giving a singular output. I want to implement ANN in estimating the input which is affecting the output most. A case of optimising the input to maximise the output.
ANN has in the input layer multiple features which can give n number of outputs. However in my case, the set of tuples gives a singular output. I want to know which combination of tuples is affecting the output most. Finding maxima or minima for the loss function.
The challenges here is there is no specific output mapped to a tuple. But a combination of 120 tuples give a singular output. In the proposed model i aim to feed further randomised combination of these tuples to train the model and later fetch the instruction(or tuples affecting the output most). Can we proceed by normalizing the output and then mapping the input tuples.
How can this be done?


texts: How to choose the correct data's shape for time series with Keras

texts:
I would like to create an LSTM using Keas, which takes as input a multivariate time series.
Following that example, we can see how the label hasbeen shaped:
y_train = features.iloc[start:end][[1]]

First question: why this data format?

I have tried with: y_train = train_data.iloc[:, -1]
Later, I have passed that label to timeseries_dataset_from_array():
dataset_train = keras.preprocessing.timeseries_dataset_from_array(
    x_train,
    y_train,
    sequence_length=sequence_length,
    batch_size=batch_size,
)

Which returns a wrong target's shape (64, ), where the correct shape should be (64, 1)


texts: Tools for nutritional value tracking and visualization

texts:
I want to track the nutritional value of different flours over time. What I got so far is a
data table like this:

|------------|------------|------------|------------|
|   Flour    |   Date     |   Energy   |   Sugar    |
|------------|------------|------------|------------|
|   type 1   |   03/20    |   310      |   2        |
|   type 2   |   03/20    |   290      |   1.8      |
|   type 1   |   04/20    |   310      |   2.1      |
|   type 2   |   04/20    |   287      |   1.7      |

Over time, I will add more nutritional values for the flours already in the table and
might add more types of flour. I might also want to add additional columns e.g. for protein.
The nutritional values of the different flour types should then be plotted over time
like this:

Energy
|
|     *         *             * type 1
|     #                       # type 2
|               #
|
|-----------------------Date
    03/20     04/20

What are the best tools to implement such a project?
My constraints are that I only want to use free and open-source software which runs under Arch Linux and that that i can export the plots


texts: Spark Scala alternative Machine Learning Library?

texts:
I’m using Spark Scala for multiclass classification,  and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, Multinomial and Bernoulli are supported, where I would need to use Gaussian, and LogisticRegressionWithLBFGS would not be suitable either.

I know that in Python, you can integrate sci-kit learn with Spark, but are there any options when using Scala?

Interested to hear people’s thoughts on this.


texts: Difficulty understanding the dimension differences in kernel PCA

texts:
In Kernel PCA, the kernel trick works because we can show that there is an equivalency between eigenvectors of the kernel matrix and eigenvectors of the covariance matrix. I know the math to go from one to the other but there is still one point that puzzles me very much:
These matrices have potentially very different dimensions, so I don't completely understand how the equivalence works. Let's say the covariance matrix is $(d\times d)$ dimensional and the kernel matrix $(N \times N)$, that means we have $N$ eigenvectors for the kernel matrix. Applying the formula to compute the eigenvectors of the covariance matrix from the kernel's eigenvectors would give $N$ vectors in total. Are some of the computed vectors identical? I don't understand how to reconcile the dimensions here.


texts:
I have a dataset of about 53000 points. It has been clustered twice, based on two sets of unrelated attributes. For the first clustering (clustering 1) I used DBScan, and it ended up with about 700 clusters, with 30000 of the points being labeled as noise. Then, on another set of attributes, I clustered them using kmeans, which resulted in 5 different attributes (clustering 2).

However, I have a feeling the two sets of attributes may have some relation after all. Therefore, I want to measure how variable the points are in terms of cluster membership of clustering 2 within each cluster of clustering 1. I don't have a strong mathematical background, I've tried simply calculating the SD within each clustering 1 group, but this does not work since the cluster labels are of influence (a group with 50% in cluster 0 and 50% in cluster 1 will have different results than a group with 50% in cluster 0 and 50% in cluster 2)

Unfortunately I cannot share my data, but I have included a screenshot, I hope that clarifies what I mean.


texts: Normalize the output of a dense layer with linear activation

texts:
I have the following architecture of my network:

def net_one(message):
    weight1 = np.random.normal(loc=0.0, scale=0.01, size=[16, 16])
    init1 = tf.constant_initializer(weight1)
    out1 = tf.layers.dense(inputs=message, units=16, activation=tf.nn.relu, kernel_initializer=init1)
    weight2 = np.random.normal(loc=0.0, scale=0.01, size=[16, 7])
    init2 = tf.constant_initializer(weight2)
    out2 = tf.layers.dense(inputs=out1, units=7, activation=None, kernel_initializer=init2)
    return out2

Now as the output of the network is linear (None in tensorflow corresponds to a linear activation function), the output is unbounded. I need the square of the 2-norm of the output to be a constant, n (for energy constraint purposes). I do not want to use sigmoid or tanh as they hamper the performance. I tried the following:

code = net_one(input_bits)
code = code * tf.sqrt(n) / tf.linalg.norm(code)

I have two questions:

Does it achieve what I expect it to achieve?
Is there any better way (if this is indeed right) or any alternate way to achieve this?


texts: Optimal parameter estimation for a classifier with multiple parameters

texts:
The image on the left shows a standard ROC curve formed by sweeping a single threshold and recording the corresponding True Positive Rate (TPR) and False Positive Rate (FPR).

The image on the right shows my problem setup where there are 3 parameters, and for each, we have only 2 choices. Together, it produces 8 points as depicted on the graph. In practice, I intend to have thousands of possible combinations of 100s of parameters, but the concept remains the same in this down-scaled case.

I intend to find 2 things here:

Determine the optimum parameter(s) for the given data
Provide an overall performance score for all combinations of parameters

In the case of the ROC curve on the left, this is done easily using the following methods:

Optimal parameter: Maximal difference of TPR and FPR with a cost component (I believe it is called the J-statistic?)
Overall performance: Area under the curve (the shaded portion in the graph)

However, for my case in the image on the right, I do not know if the methods I have chosen are the standard principled methods that are normally used.

Optimal parameter set: Same maximal difference of TPR and FPR

  Parameter score = TPR - FPR * cost_ratio

Overall performance: Average of all "parameter scores"

I have found a lot of reference material for the ROC curve with a single threshold and while there are other techniques available to determine the performance, the ones mentioned in this question is definitely considered a standard approach. I found no such reading material for the scenario presented on the right.

Bottomline, the question here is two-fold: (1) Provide methods to evaluate the optimal parameter set and overall performance in my problem scenario, (2) Provide reference that claims the suggested methods to be a standard approach for the given scenario.


texts: Calculate euclidean distances for KNN and cross validation given a 99x16 10 folds

texts:
I'm trying to implement KNN classification with cross-validation implementation in python. The data consists of 10 folds of size 99x64, each with their corresponding label of size 99x1.
Do I have to calculate distances row by row between each fold, for a resulting $99x1$ distance-vector for every $k-1$ testing fold?
E.g. Let's say we have three folds, each of dimension $[99x64]$. Assuming we are testing on fold $1$ with folds 2, 3 being the validating datasets, I would end up with two distance-vectors of size $[99x1]$ since I'm calculating the distance between rows of each fold.


texts: What model do I use to predict a regression problem with timeseries data

texts:
Overall Goal
To predict how much reagent "A" I started with in a reaction.

Data:
To predict this I have timeseries data of reagent "B". For each time step a measurement of reagent "B" is taken (the amount of reagent "B" present at that time point). The overall timeseries is a curve. This curve may change based on how much reagent "A" I start with.

My question is what model should I use to predict reagent A? Will a Recurrent NN work? I have only seen RNN used in predicting the very next time steps or to classify something based on the timeseries. I am looking for the model to use time series data to predict a regression problem.


texts: Tuning C hyper parameter in Soft Margin SVM in Matlab

texts: How to tune the C 'BoxConstraint'  hyperparameter in soft margin SVM to get the best optimal value?


texts: Linear Regression of Sine wave using Gradient descent Not working

texts:
I am writing an algorithm to fit a sine wave. I want to have 4 parameters ( amplitude, frequency, phase & centre position). When I tried to program with all 4 parameters I wasn't able to find a good fit. So I tried to first make it work with each parameter separately and making the other 3 constant hardcoded values. I managed to make this work for all parameters except for the frequency. So I think the problem is the derivative of the error function w.r.t the frequency parameter. As seen below all the other parameters of the sine are hardcoded values except the frequency parameter

%This is iterated for a large number of times
for i = 1:length(x)
%computing hypothesis and error
    TrialY(i) = 4* sin(w2(end) *x(i) + 4) -2;
    Error(end) = Error(end) + (TrialY(i) - y(i))^2;
end
Error(end) = Error(end) / (2*length(x));

dw2 = 0; %derivative value

%derivative of error function with respect to frequency (w2)
for i = 1: length(x)
    dw2 = dw2 + (4* sin(w2(end) *x(i) + 4) -2 - y(i)) * 4 * x(i) * cos(w2(end) * x(i) + 4);
%hardcoded derivative
end
dw2 = 2 * dw2 / length(x); %normalising error

w2(end + 1) = 0;

w2(end) = w2(end - 1) - alpha * dw2; % alpha is the update constant

It seems like the parameter is not converging on the right value with whatever alpha and initial values I give it. While the plot of the Error with time is decreasing but converging on a value > 0.


texts: What are the inputs to a logistic regression? Probability or trial result?

texts: It is a very basic question, but cannot find a satisfactory answer to. When we do logistic regression, what are the inputs? Suppose we have a dataset of students giving number of hours each student spent studying, and the end result of whether they passed or failed. The wikipedia article on LR goes to do a fitting with the logistic curve based on probability as a function of number of hours. But how do I get the probabilities if all I have is whether they passed or failed?


texts: Prediction for Current month based on last month's Labels

texts:
I have monthly data of loan installment repayment. The data contains basic features like salary, age, gender, credit score, etc. Along with the above features, I have the data for the last 6 installment failure/success. Now based on this, I want to predict which customers are going to default next month.
The problem I would like to highlight, to apply any machine learning algorithm, I need labels for the same month, i.e., for Feb 20 predictions, I need some rows with labels success/failure in Feb 20 (so that I can train model on this) and then can predict on the remainder of data from Feb 20.
But here I don't have any labels for Feb 20 (I have data for past failures instead).


texts: Method to create master product database to validate entries, and enrich data set

texts:
We have a ruby-on-rails platform (w/ postgreSQL db) for people to upload various products to trade. Of course, many of these products listed are the same, while they are described differently by the consumer (either through spelling, case etc.) "lots of duplicates"

For the purposes of analytics and a better UX, we're aiming to create an evolving "master product list", or "whitelist", if you will, that will have users select from an existing list of products they are uploading, OR request to add a new one. We also plan to enrich each product entry with additional information from the web, that would be tied to the "master product".

Here are some methods we're proposing to solve this problem:

A) Take all the "items" listed in the website (~90,000), de-dupe as much as possible by running select "distinct" queries (while maintaining a key-map back to original data by generating an array of item keys from each distinct listing in a group-by.)

THEN

A1) Running this data through mechanical turk, and asking each turk user to list data in a uniform format.

OR

A2) Running each product entry through the Amazon products API and asking the user to identify a match.

or

A3) A better method?


texts: How does TF-IDF classify a document based on "Score" alloted to each word

texts: I understand how TF-IDF "score" is calculated for each word in a document, but I do not get how can it be used to classify a test document. For example, if the word "Mobile" occurs in two texts, in the training data, one about Business (like the selling of Mobiles) and the other about Tech, then how does the "score" for word "Mobile", in both training and test document over the given dataset, help the algorithm to classify whether the text (a new test document) belongs to "Business" category or "Tech" category? I'm new to NLP, thanks in advance!


texts: What does the relative sum of squares error signify in neural network analysis in SPSS?

texts: What range of values of the relative sum of sqaures error is acceptable for a good neural network? I am getting around 0.9 of the relative error for 1 model and around 0.4 for another. Are both the networks significant?


texts: Feature selection for gene expression dataset

texts:
I am searching for a feature selection algorithm which selects features that are:

relevant to discriminate groups of samples (for each sample a group label is provided)
endowed with high variance across all the samples

This should be applied to gene expression dataset, in which each sample has a group label, therefore it should be possible to select for each group a set of features to be checked against.

I have now two candidates:

selecting features by the feature importance result of a Random Forest classifier
using the Minimum Redundancy Maximum Relevance (mRMR) algorithm

However, I am unsure of which may be the best or if there are better candidates for this purpose.

If the algorithm is implemented in Python scikit-learn it would be a plus.


texts: Hide element in the table of contents in google colab

texts:
How do we hide an element from the table of contents in google colab? I want to hide some elements of the table of contents but keep others.
In the example, I would like to hide SubSubtitle 2.1 but keep all other elements in the Table of contents on the left.


texts: How to approach mapping families of vectors on a lattice and forecast resulting value

texts:
I describe here a model to describe how neighbours influence a node.

I wish to implement it to attempt forecasting to values associate nodes;
I post here asking for suggestions on mathematical model and machine learning techniques that could have already considered a similar approach, but I am not aware of, and hints for their implementation (python).

Suppose you a have a squared 2D lattice (a grid of 9 squares for simplicity), and:

for each time t
from each cell in the grid

you have vectors pointing to or from the central square.

Like this example, at time t1 :

I want to "map" the vectors to a parameter, associated to the central square.

I think vectors could be modelled as "a matrices X " ( or a family of vectors), and a resulting value y, but I am actually struggling to correctly represent it in a formal way.

Like in this second image:

I then want to forecast the possible y value in the future.
I think I could explore linear or non linear models, like regressions or forecasting applied to times series:

but I want to train on the time-series of vectors, not on the time-series of ys (Hope it s clear).

Could you help to suggest models to look at and possible implementations (python), which may have already approached this dynamics?

Could you also suggest approach to generalise to manifolds structures (3d) ?

As examples for generalisation, I am thinking at:

a lattice of 3D ( neighbors cubes influences a central cube);
Ys as vectors, not scalars ( see final image to explain what I mean ) where a resulting vector is interpreted as the resulting vector at the next step in the future t + 1


texts: Matlab simulation through FIS and Markov Process

texts:
I need to simulate for an academical project how the traffic fluxes (input/output with respect to a monitored area, measured in number of cars) of a city area evolves in correspondence of an event (i.e. the opening of a restricted traffic area to decongest the traffic).
I have some simulated sensors that provide the data: I was thinking to use a combination of a fuzzy system (to assign a membership function to each type of data, e.g. PM10 value and CO2 value) and a markov process: I would need to modify the probability to decrement the number of car in the monitored area (simulating that a car is going out the congested area, towards the new opened area) basing on decisions made by means of a fuzzy system.
So my questions are:

It is a good way to interpret the problem or there are better ideas that I have not taken into account yet?
How to implement such a combination of markov chain and fuzzy systems in matlab?

Thanks


texts:
cols = ['permit_creation_date', 'current_status_date','filed_date', 'issued_date', 'completed_date',
           'first_construction_document_date', 'permit_expiration_date']

df_raw[cols] = df_raw[cols].apply(pd.to_datetime)

I wrote this code to convert some of the columns in the dataframe to convert the datatype to a datetime. When I run the cell in jupyter notebook it takes about 1 min in runtime. Any thoughts on how I could speed up the runtime.


texts: I have a labeled dataset and I created a duplicate of this dataset and removed the labels and applied K-means clustering with k= the number of labels in the original data set I want to compute similarities between labels to test the efficiency of the algorithm any ideas?


texts:
I'm running a random forest model using the randomForest package in R, using the TuneRF function. I have the option to set the 'step factor', which is how much the mtry parameter is changed at each interaction. It allows me to set an step factor of <1 which doesn't make sense to me because I don't see how it can change the number of variables in a model by less than 1?
What is going on here?


texts: How can I evaluate my sequence prediction model?

texts:
I want to evaluate the performance of my prediction model , which is an VED (Variational Encoder Decoder) used for sequences prediction (it predicts the next sequence knowing the actual)
I want to evaluate the model by calculating the similarity between the input sequence (x_train) and the predicted one.

I found that there is a score called BELU can be used for this, but I'm not sure.
If someone has an idea, please help me .


texts: How can I combine two datasets and assign factors to each?

texts:
I have a very basic question (just starting with R).

I have two datasets (xlsx files) with one variable (x) and 11 observations.

I need to combine these two tables to one table with the variable x (ending up with 22 observations and 1 variable).

But I still need to be able to distinguish which observations belonged to group1 and which to group2 --> so I guess I have to add a Factor like A for group1 and B for group 2.

How can I add this factor so that it remains with the values after combining/stacking the datasets?

Thanks already in advance!


texts:
I have a dictionary containing people and the distance between each pair in the following format:
{
    "ID_person1": {"ID_person2": 100, "ID_person3": 50},
    "ID_person2": {"ID_person1": 100, "ID_person3": 40},
    "ID_person3": {"ID_person1": 50, "ID_person2": 40},
}

Since I have all distances between pairs, I'm using K-means to divide them into k clusters (groups), given k, where all elements in a group are as close as possible to the other elements in the same group.
Since K-means is iterative, is there any ohter way to detect crowds with better performance.


texts: Create price matrix from tidy data without for loop

texts:
I have a dataframe in the following format

     symbol     name        date   close      market
0    BTC  Bitcoin  2013-04-28  134.21  1500520000
1    BTC  Bitcoin  2013-04-29  144.54  1491160000
2    BTC  Bitcoin  2013-04-30  139.00  1597780000
3    BTC  Bitcoin  2013-05-01  116.99  1542820000
4    BTC  Bitcoin  2013-05-02  105.21  1292190000
...
          symbol    name        date     close  market
752109    W3C  W3Coin  2018-04-07  0.000277       0
752110    W3C  W3Coin  2018-04-08  0.000286       0
752111    W3C  W3Coin  2018-04-09  0.000270       0
752112    W3C  W3Coin  2018-04-10  0.000286       0
752113    W3C  W3Coin  2018-04-11  0.000300       0

I want to create a matrix with prices and a matrix with marketcaps. Because I'm interested in re-creating the S&P500 index. I have done this with a for loop below. Question: How do I do this without a for loop?

edit: Note that some symbols have different number of dates, for example BTC might have 1000 days but W3Coin only 50 days. I think that is what the error message is telling me.

import data:

df = pd.read_csv('data-coins-scrape/CryptoData.csv', parse_dates=True)
cols = ['symbol', 'name', 'date', 'close', 'market']
df = df[cols]
symbols = df.symbol.unique()
nr_coins = df.symbol.nunique()

for loop to get prices and makretcaps

prices = []
marketscaps = []
shapes = []
for s in symbols:
    # slice out one coin
    dfc = df.loc[df.symbol == s]
    shapes.append(dfc.shape)
    prices.append(dfc['close'])
    marketcaps.append(df['market'])
del dfc

check the output

len(prices), len(marketcaps)
# (1531, 3065) is the output

type(prices[0]), prices[0].shape, prices[0].name
# output: (pandas.core.series.Series, (1810,), 'close')

df.loc[df.symbol == 'BTC', 'close'].head() == prices[0].head()
# returns True


texts: I have a use-case to train graph embeddings, looking for a way to do it in PyTorch and TensorFlow. The restriction is the methodology should incorporate edge weights in calculating transition probability. Can someone please advise the best framework/library to do? The node2vec implementation by the authors is in spark only, however, my current ML pipeline is more PyTorch/TF friendly.


texts: Regularization hyperparam tuning during training

texts:
I have an idea for a regularization-hyperparam selection method, which I haven't encountered before and can't find on Google, but I'm sure someone has already tried it and I'm wondering what are the best practices.
The most common method for hyperparam selection is to select different hyperparams (e.g some value for L2 regularization), train NNs with them, and test the NNs on some validation set - and select the best one.
My idea is to train a single NN and test the NN on a validation set between epochs, and then auto-adjust the regularization hypeparam between epochs - if we see that the accuracy on the validation set is decreasing between epochs, then we should increase the value of the L1/L2/dropout.
Naturally, this can be more efficient than training multiple NNs.
It's still a basic idea, and I'm sure it can be developed further. Is there research and best practices in this field?


texts: What is the purpose of linear activation functions in keras, isn't the entire point of activation functions to introduce non-linearity?


texts: Regression model Giving the same prediction for all new inputs until i load the model again

texts:
I have build a regression model that has some decent accuracy measures.

I have pickled it and loading it another project. However it is producing the same predictions every time when i pass new elements. After again i reloaded the model and gave the same set of new inputs then it gave new predictions and again it is producing the same outputs again and again though i have passing different inputs. This scenario repeating until i reload the model again.

The algorithm i have used Linear Regression, RandomForest Regressor.

Is there any cache issue or object defaulting it to first inputs. i was confused to understand the behavior of it. how can i proceed further.

Please anyone can suggest me. it would be really appreciated. Thank you.

Edit (add code from comments)

import pickle # Dump the trained model with Pickle

model_filename = 'lrmodel.pkl' # # Open the file to save as pkl file
model_pkl = open(model_filename, 'wb')
pickle.dump(model, model_pkl, protocol=2) # Close the pickle instances
model_pkl.close() #Load the Model
model_pkl = open(model_filename, 'rb')
model = pickle.load(model_pkl) #Prediction of Model model_pred =
model.predict(df_test_model)


texts: How to move forward on Regression problem

texts:
I'm an undergrad interested in machine learning, and I'm playing around with some data in order to get a better understanding of the field.

DATA

I'm working with the following data:

To give you an idea of what you're looking at (sorry for the unlabeled axes) -- each data point corresponds to a basketball player's points per game (y-values) over a one-week period (x-values).

END GOAL

I want to predict progress throughout the year, so in order to test/train, I start with the first two points, fit a regression model, and then predict the third point. I then add the actual value of the third point to the set, re-train, and predict the fourth point, etc.

ISSUES

As you can see, the data is all over the place, and the predictions are just as messy (sometimes getting up to ~200 points per game, which is totally impossible).

I've tested different degrees of linear regression (quadratic, cubic, etc.) and degree=1 is always the best predictor because of how wonky the data is.

IDEAS

I have thought of the following ways to get more accurate predictions:

Smoothing the data, maybe using a moving average or some variant
Set an upper limit for predictions
Non-linear regression

But outside of smoothing the data, I'm not sure if the rest are even possible in a regression model (upper limit) or applicable to this situation (non-linear).

QUESTION

Are any of the ideas I had above worth pursuing? If not, is there anything I should look into that might help me solve this problem?

Thanks!


texts: Identifying software and version range in a sentence

texts:
I have sentences similar to the following format

  This vulnerability happened in Firefox 1.x before 1.8, Safari
  2.x before 2.8.

Given the above sentence, I want to extract a dictionary

{Firefox: 1.0-1.8, Safari: 2.0-2.8}

Problem is how should I identify the version range with the software they belong to, using NLP techniques?


texts: Google TPU: when/how will it be available to me?

texts:
Google recently announced the TPU custom chip.

They stated it is available in Google Cloud Platform, but only for their internal usage.

When will I be able to use it? Will it be possible to buy this chip or will it be only accessible through their cloud?


texts: Neural Networks: How to prepare real world data to detect low probability events?

texts:
I have a real world data set of credit borrowers (50,000 records). The set contains categories such as Married, Single, Divorced, etc. as well as continuous data such as Income, Age, etc. Some records are incomplete or contain outliers. The dependent variable is Defaulted / Good (0,1). We are trying to train a neural network to predict defaults based on the training data. I have experience with neural networks and used them for sample data with great results, however, I never had to normalize noisy real world data.

Any thoughts what I should keep in mind in respect to:
 - How to normalize the categories. Can I assign an indexed number? Should I tried to stratify them?
- How to deal with missing data. Assign 0 ?
- How to deal with the fact that defaults are only about 5 % of the data set. What transfer function would be useful to predict these low probabilities.
- Basically any other real world data advice is very much appreciated.

Thanks in advance!


texts: My models performs better with the arbitrary random feature. How can I interpret this?

texts:
I am training 6 different classifiers 'Decision Tree', 'Random Forest', 'Logistic regression' and 'SVM' with different kernels. There are about 80 dependent variables including categorical and numerical variables.
For my experiment, I added a 'random' column which is generated by any arbitrary random numbers, but all the model performs better on both validation set and test set. Is there any good explanation about this phenomenon?


texts: Extending k-means clustering to include interactions between elements

texts:
I'm searching for the right literature on how to optimally divide a set into a fixed number of bins with pre-determined sizes where the contribution of each element to the objective depends on the other elements in the same bin.

For example, in a scheduling context, certain employees work better than others so that the total output of one worker includes both the individual worker's productivity and his/her added (subtracted) productivity due to the other members on the team. Suppose I want to create a fixed $K$ number of teams of size $P$ that maximizes overall productivity, or the one that creates teams with the greatest difference in productivity.

Another example from finance would be one where I want to group assets together to minimize variance. Since assets covary, grouping assets together depends both on the variance of each asset and its covariance with other assets.

I've considered several streams of literature that don't quite fit:

Bin Packing - doesn't fit since I have a fixed number of bins
Knapsack - again, fixed number of objects that I want to take, but with the added complexity that the value of an object is both its inherent value and the additional value it gains when taken with something else.
$k$-means clustering - I think this is the closest, but the measures of distance does not depend on which other objects are in the same cluster, and also cluster sizes are not fixed.

Any suggestions on literature that is related if it is already out there or perhaps thoughts on ways to tackle the problem? If this is the wrong forum, is there a better place to ask this question?


texts: Training Inception V3 based model using Keras with Tensorflow Backend

texts:
I am currently training a few custom models that require about 12Gb GPU memory at the most. My setup has about 96Gb of GPU memory and python/Jupyter still manages to hog up all the gpu memory to the point that I get the Resource exhausted error thrown at me. I am stuck at this peculiar issue for a while and hence any help will be appreciated.

Now, when loading a vgg based model similar to this:

from keras.applications.vgg16 import VGG16
from keras.models import Model

import keras

from keras.models import Model, Sequential
from keras.models import Input
input_shape = (512, 512, 3)
base_model = VGG16(input_shape=input_shape, weights=None, include_top=False)

pixel_branch = base_model.output
pixel_branch = Flatten()(pixel_branch)

new_model = Model(inputs=base_model.input, outputs=pixel_branch)

text_branch = Sequential()
text_branch.add(Dense(32, input_shape=(1,), activation='relu'))

# merged = Merge([new_model, text_branch], mode='concat')
merged = keras.layers.concatenate([new_model.output, text_branch.output])

age = Dense(1000, activation='relu')(merged)
age = Dense(1000, activation='relu')(age)
age = Dense(1)(age)

# show model
# model.summary()
model = Model(inputs=[base_model.input, text_branch.input], outputs=age)

When I just run a jupyter cell with this code and monitor the GPU usage using nvidia-smi, it is 0% .
However, I replace the code in the above Jupyter cell with the following:

from keras.applications.inception_v3 import InceptionV3
from keras.models import Model
import keras
from keras.models import Model
from keras.models import Sequential

from keras.models import Input
input_shape = (512, 512, 3)
base_model = InceptionV3(input_shape=input_shape, weights=None, include_top=False)

pixel_branch = base_model.output
pixel_branch = Flatten()(pixel_branch)

new_model = Model(inputs=base_model.input, outputs=pixel_branch)

text_branch = Sequential()
text_branch.add(Dense(32, input_shape=(1,), activation='relu'))

# merged = Merge([new_model, text_branch], mode='concat')
merged = keras.layers.concatenate([new_model.output, text_branch.output])

age = Dense(1000, activation='relu')(merged)
age = Dense(1000, activation='relu')(age)
age = Dense(1)(age)

# show model
# model.summary()
model = Model(inputs=[base_model.input, text_branch.input], outputs=age)

The GPU usage goes crazy and suddenly almost all the memory is over in all the GPUs even before I do model.compile() or model.fit() in Keras!

I have tried both allow_growth and per_process_gpu_memory_fraction in Tensorflow as well. I still get the resource exhausted error the moment I run model.fit when using the Inception based model.
Please note that I do not think this is a GPU memory error as I have about 96GB of GPU memory using an instance with 8 Tesla K80s .

Also kindly note that my batch size is 2.


texts: How to know the name of the predicted class?

texts: As we know all, after training a neural network, each neuron in the output layer presents one class. These class are in a numeric format for example 0 for sunny 1 for overcast and 3 for rainy. after the prediction phase, the model print for example 1, how to know that 1 corresponds to the class overcast?


texts: Data analysis of unstructured data that have non linear Key value position

texts:
I have unstructured data consist of three excel sheet

Text contents of 3 tables some how related to each other But there is no linear direct relationship between them e.g First row of table 1 is  not related to first row of table 2  and table 3 although they sharing same numeric and letter key but may be there is relationship between First row of table 1, second row of table 2  and first row of table 3.  so table 1 is related to both table 2 and table 3 via sharing common numeric key and letter key but not in orderly way.

You can consider Table 1 as questions table each row is individual question and Table 2 is Responses from one group of people and table 2 is response from other group of people. But in collected responses some people did not reply the answers orderly way . for example they Gave answer of first question then skip the answer of second, third and wrote answer of fourth one. all of data was collected on paper. so they just place bullet points in front of answers  without specify the the answer belong to what number of question.

I have to figure out some insight from data, so what kind of data analysis or approach should I use to analysis this unstructured data.

I am already using Power BI for creating word cloud to visualize the data. What kind of other machine learning approach I can use to figure some insights from this data


texts: What measures can I use to find correlation between categorical features and binary label?

texts:
For analyzing numerical features, we have correlation. What measures do we have to analyse the relevance of a categorical feature to the target value? If there isn't a direct measure, how can we achieve this?

Chi-squared test is known, but I can't find any implementation of it for categorical values.
One other way is to label encode into numerical values, but that assigns certain priority to higher valued labels.


texts: Why does adding data augmentation decrease training accuracy a tiny bit?

texts:
Before data augmentation, my model clearly overfits and hits a 100% training accuracy and a 52% validation accuracy. When only adding data augmentation with Keras, as a regularization technique, it achieves a 95% training accuracy with slower convergence and a 80% validation accuracy (which is a way better result). But why does the training accuracy gets reduced by around 5%?
If somebody could provide the link to a research paper or explain the reasoning behind this, it would be greatly appreciated!


texts: 400 positive and 13000 negative: how to split dataset up (train, test, validation)

texts:
Working on a medical diagnostic convolutional neural networking problem, and it's not obvious (to me) how the dataset should be split up.

Do I have enough data to split it in 3, or should I just have train and validation?

What proportion of images should I put in each?

(research article links appreciated :])


texts: Multivariate, multistep forecasting with LSTM

texts:
I want to use an RNN with LSTM to forecast multiple steps into the future, based on multiple inputs. I have some ideas for different ways to approach this, but I'm afraid I'm missing the "right way" to do it. Please let me know if any of these approaches are generally better or worse than others, or if I'm missing any.

More specifically, I have ~1,000,000 periods of financial data to train on: a stock closing price (X), the value of stock indicator 1 (Y), and the value of stock indicator 2 (Z). I want to train on that data and predict the closing price (X) up to and including 10 periods in the future.

Here's how I can imagine approaching this:

1. Separate models for predicting each input value

Have one model that predicts the next X based on the previous X, Y and Z.
Have a second model that predicts the next Y based on the previous X, Y, and Z.
Have a third model that predicts the next Z based on the previous X, Y, and Z.

Use the predictions from these three models as the inputs for the subsequent step.

2. Separate models with different period width

Train one model on every period in the training data, use it to predict one period into the future.
Train a second model on every second period in the training data, use it to predict two periods into the future.
Train a third model on every third period in the training data, use it to predict three periods into the future.

Etc

3. Train one model that predicts a vector of ten values, each representing the predicted X at each period up to ten periods into the future.

4. Train one model that predicts a vector of three values, each representing the predicted next X, Y, and Z. Use them as inputs for the subsequent step.


texts: How to explain MAE/MSE at each node of decision tree for regression in sklearn python?

texts: If the mean value at any node is 60 and MSE = 169 so RMSE is 13. Can I conclude that the error at my node is 60 +-13 i.e my values in this particular sample split ranges from 60-13 to 60+13. If not , what is mae/mse at each split and how do I interpret this ?


texts:
A ml beginner here, so please bear with me. If I understand correctly RNNs seem to be the go to method right now for sequence prediction for a given input (single/as a sequence). But I do not have sufficient data to train a RNN. I have discounted Markov decision process based mechanisms for the same reason.

Are there any online learning algos that I can use to get coarse/approximate predicted sequences with only a small training set?
I have looked at Q-learning but it seems to be ideal for best path problems where the end goal is definite.

Any pointers would be greatly appreciated.

UPDATE: Adding more clarity on the type of data post the comments.
My data is video content consumption data. About 100 users consuming from a library of 1000 video titles. Intent is to exploit (if it exists) the likelihood of consuming content in a probable sequence. I currently have 300 - 400 such sequences spanning 3 videos each.


texts: R - newdata has X rows but variables have X rows

texts:
I have a dataset dimensions 1142obs in 454 variables.
I've used 'caret' to separate into training and testing datasets.
training =858 obs of 99 var
testing =284obs of 99 var

I make a linear regression model like so (apologies in advance for the extra $ sign before response, which I include here otherwise it won't show correctly):

lm1 <-lm(training$$response ~ training$balloons, data=training)

I then try to make a prediction like so:

lm.predict <-predict(lm1, newdata=testing)

R comes up with:

Warning message:
'newdata' had 284 rows but variables found have 858 rows

lm.predict also generates a vector 858 numbers long, instead of 284.
Any ideas?


texts: Spatial Data Mining and ATM banking transactions

texts:
I have the following dataset. This dataset contains ATM banking transactions in different places over 30 days. These ATMs are located in different cities and provinces. The problem that I need to solve is to develop a model that can be able to detect which ATMs are in the same city or  in the same province. The only thing I know about this problem is that is related to Spatial Data Mining but I have no idea which techniques, algorithms, and tools I should use to solve this problem.

In this dataset PS ID column corresponds to the ATM IDs, AMOUNT column corresponds to the amount of money transferred.

My question is which techniques, algorithms, and tools I should use to solve this problem???

Should I use clustering or I should use other techniques?


texts: making conclusions after sentiment analysis

texts:
After performing some sentiment analysis, I have a dataset that looks like this:
For different products, using online reviews, I have obtained some values for positive/negative sentiments. However, now I am unable to figure out how to draw conclusions for this.
I had the idea of using correlation but need ideas on what features could be created & what comparisons could be made?
The dataset includes different "Features" like webcam, screen, mousepad for different products (product name).
id     Date       Website       Product Name    Brand   Stars   Feature   Sentiment  Positive     Negative       Anger      Happiness       Annoyance
0     2020.8.03   eBay    Lenovo Hi-Fi 320    Lenovo     4      Screen      NEGATIVE    0.000047    0.999851    0.000101    0.108132    0.248220


texts:
A year or two ago I wrote a keylogger that has been quietly running in the background of my computer. Each line consists of a timestamp, a keycode, and a value representing modifier keys (e.g. ctrl, shift, etc.).

Separately, I have a web app that I use to manually make notes about how I spend my time: TaskRanger.com

As you can see in this screenshot, Task Ranger is basically a daily log of work done. Each day consists of a list of intervals. Each interval consists of a start time, a duration, and a string briefly describing the work I did during that period. The description also contains a twitter style hashtag that manually classifies the type of work I was doing.

I'm interested in combining both of these data sources to create a classifier that can take a raw log of keystrokes as input and output a sequence of tags indicating the type of work I did over time. For example, I might have been doing research from 11am to 1pm, then planning a project from 1pm to 2:30, then alternating between coding and debugging for the next few hours, etc.

The intent is to basically build a personal profiler. I want to become more productive. And I think I ought to be able to treat my workflow as a software program: figure out what the bottlenecks are, drill down into them, and focus my improvement efforts on those areas.

I tried to build some kind of naive classifier myself without much luck. One problem is I couldn't find an easy way to figure out when one kind of work ends and the next kind begins (segmentation I think it's called?). I think this whole classification task might be a lot harder than it initially seems. I have this vague idea that a Hidden Markov Model should work. Basically the type of work that I'm doing is a hidden state, and the keylogs provide clues as to that state. But I know next to nothing about signal processing or machine learning, so I'm not sure if I'm on the right track here.

Any advice or suggestions on how to build this would be appreciated. Some sample code demonstrating something similar would be especially useful.


texts: Currency Normalization for Salary Prediction

texts:
I have a dataset (350k data points) with data of employees across different regions over the last 10 years. The dataset consists of their skills, the region they are in, the industry, their current role, their salary in the respective currency.
After doing some analysis, I have found 60% of the salaries are in SGD, 30% in INR, and the rest are divided across 15 other currencies.
Is it recommended that I have a model for each currency or is there a way I can convert all the currencies to a universal value so I can use all my data points to train?
Currently, I have used the 40% of the points available in SGD to train a random forest model and I have found that the results on the test set are reasonably accurate. For this model, I have considered skills, role, and industry as features and nothing else. Is there any better model I can explore?
Thank you


texts:
I am working on this dataset.
Dataset has missing values. What would be the best method to impute the missing values. Also values are missing from target feature as well. So far I have dropped those observations from the dataset.

There are multiple instances of the same ID for which some variables will not change i.e. age, weight, height etc., but some values are missing. I am trying to impute the missing values by other given values for the same ID. How can I go about it in Python? Thanks in advance.


texts: When would one use Manhattan distance as opposed to Euclidean distance?

texts:
I am trying to look for a good argument on why one would use the Manhattan distance over the Euclidean distance in machine learning.

The closest thing I found to a good argument so far is on this MIT lecture.

At 36:15 you can see on the slides the following statement:

"Typically use Euclidean metric; Manhattan may be appropriate if different dimensions are not comparable."

Shortly after, the professor says that, because the number of legs of a reptile varies from 0 to 4 (whereas the other features are binary, only vary from 0 to 1), the "number of legs" feature will end up having a much higher weight if the Euclidean distance is used. Sure enough, that is indeed right. But one would also have that problem if using the Manhattan distance (only that the problem would be slightly mitigated because we don't square the difference like we do on the Euclidean distance).

A better way to solve the above problem would be to normalize the "number of legs" feature so its value will always be between 0 and 1.

Therefore, since there is a better way to solve the problem, it felt like the argument of using the Manhattan distance in this case lacked a stronger point, at least in my opinion.

Does anyone actually know why and when someone would use Manhattan distance over Euclidean? Can anyone give me an example in which using the Manhattan distance would yield better results?


texts: Xgboost regressor : prediction overestimated or underestimated

texts:
Hi,
Recently I am working on a project about synthesizing wind turbine power by the other wind turbines, like the graph above , axis X is wind turbine A power and axis Y is wind turbine B power. All the data points were derived from 10 minutes SCADA time-series data.
For now, I used xgboost regressor to predict predict the power A or B, my features are poor , power difference between A and B , wind shear , wind veer , air density and Power A/B . The metric for the model is good, but the vertical or horizontal data points which I draw the red frames in the graph make some of the predict results overestimated or underestimated since one x points or y points map to several different points.
May I ask some advice to avoid this issue ? Or Should I add some specific data engineering in addition ?
Thanks in advance .


texts: Unable to print in Jupyter Notebook using Pandas

texts:
I am doing basic data analysis on an csv file in jupyter notebook

def answer_two():
    return (df['Gold']-df['Gold.1']).argmax()
answer_two

The above code snippet is to subtract two columns of the dataframe.I am expecting an answer in the form of a country name but I am getting the following output.

<function __main__.answer_two>

I am unable to figure out why this is happening .Occasionally ,the required output in the form of country name is coming but not running it everytime.


texts:
At the moment I'm playing with Restricted Boltzmann Machines and since I'm at it I would like try to classify handwritten digits with it.

The model I created is now a quite fancy generative model but I don't know how to go further with it.

In this article the author say, that after creating a good generative model, one "then trains a discriminative classifier (i.e., linear classifier, Support Vector Machine) on top of the RBM using the labelled samples" and furtherly states "since you propagate the data vectors to the hidden units of the RBM model to get hidden unit vectors, or a higher-level representation of the data". The problem is that I'm not sure if I get that right.

Does that mean all I have to do is propagate the input to the hidden units and there I have my RBM feature for classification?

Can somebody explain this process to me?


texts: What is Pruning & Truncation in Decision Trees?

texts:
Pruning & Truncation
As per my understanding
Truncation: Stop the tree while it is still growing so that it may not end up with leaves containing very low data points. One way to do this is to set a minimum number of training inputs to use on each leaf.
Pruning is a technique in machine learning and search algorithms that reduces the size of decision trees by removing sections of the tree that provide little power to classify instances.

Can somebody explain the in-detailed implementation of these techniques in GBDT frameworks like XGBoost/LightGBM.

Which parameters are used in implementing these techniques?


texts: Run ML model on a distributed manner on dataframe rdd partitions

texts:
When running a logistic regression on a large dataset with repartition, how do we stitch the results of the each partition results back to a dataframe to analyze.
from pyspark.ml.feature import StringIndexer, VectorAssembler, OneHotEncoder, StandardScaler
from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression

model_indexing = StringIndexer(inputCol= 'model', outputCol= 'model_index')

one_hot_encoding_model = OneHotEncoder(inputCol="model_index", outputCol="model_encoded")

label_stringIndexer = StringIndexer(inputCol = 'is_usage_higher', outputCol = "label").setHandleInvalid("skip")

assembler_inputs  = model_columns +  ['model_encoded']

assembler = VectorAssembler(inputCols = assembler_inputs,
                            outputCol = "features")

scaler = StandardScaler(inputCol = "features",  outputCol = "scaledFeatures")

Stages = [model_indexing, one_hot_encoding_model, label_stringIndexer, assembler, scaler]

lr_model = LogisticRegression(featuresCol='scaledFeatures',labelCol='label')

def model_parallelly(partition_df, model_pipe):
    train, test = partition_df.randomSplit([0.8,0.2])
    modelPipeline = Pipeline().setStages(Stages + [model_pipe])
    run_model = modelPipeline.fit(train)
    predictions = run_model.transform(test)
    return predictions

model_partition_df = df.repartition('hour')

lr_model_result =  model_partition_df.rdd.mapPartitions(lambda part: model_parallelly(part, lr_model))

What are different ways to run a ML model in parallel on a large dataset


texts:
I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there other methods?

My search results came up with a solution

df['col']=(df['col']-df['col'].min())/(df['col'].max()-df['col'].min())

But this works only with panda dataframes (they have column names). Is there a way to implement column headers in the matrix?

import pandas as pd
import numpy as np
import random
import sklearn.preprocessing import StandardScaler

data = pd.DataFrame({'sepal_length': [3.4, 4.5, 3.5],
                     'sepal_width': [1.2, 1, 2],
                'petal_length': [5.5, 4.5, 4.7],
                'petal_width': [1.2, 1, 3],
                    'species': ['setosa', 'verginica', 'setosa']})

#Shuffle the data and reset the index
from sklearn.utils import shuffle
data = shuffle(data).reset_index(drop = True)

#Create Independent and dependent matrices
X = data.iloc[:, [0, 1, 2, 3]].values
y = data.iloc[:, 4].values

#train_test_split
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1, random_state = 0)

#Impute missing values at random
prop = int(X_train.size * 0.5) #Set the % of values to be replaced
prop1 = int(X_test.size * 0.5)

a = [random.choice(range(X_train.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
b = [random.choice(range(X_train.shape[1])) for _ in range(prop)]
c = [random.choice(range(X_test.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
d = [random.choice(range(X_test.shape[1])) for _ in range(prop)]
X_train[a, b] = np.NaN
X_test[c, d] = np.NaN

This is where I get the error :Input contains NaN, infinity or a value too large for dtype('float64').

from sklearn.preprocessing import StandardScaler #importing the library that does feature scaling

sc_X = StandardScaler() # created an object with the scaling class

X_train = sc_X.fit_transform(X_train)  # Here we fit and transform the X_train matrix
X_test = sc_X.transform(X_test)


texts: Information about LTSM RNN backpropagation algorithm

texts:
I am attempting to make a LTSM RNN in python from scratch and I have completed the code for forward pass but I am struggling to find a clear outline of the equations I need to calculate to get the gradients using back-propagation. Is there any straightforward resource that I can learn these equations from and how to implement them(as my programming skills are limited)?
Thanks for any help,

The notation for my code is that; hu is the update gate, hf is the forget gate and ho is the output gate.

def forward(inp,target):
    loss = 0
    c_temp,c,x,a,y,prob = {},{},{},{},{},{}
    c_old = {}
    c[-1] = np.zeros((hidden_size,1))
    a[-1] = np.zeros((hidden_size,1))
    for t in range(len(inp)):
        x[t] = np.zeros((vocab_size,1))
        x[t][inp[t]] = 1
        X = np.concatenate((x[t],a[t-1]))
        c_temp[t] = tanh(wc @ X  + bc)
        hf[t] = sigmoid( wf @ X + bf)
        hu[t] = sigmoid( wu @ X + bu)
        ho[t] = sigmoid(wo @ X + bo)

        c[t] = hu*c_temp[t] + hf * c[t-1]
        a[t] = ho * tanh(c[t])

        y[t] = wy @ a[t]  + by
        prob[t] = softmax(y[t])
        loss += loss(prob[t],target[t])


texts: Working with others: Tidy data vs Pretty data

texts:
When working with someone whose background and skill level in data work may not be strong, how do you best make the argument for tidy data over "pretty" data?
There are notes of what I want to ask/discuss in this StackExchange post, but I want to ask more about the collaborative aspect.
Part of my job is to collect data from many different organizations which are all members of a common group. The spreadsheets they send me are untidy in ways as varied as the companies themselves. I'm sure you're familiar. A thought I keep having is that I could go meet with each company and help them organize what data they have if they're amenable to that. My own superiors are fond of this idea as well.
So the conversation I'm imaging is with, say, GizWidCo, whose product data lives in a spreadsheet that someone lovingly crafted to have the information they need at a glance, formatted in a way they find aesthetically pleasing, with adjacent tables, variables in column names or even table headers, with cells that contain multiple values. Because that's how they've always done it. Because they don't want to spend all day running analyses, they just need this information stored somewhere. They're experts in widget design and gizmo production, not number crunching.
Do you have any experience guiding someone toward using better data management principles? I don't want to just tell someone how to do their job, and I don't even want my influence to serve only my needs either (even if getting them to send me better data is the impetus of all this). My hope is that I can educate someone who could take value from improving their data management and thus also improving all related processes. I'm intending this as a win/win.
If you have any thoughts, perspective, or lived experience here, I would love to hear it. If this is too off-topic for a StackExchange question, I would be grateful to know the right forum. (For what it's worth, I first asked this on Cross Validated and they suggested here.)


texts: How to interpret PR and ROC Curve for an unbalanced test set

texts:
I have trained a neural network on a dataset, the test set is very unbalanced, ratio between positive examples and negatives is 1:25000.
All positive examples are correctly predicted, instead negatives elements correctly predicted are 99% of total negatives.

Plot of PR and ROC curves are those:

What can be inferred from these curves?
Those are my firsts works with classifiers and i'm confused.
I think that precision is always low, because the negatives that are wrong predicted as positive have an high score assigned by the classifier (close to 1).
ROC instead i think that is high because all positive examples are correctly predicted. These are my suppositions, correct me if I am wrong.


texts:
I have trained an LSTM in PyTorch on financial data where a series of 14 values predicts the 15th. I split the data into Train, Test, and Validation sets.  I trained the model until the loss stabilized.  Everything looked good to me when using the model to predict on the Validation data.
When I was writing up my research to explain to my manager, I happened to get different predicted values each time I ran the model (prediction only) on the same input values. This is not what I expected so I read some literature but was not able to explain my results.  Intuitively, my results indicate there is some random variable, node, gate, that is influences the prediction, but I cannot figure out where this is or if/how this can be configured.
Here is my model definition:
class TimeSeriesNNModel(nn.Module):
    def __init__(self):
        super(TimeSeriesNNModel, self).__init__()
        self.lstm1 = nn.LSTM(input_size=14, hidden_size=50, num_layers=1)
        self.lstm2 = nn.LSTM(input_size=50, hidden_size=25, num_layers=1)
        self.linear = nn.Linear(in_features=25, out_features=1)

        self.h_t1 = None
        self.c_t1 = None
        self.h_t2 = None
        self.c_t2 = None

    def initialize_model(self):
        self.h_t1 = torch.rand(1, 1, 50, dtype=torch.double)
        self.c_t1 = torch.rand(1, 1, 50, dtype=torch.double)
        self.h_t2 = torch.rand(1, 1, 25, dtype=torch.double)
        self.c_t2 = torch.rand(1, 1, 25, dtype=torch.double)

    def forward(self, input_data, future=0):
        outputs = []
        self.initialize_model()

        output = None
        for i, input_t in enumerate(input_data.chunk(input_data.size(1), dim=1)):
            self.h_t1, self.c_t1 = self.lstm1(input_t, (self.h_t1, self.c_t1))
            self.h_t2, self.c_t2 = self.lstm2(self.h_t1, (self.h_t2, self.c_t2))
            output = self.linear(self.h_t2)
            outputs += [output]

        outputs = torch.stack(outputs, 1).squeeze(2)
        return outputs

If anyone can point out what is wrong with my model or my understanding, I'd be really grateful.
UPDATE 1: Researching the Forget Gate
I suspect that the variability is introduced by the LSTM Forget Gate although I cannot see how.
In "Deep Learning", by Goodfellow, Bengio, and Courville, on page 399, the authors present the formula for the forget gate as:

the self-loop weight (or the associated time constant) is controlled by a forget gate unit $f_i^{(t)}$ (for the step $t$ and cell $i$), which sets this weight to a value between 0 and 1 via a sigmoid unit:
$$f_i^{(t)} = \sigma (b_i^f + \sum_j U_{i,j}^fx_j^{(t)} + \sum_j W_{i,j}^fh_j^{(t-1)})$$
where $x^{(t)}$ is the current input vector and $h^{(t)}$ and is the current hidden layer vector, containing the outputs of all the LSTM cells, and $b^f$, $U^f$, $W^f$ are respectively the biases, weights, and recurrent weights for the forget gates.

Again, from "Deep Learning" by Goodfellow et al. page 397, the authors state,

A crucial addition has been to make the weight of the self-loop conditioned on the context, rather than fixed (Gers et al., 2000).  By making the weight of this self-loop gated (controlled by another hidden unit), the time scale of integration can be changed dynamically. In this case, we mean that event for an LSTM with fixed parameters, the time scale of integration can change based on the input sequence, because the time constants are output by the model itself.

I read this as saying the time frame that can impact $y^{(t)}$ is variable based on the input sequence, but the sequence itself is fixed, so I would expect consistent predictions from the forget gate
What I don’t understand is how this would introduce a stochastic effect. It seems that the weights, once trained, are constant, and the input don’t change like any other NN, right?


texts: How to transpose rows into single column based on time-stamp index in python?


texts: Need for Dense layer in Text Classifcation

texts:
While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense Layer?
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(encoder.vocab_size, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(1)
])

Consider the following sentence in 5 class classification:
"movie is good" . The model structure could be:
a = activation_unit
emb= embedding_vector(word)

a0 -> emb("movie") ->a1->emb("is") ->a2->emb("good") ->a3, and

sample_y = softmax(np.dot(Wya,a3))

and
sample_y = [0.1,0.2,0.2,0.4,0.1]

which says the sentence belongs to "class 4". So where is the need for a "Dense Layer"? Can anyone please explain this


texts: Computing Rare values After SMOGN - Imbalanced Regression

texts:
I am dealing with a regression problem where I have the phenomenon "Imbalanced Regression". In my problem, the most relevant events are scarcely represented. In order for me to evaluate my models' predictions, I have stubmled upon this paper that deals Imbalanced Regression problem as a classification task.

The authors define a relevance function:
Φ belongs to [0,1] which maps the target variable domain into a scale of relevance,
where 1 corresponds to the maximal relevance and 0 to the minimum relevance.

With a relevance function defined we can determine the sets of normal and rare
values. To achieve this, the user is required to set a threshold tR on the relevance values.

Given this threshold we can formally define the set of rare and relevant cases, DR, and the
set of normal and uninteresting cases, DN , as follows: DR = { ∈ D : φ(y) ≥ tR} and
DN = { ∈ D : φ(y) < tR}.

The authors use an R code for computing rare values as follows:

 if (both) {  # we have both low and high extrs
    # see this part of the code.
    rare.low <- which(y.relev > thr.rel & y < pc$control.pts[4])
rare.high <- which(y.relev > thr.rel & y > pc$control.pts[4])
    rare.cases <- c(rare.low,rare.high)
    total <- length(rare.cases)
  } else {
    # the indexes of the cases with rare target variable values
    rare.cases <- if (pc$control.pts[2] == 1)  which(y.relev > thr.rel & y < pc$control.pts[4]) else which(y.relev > thr.rel & y > pc$control.pts[4])
    total <- length(rare.cases)

  }

Knowing that y.relev can be computed using this R package as well as this one

What I want to do

In my project, I am interested on computing the percentage of rare values after doing SMOGN (in this paper, they compute the percentage of rare values before applying SMOGN) in order to see the effect of applying SMOGN on my data. Having this said, should I:

Define a new relevance function for the newly 'SMOGN'-ed y obtained?
use the same relevance function as before ? (I actually tried this, but SMOGN adds some noise to the data and thus generates some new values from y, so mapping the old relevance values to the newly generated ones won't work out)


texts: How to find feature importance with multiple XGBoost models

texts:
My problem statement  : Time Series forecasting(Month wise data), training on 96 months of data and predicting next 12 months with a 3 months empty window in between.

Example :

Batch 1

***Training Data index*** <2010-01-01 -----------------------------2017-12-01>

***Unused Month Window*** <2018-01-01---2018-03-01>
***Test Month*** <2018-04-01> [Trained model with Batch 1 training data
                               can ONLY be used for predicting this month,
                               not any other]

Batch 2

***Training Data index*** <2010-01-01 -----------------------------2018-01-01>

***Unused Month Window*** <2018-02-01---2018-04-01>
***Test Month*** <2018-05-01> [Trained model with Batch 1 training data
                               can ONLY be used for predicting this month,
                               not any other]

and so on till Batch 12...

I am training 12 XGBoost models to get predictions for each of the 12 months of FY 18, hence getting  12 different feature importances against each model for the predictors used. But i want to report the feature importance of entire FY 18, instead of giving 12 different set of feature importances against each month. How would I approach that??

Evaluating a single model on the entire test dataset is not an option.

Any help is appreciated. Thanks.


texts: Large no of categorical variables with large no of categories

texts:
I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1).

Most of my features are categorical with large number of categories. For example: one of them has 310 categories but the top 10 most frequently occurring variables account for ~50% of the training and test data.
I was thinking of keeping these top k frequently occurring values and encoding all the other values as another category "others".

Would it be okay to do so?

Also, what kind of classifier would be optimal for such a task? I was looking at random forests. However, due to noisy data (too many categories, too many features) my model doesn't generalize too well (low recall and precision).

Thank you for your time.

PS: Data not time series


texts: Which ML algorithm should I use for following use case for classification and Why?

texts: I have data in table format having total 3 columns. One column for label, other two columns are features. So, such 30 rows(1 row contains 2 feature and 1 label) make one set of data with all 30 rows having same. In these 30 rows there could be some patterns of features. There could be more such sets of 30 rows(for 30 rows label will be same). So there will be 6 to 7 labels or class. I want algorithm to predict one label from 30 rows(one set). Which algorithm should I choose and how should I pre-process data?


texts: PyTorch does not converge when approximating square function with linear model

texts:
I'm trying to learn some PyTorch and am referencing this discussion here

The author provides a minimum working piece of code that illustrates how you can use PyTorch to solve for an unknown linear function that has been polluted with random noise.

This code runs fine for me.

However, when I change the function such that I want t = X^2, the parameter does not seem to converge.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

# Let's make some data for a linear regression.
A = 3.1415926
b = 2.7189351
error = 0.1
N = 100 # number of data points

# Data
X = Variable(torch.randn(N, 1))

# (noisy) Target values that we want to learn.
t = X * X + Variable(torch.randn(N, 1) * error)

# Creating a model, making the optimizer, defining loss
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.05)
loss_fn = nn.MSELoss()

# Run training
niter = 50
for _ in range(0, niter):
    optimizer.zero_grad()
    predictions = model(X)
    loss = loss_fn(predictions, t)
    loss.backward()
    optimizer.step()

    print("-" * 50)
    print("error = {}".format(loss.data[0]))
    print("learned A = {}".format(list(model.parameters())[0].data[0, 0]))
    print("learned b = {}".format(list(model.parameters())[1].data[0]))

When I execute this code, the new A and b parameters are seemingly random thus it does not converge. I think this should converge because you can approximate any function with a slope and offset function. My theory is that I'm using PyTorch incorrectly.

Can any identify a problem with my t = X * X + Variable(torch.randn(N, 1) * error) line of code?


texts: What is the interpretation for quadratic functions?

texts:
I am working through the book Applied Predictive Modeling and came across something that was a bit confusing.

It discussed adding non linearity to a model to improve its fit - I get this part.

For example: $x^2 + 2x - 4$

What is the interpretation of these values though?

When we are using just a normal linear regression or a multivariate regression, we would say that the coefficients like 2 would suggest its relative importance relative to the other features being included in the model. However, what does this mean in the context of quadratic functions?

ie. Fuel efficiency of a car based on 2 Displacement + Displacement$^2$ -4

What exactly does displacement squared mean?

Any help would be greatly appreciated.

Thanks.


texts: How to apply class weight to a multi-output model?

texts:
I have a model with 2 categorical outputs.
The first output layer can predict 2 classes: [0, 1]
and the second output layer can predict 3 classes: [0, 1, 2].

How can I apply different class weight dictionaries for each of the outputs?

For example, how could I apply the dictionary {0: 1, 1: 10} to the first output,
and {0: 5, 1: 1, 2: 10} to the second output?

I've tried to use the following class weights dictionary
weight_class={'output1': {0: 1, 1: 10}, 'output2': {0: 5, 1: 1, 2: 10}}
But the code fails with an error.

My script also runs normally when i remove the class_weight parameter

Code Example

I've created a minimal example that reproduces the error

from tensorflow.python.keras.models import Model
from tensorflow.python.keras.layers import Input, Dense
from tensorflow.python.data import Dataset
import tensorflow as tf
import numpy as np

def preprocess_sample(features, labels):
    label1, label2 = labels
    label1 = tf.one_hot(label1, 2)
    label2 = tf.one_hot(label2, 3)
    return features, (label1, label2)

batch_size = 32

num_samples = 1000
num_features = 10

features = np.random.rand(num_samples, num_features)
labels1 = np.random.randint(2, size=num_samples)
labels2 = np.random.randint(3, size=num_samples)

train = Dataset.from_tensor_slices((features, (labels1, labels2))).map(preprocess_sample).batch(batch_size).repeat()

# Model
inputs = Input(shape=(num_features, ))
output1 = Dense(2, activation='softmax', name='output1')(inputs)
output2 = Dense(3, activation='softmax', name='output2')(inputs)
model = Model(inputs, [output1, output2])

model.compile(loss='categorical_crossentropy', optimizer='adam')
class_weights = {'output1': {0: 1, 1: 10}, 'output2': {0: 5, 1: 1, 2: 10}}
model.fit(train, epochs=10, steps_per_epoch=num_samples // batch_size,
          # class_weight=class_weights
          )

This code runs successfully without the class_weight parameter.
But when you add the class_weight parameter by uncommenting the line
# class_weight=class_weights than the script fails with the following error:

Traceback (most recent call last):
  File "test.py", line 35, in <module>
    class_weight=class_weights
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1536, in fit
    validation_split=validation_split)
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 992, in _standardize_user_data
    class_weight, batch_size)
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1165, in _standardize_weights
    feed_sample_weight_modes)
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1164, in <listcomp>
    for (ref, sw, cw, mode) in zip(y, sample_weights, class_weights,
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py", line 717, in standardize_weights
    y_classes = np.argmax(y, axis=1)
  File "venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 1004, in argmax
    return _wrapfunc(a, 'argmax', axis=axis, out=out)
  File "venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 62, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 42, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
numpy.core._internal.AxisError: axis 1 is out of bounds for array of dimension 1

Edit

I've also opened an issue in the Keras github page, but i wanted to ask the same question here to see if perhaps i'm missing something and doing something wrong.


texts: Why does my model sometimes not learn well from same data?

texts:
I have a dataset of 2 classes, both containing 2K images. I have split that into 1500 images for training and 500 images for validation.

This is a simple structure for testing purposes, and each image is classified depending on the colour of a particular pixel. Either Green or Red.

I have run this model many times and I find that sometimes the models gets   low loss/ high accuracy within a few epochs, but other times it gets stuck at accuracy 50%.

The datasets are exactly the same each time with only difference coming from model.fit "shuffle" option.

I tested the LR Range first:

and I "cycle" the learning rate through an appropriate range.

model = keras.Sequential([
        keras.layers.Dense(112, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.4),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])

LRS = CyclicLR(base_lr=0.000005, max_lr=0.0003, step_size=200.)

model.fit(train_images, train_labels, shuffle=True, epochs=10,
              callbacks=[checkpoint,
                         LRS],
              validation_data = (test_images, test_labels)
              )

Why does the model sometimes NOT get a good fit?

  EDIT 1

Re Serali's suggestion:

myOpt = keras.optimizers.Adam(lr=0.001,decay=0.01)
model.compile(optimizer=myOpt, loss='categorical_crossentropy',  metrics=['accuracy'])

reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2,verbose=1,patience=5, min_lr=0.00001)


texts: Why does test data need to be normalized on train data mean and std?

texts:
I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data.

Why is it that we can not take the mean and std of the whole dataset combined (train and test data) and then normalize around these values?


texts:
I'm using Kaggle's titanic set. I'm using pieplines and I'm trying to prune my decision tree and for that I want the cost_complexity_pruning_path. The last line of code produces the error:
   ValueError: could not convert string to float: 'male'
Do you know what I'm doing wrong? I have looked at Sklearn: applying cost complexity pruning along with pipeline but that doesn't seem to be helping in my case
cat_vars = ['Sex','Embarked']
num_vars = ['Age']

num_pipe = Pipeline([('imputer', SimpleImputer(strategy='mean')),('std_scaler', StandardScaler())])
cat_pipe = Pipeline([('imputer', SimpleImputer(strategy='most_frequent')),('ohe', OneHotEncoder())])

col_trans = ColumnTransformer([('numerical', num_pipe, num_vars),('categorical', cat_pipe, cat_vars)] ,remainder='passthrough')

final_pipe = Pipeline([('column_trans', col_trans), ('tree', DecisionTreeClassifier(random_state=42))])
final_pipe.fit(X_train, y_train)

path = final_pipe.steps[1][1].cost_complexity_pruning_path(X_train, y_train)


texts: What is the minimum/suggested sequence length for training an LSTM?

texts:
My dataset consists of short videos of 4/5 time-steps each (frames), and the problem is classifying this video (multi-label classification). The idea is to use an LSTM but I'm wondering if the sequence length is not enough.
 What are the suggested sequence length? May 4/5 time steps be enough?

P.S. could you please post some link to scientific articles to endorse your claims?