texts:
I'm using the tsfresh library to extract features on my data but I keep getting an error:
ValueError: Could not guess the value column! Please hand it to the function as an argument.

In the source code for tsfresh feature extraction I see:
def _get_value_columns(df, *other_columns):
    value_columns = [col for col in df.columns if col not in other_columns]

    if len(value_columns) == 0:
        raise ValueError("Could not guess the value column! Please hand it to the function as an argument.")

    return value_columns

Which seems to indicate that my column length is 0 but when I print the length of my column, I see it is 64.
print(len(df.temperature))

[64 rows x 1 columns]
64

Here is the code I am working with:
from tsfresh import extract_features, extract_relevant_features, select_features
from tsfresh.utilities.dataframe_functions import impute

df = pandas.DataFrame(data=tempArray)

df.columns = ['temperature']
print(len(df.temperature))
extracted_features= extract_features(df, column_id='temperature')
impute(extracted_features)

Thanks so much!


texts: Machine Learning: Repetitive Pattern recognition regardless of amplitude and frequency

texts:
I am trying to scope out an approach to learn to identify repetitive human body activity based on collected motion data and would like some advice on some challenges I identified. I determined a set of data, the approach, and a set of potential challenges using cosine waveforms as an analogy before I start on the actual work.
Data:
Self generate the data by creating a long series of cosine waves of varying frequencies and amplitudes.
Machine Learning approach:

Sequence input with X dimensions
LSTM with Y hidden units
Z fully connected layer
Softmax
Classification Output with cross entropy

Challenge:
One particular challenge I will face is that the network is likely to only learn to classify cosine waveforms based on the frequencies and amplitudes that appears in the training set.
How should I modify my approach to make it less sensitive to the frequency and amplitude and more on the signature cosine waveform that we see?


texts: Time series prediction without sliding window

texts:
I've got a collection of historic log files for a process that repeats daily, roughly the same each day.
I want an NN that, from the moment that the daily process starts, estimates what time the whole process will finally complete.
As the daily process progresses, the estimate should become more accurate and converge on the actual completion time as soon as possible.
A degree of confidence in the prediction would be useful too.

The daily process is a collection of mostly serial subprocesses. Mostly serial because some subprocesses have no impact on the end time ( critical path vs non ).
There are other pattens hidden in the data. Weekends take longer than weekdays. In the real world, maybe others too.
There is a small amount of noise in the data, subprocess duration can be affected by completely external factors.
The performance of one given day has little / no expected correlation with the next day, although there may be long term trends.

I've had a little success with a simple RNN but results have been underwhelming, even after trying a lot of hyperparameters.
Best results so far have been from one-hot encoding and "cross joining" to expand each day into multiple samples and padding them to the same maximum dimensions...

Xsample1.1 = [Event1, ]              Y = EventN
Xsample1.2 = [Event1, Event2, ]      Y = EventN
Xsample1.N = [Event1, ..., EventN ]  Y = EventN

I've encoded weekday as one-hot variables and normalized datetimes to one universal day in preprocessing.

All the examples I find are for predicting the next N samples, sliding forward, not a fixed end sample.
I'm using ipython3, pandas and keras and I've generated a test population  using some simple logic I'd expect an NN to easily handle. YHat for each file is the last datetime. The .debug files capture how the prediction should evolve but that information is not available from the training data and so is not used in the NN.

Is this a time series or regression problem, what type of NN should I use and how should i structure the input data for best results.


texts: NLP: How to group sub-field into fields?

texts:
Suppose I have a list of strings that captures a sub-field of academic research and would like to group them as higher-level fields.
For example,

'Quantum Mechanics'  => 'Physics'
'Abstract Algebra'   => 'Mathematics'
....

My understanding is that standard NLP techniques may not fit here, because the relationship between sub-fields and fields are linked by its meanings but not word-frequency or word-embedding etc.

I wonder if there is anything done that could be useful to tackle this problem (papers or packages)?


texts: Is addition of a random control variable a good idea, to ensure that results are correct?

texts:
I'm dealing with a complex dataset with many patients who have a condition, and various qualities about these patients.  I'm trying to determine patient outcome based on patient qualities.
I'm using xgboost Classifier to try to determine one of the patient outcomes.
In order to ensure that my results are making sense, I've added a random variable, rand which is a random number within (0,10) that will have no impact on patient outcome.
The rand variable should show last in variable importance.
rand is meant as a control variable.
Is adding a control variable like rand a valid method to ascertain that my model makes sense?


texts: Choosing the first node in a decision tree, basic example

texts:
I'm wondering whether I'm understanding the process of choosing a node correctly and would like to see if this example makes sense.
using the following data :

> df
  Y A B C
1 1 0 1 1
2 0 0 0 1
3 1 1 1 0
4 1 0 1 0
5 1 1 0 1
6 1 0 0 1

I will split the data on A,B,C and evaluate the entropy of each split, where the entropy is computed using

$$
p \; \log_2(p) + (1-p)\; \log_2(1-p)
$$

where $p$ is the proportion of successes for a particular split.

When splitting on A I have

  Y A B C
3 1 1 1 0
5 1 1 0 1

and

  Y A B C
1 1 0 1 1
2 0 0 0 1
4 1 0 1 0
6 1 0 0 1

For $A = 1$ (the first data table) I have $p = 1$, then entropy is 0.

for $A=0$ (second table) I have $p=0.75$ and entropy 0.81

So for splitting on $A$ I would state that the entropy is

$$
0 + 0.81 = 0.81
$$

This is then carried out in a similar manner for $B,C$.

For $B=1$ I find $p=1$ so entropy = 1

For $B=0$ I find $p=0.66$ so entropy = 0.91

Then the entropy for splitting on $B$ is

$$
0 + 0.91 = 0.91
$$

For $C=1$ I find $p=0.75$ so entropy = 0.81

For $C=0$ I find $p=1$ so entropy = 0

Then the entropy for splitting on $C$ is

$$
0.81 + 0 = 0.81
$$

Given the above the split which has the highest entropy is $B$, therefore I would choose to split on $B$ first.

I now have a decision tree with one node, and need to select the next node for each branch of $B = 1$ and $B=0$.

This selection is carried out in the same manner as the above.

Is the computation and reasoning in the above valid?


texts: How to change random number of random elements in a column by group in R?

texts:
I have the following data frame:

>df
   Name
 1 Ed
 2 Ed
 3 Bill
 4 Tom
 5 Tom
 6 Tom
 7 Ed
 8 Bill
 9 Bill
10 Bill

My goal is that from each group by "Name" change the "Name" values 25-75% of random rows to "Name"+"_X" (the remaining rows don't change). So the expected output is similar to:

   Name
 1 Ed
 2 Ed_X
 3 Bill_X
 4 Tom
 5 Tom_X
 6 Tom
 7 Ed_X
 8 Bill
 9 Bill
10 Bill_X

I have tried with for cycle like this (a the moment, for 50% random rows:

for (n in unique(df$Name)){
  df[sample(which(df$Name==n), nrow(df[df$Name==id,])/2), 1] <- paste(df$Name, "_X", sep="")
}

Unsuccessfully, however.


texts: Intuitive explanation of difference between PCA and SVD

texts: Can someone explain the difference between SVD and PCA with real life example?


texts:
I have data which have complete information. Each record has one class assigned. On production, I won't be able to get so many information from a user, so I want to create a model which will be able to make inferences from incomplete data and making more and more reliable predictions with getting more information from a user.

For Example:

Say,

Training data - 200 binary features, In rows in average 11 of them
are marked as 1, another 189 as 0.
Production - We give our user
    list of features to mark which of them is most common(Let's suppose
    that user picked 3 of them), then I want to make some inferences and
    ask good questions to get more accurate predictions.

My second problem is that my dataset assumed that user can belong only to one class, whereas in fact user on production can be multi label.
Say, in my dataset there is situation that:
-class 1 has 5 features annotated as true at random(allways the same), rest always false
-class 2 has 5 DIFFRENT features annotated as true at random(allways the same), rest always false

Notice, I have a list of possible features for each class, so I can easily calculate conditional probability of class depends on features, so I tried to use Naive Bayes, but my features are not independent.

On production can occur situation that user will be in class 1 and 2 and will have some features from class 1 and some from class 2. I want to that my algorithm will figure it out that it is in class 1 and class 2.

What algorithm should I use? I was thinking about decision trees, but in that, I cannot give user choice what is the most common for him and in that situation.

Thanks in advance for your help!


texts: How to best visualize data when outliers lead to lack of contrasting colors for the rest of the plot?

texts:
I have plotted the following map of Philippines according to the population in its regions, using matplotlib.pyplot in Python 3:

As can be seen, there's one province (in red) that is an outlier. When looking at the data, it is clear:

 population = [16.892118644067804,
 27.13259740259741,
 27.325479452054793,
 25.502352941176472,
 29.62087999999999,
 25.593870967741932,
 27.649384615384616,
 25.38908450704225,
 26.924931506849322,
 26.636250000000004,
 27.317105263157895,
 26.58338345864662,
 29.48689393939393,
 28.283986013986016,
 26.563118279569903,
 23.037142857142857,
 24.2674]

This is making the plot "unattractive" - there would have been a lot more contrasting colors for the other regions had this one region's data not been an outlier. What options do I have to rectify this?

Remove this region's data. In that case, this region would appear white (uncolored), and the rest will have contrasting colors. But for whatever reason, that's not an option for me currently.
Manually tweak that one data point to "blend in" with the other data points (something like a mean of all the other data points, or even a handcrafted value that is reasonably close to the other data points, like 22, or whatever). I tried this, and this does give more contrasting colors. The trouble is, this is misrepresentation of data. So, if possible, I would like to avoid it.
Stretch the yellow further down in the colormap, so that only a little space is given to red. That is, instead of the 2 colors starting to transition from one to the other around 24 in the colorbar, it should do so around, say, 19. Anybody knows how to do that?

Any other ideas? What are some of the best practice approaches to tackle this issue?


texts: Training a LSTM/any other deep learning model with temporal as well as non temporal attributes

texts:
I'm working on a project to predict the usage of all the files(rough frequency of usage) in a filesystem(a company server on which 100s of company employees are active) in near future(say the next 1 month) based on the metadata of the file system for past 6 months. I've got the following attributes about the files with me :

The temporal sequence of file usage for last 6 months(whenever the
file was read/written/modified and by whom).
All the users who are on the server and can access the files.
Last modified/written/read epoch time and by whom.
File creation epoch time and by whom.
Any compliance regulations on the file(whether the file contains any
confidential data).
Size, name, extension, version, type of the file.
The number of users who can access the file.
File path.
The total number of times accessed.
Permitted users.

Now, I plan to use LSTM but for standard LSTMs, the input is temporal sequence only. However, all the attributes that I have seem significant in predicting the future usage of the file.

How should I also make use of the attributes of the file that I have?
Should I train a Feedforward Neural Network, disregarding the fact
that it usually fails on temporal sequences?
How should I proceed?
Does a variant of LSTM exist that can take into account the
attributes of the file as well and predict the usage of the file in
near future?
Do I need to use MLP and LSTM together like a hybrid?


texts: MinMaxScaler returned values greater than one

texts:
Basically I was looking for a normalization function part of sklearn, which is useful later for logistic regression.

Since I have negative values, I chose MinMaxScaler with: feature_range=(0, 1) as a parameter.

x = MinMaxScaler(feature_range=(0, 1)).fit_transform(x)

Then using sm.Logit trainer I got and error,

import statsmodels.api as sm
logit_model=sm.Logit(train_data_numeric_final,target)
result=logit_model.fit()
print(result.summary())

ValueError: endog must be in the unit interval.

I presume my values are out of (0,1) range, which is the case:

np.unique(np.less_equal(train_data_numeric_final.values, 1))

array([False,  True])

How come? then how can I proceed.


texts: How do I get Twitter Dataset for Visualization

texts:
I am a newbie. I would like to do visualization on twitter data : top trends based on country (over map) and time variations (for each months in 1 year or for each year) . Can someone tell me where can I get the twitter data set and any advice on how to start proceeding would be really help full.

Thanks


texts:
how to write following cnn structure in simple words for better understanding
CNN with 11 layers 3×3×32 Convolutional → Dropout → 2×2 MaxPool → 3×3×64 Convolutional
→ Dropout → 2 × 2 MaxPool → Flatten → Dropout → 1 × 1024 Full
connected → Dropout → 1024 × 10 Fully connected → Softmax


texts: Adding multiple columns to spark dataframe

texts: I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and add_column while allowing for a user-defined function doesn't seem to allow multiple return values - so does anyone have  a recommendation how I would accomplish this? I'd prefer only calling the generating function d,e,f=f(a,b,c) once per row, as its expensive.


texts: A fundamental question about the gradient descent equation

texts:
So in a classic gradient descent we have

W -= epsilon * (dL / dW)

However, how does this make sense if we consider that L and W have some units? Wouldn't it be more sensible to consider instead

W -= epsilon * (dW / dL)    (where epsilon now represents a small nudge in loss and has corresponding units)

Note also that if we are not on a point with 0 derivative we can consider a "local inverse" of the function L(W) and we would have that dL/dW = 1/ (dW/dL). Therefore the signs of these two alternatives are equal - the only thing that differs is the size of the modification we apply to W.

In the standard way we take a larger step on a steeper region. "My" variant does the opposite. Imagine that a change of W results in a very small change of loss: then it would make sense to overcompensate for this during the update of W, in line with my proposed method. Where am I wrong?


texts: Do you apply outlier detection of numerical data in practical applications?

texts:
In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data.

Do you apply outlier detection in your usual routine? By this I mean: among $N$ training data decide $K<<N$ data points to be outliers and discard them.

What are your experiences doing this? Which methods do you apply for numerical data.


texts: How will a decision tree cope if it cannot find a suitable feature to choose as root node from which to split further?

texts: When I look at decision trees, they start with a root node choosing the most suitable feature from which to split further. What if the decision tree is unable to find the most suitable feature from the data as root node? How does the decision tree cope in that situation?


texts: How to measure the Optimal-number of Topics & Topic-Coherence score in Orange's LDA Topic-Modeling?

texts:
I'm trying to build a suitable "code-free" Topic Model in "Orange Data Mining" software for the experimental part of my Thesis, in order to analyze the themes and trends of research articles' corpus dataset. The model then can be used by normal researchers who are not necessarily have programming & coding skills in order to apply their own Topic Modeling workflows.
The pre-dataset (corpus) were successfully collected as (.CSV/.TAB) format.
The right selection of the optimal model, number of topics, and the visual results should be simple, applicable, and reliable for best understanding the relationship & coherence between the discovered topics.
In-terms-of text preprocessing steps, Orange provides a nice widget that contains all-steps-in-one interface.
The issue with Orange's Topic Modeling approach is that I couldn't find a clear method (or widget) for measuring either the optimal-number of topics or the topic-coherence score in order to evaluate the topics extracted from the Latent Dirichlet Allocation "LDA" algorithm in Orange's Topic-Modeling widget.
ِAs in many literature-reviews, the usual solution to this issue is usually through through programming, which will be included in the code-script section of my Thesis, using Jupiter-Python code similar to the method described in the following article:
LDA Topic Modeling using Python
The code-cells are assembled into a single file attached with the following link:
Full Python Code
Here I got 2 questions:

Is it possible to use the same steps described in the previous article's coding-cells of Python, that are related with selecting the optimal-number of topics & topic-coherence score calculations into Orange, so that it can be transformed into one or mode connected widgets in Orange; leading to much easier use by normal researchers?.

What is the necessary widgets' sequence that better visualizing the extracted topics in order to better analyze the research articles' themes & trends?.

Thanks for your collaboration in advance..


texts: Multi-class classification with mostly zero valued data

texts:
I implemented a multi-class classification and wanted to test it using the MNIST dataset. I realized that if I use standardization

$X \leftarrow \frac{X-mean(X)}{std(X)}$,

over 50% of all features will be zero. Is that a problem?

Does it make more sense to work in such a case with normalization

$X \leftarrow \frac{2(X-min(X))}{max(X) - min(X)} - 1$,

such that all features are between -1 and 1?

What about doing first a standardization followed by a normalization step?


texts: Distance calculation/vector range significance

texts:
I'm trying to implement item based collaborative filtering. Do any distance calculations allow for weighting of certain ranges of values within each vector?

For example, I would like to be able to say values 10..22 within each vector are more significant than values within the range 0..10.

I've been experimenting with Pearson, Tanimoto and Euclidean algorithms, but they all seem to assume equal weighting for each value within the vector.

Am I approaching this problem in the right way, and if not, how do others deal with this problem?


texts: Weighting the loss function based on previous seen true positive rates

texts:
Similiar to class imbalance there is always something I would call "learnability imbalance" in multi-class classification. What I mean by that: Even when the classes are evenly distributed in the dataset some classes will be classified more easily by the model than others. An example would be a CNN model that classifies dog, cat and car. Dog and cat will most likely have a lower true positive rate than car because cats and dogs look more similiar to each other. But I would like evenly distributed recall/precision/f1-scores/true-positive rates of all classes.
Here is the solution I tried:

train model normaly
evaluate the model and return a list with the true positive rates of each class
use some function to create weights using the true positive rates with which I can weight the CrossEntropyLoss
train again using the weighted loss function

I implemented this for a CIFAR-10 classifier. To calculate the weights I tried a lot of stuff for example
# car, dog, cat
true_positive_rates = [0.9560878243512974, 0.7365384615384616, 0.7807933194154488]

average = np.mean(true_positive_rates)
loss_weights = [pow(1 - ((score - average) / score), 4) for score in true_positive_rates]

>>> print(loss_weights)
[0.743631046434835, 1.2530321887900169, 1.1150155156144919]

I tried different exponents to create more or less extreme weights (in this example 4).
Since Im using Pytorch the weighting looks like this:
nn.CrossEntropyLoss(weight=loss_weights)

My results:
Even though classes were more difficult to classify than others as I though, giving them a higher weights made almost no difference.
Here is a figure (the blue bars are the true positive rates from the normal first model, the orange bars are from the second model which was trained using the loss that was weighted with the true positive rates from the first model):

As you can see there is no improvent (desired result would have been that all bars have the same height). Is there a obvious reason why this doesnt work?


texts: Best Approach and Classifier for Binary Classification Problem

texts:
I am trying to build a binary classifier, and I am wondering what is the best approach for data segmentation, training/testing, performance evaluation, selecting classifier type, and overall approach to this?
Data type: Data from 100 subjects. For each subject, multiple instances (Ni for the i-th subject) of measurements were 30 seconds segment of 3 channel ECG were taken, and 5 features were extracted for each channel and each 30 seconds segment of ECG. Hence the dataset is 100 x Ni x 3 x 5, where Ni is the number of measurement taken for the i-th subject (i=1 to 100). The outcome for the subject is binary, and hence say 0 and 1.
In this case, since there are multiple and variable numbers of measurements, how should that fact should be adjusted for training and testing?


texts: Earlystopping in multi-output deep learning

texts:
When working with a neural network with more than one output, what is generally advised as the best strategy for early-stopping the training process?

Given that I am currently monitoring the net validation loss (validation loss from n different output neurons added together), and have noticed my training early-stopping before convergence in one or more of my losses, I am curious as to what the best practices in this situation are.

Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold.
How would one implement this without risking overtraining some output nodes while preventing others from under-training?

I am a newcomer to this field, if you hadn't guessed already so please bear with me if this question is not extremely well put together!


texts: How can I tokenize a text file with BERT or something similar?

texts:
I want to use the twitter datasets in a project and the tweet contents look something like this:

tweet_ID         tweet_text

12324124         some text here that has been twitted bla bla bla
35325323         some other text, trump, usa , merica ,etc.
56743563         bla bla text whatever tweet bla bla

Now I would like to end-up with a file that contains tweet_IDs and some vector encodings. I was reading about BERT, ROBERTA, etc. Is there a way to simply generate these encodings without writing a huge amount of boilerplate code?


texts: Multivariate Gaussian distribution - Covariance vs linear dependence

texts:
From prof. Andrew Ng's Multivariate Gaussian distribution lecture, covariance measures linear dependency between features, in which case we might use Multivariate Gaussian distribution with covariance matrix. And also, if features are redundant (for ex: x1= 2 * x2; clearly linear dependency exists between features), covariance matrix is not invertible and can't use Multivariate Gaussian distribution with covariance matrix. For me, these statements looks contradictory.

Question:
Whats difference between covariance - linear dependency and features linear dependency?


texts: Error when checking input while running the LSTM model using panda python

texts:
I have a data set include with temperature, humidity and wind. Here I want to predict future temperature value in next hour.

I used LSTM to predict future temperature value.
But when I run the model it showed up this error

Can anyone help me to solve this problem?

Here is my code:

    import datetime
    import time
    from sklearn.metrics import mean_squared_error
    import matplotlib.pyplot as plt
    from matplotlib.dates import DateFormatter
    import numpy as np
    import pandas as pd
    from sklearn.preprocessing import MinMaxScaler

    from sklearn import preprocessing
    from keras.layers.core import Dense, Dropout, Activation
    from keras.activations import linear
    from keras.layers.recurrent import LSTM
    from keras.models import Sequential
    from sklearn.preprocessing import MinMaxScaler

    data = pd.read_csv('data6.csv' , sep=',')
    data['date'] = pd.to_datetime(data['date'] + " " + data['time'], format='%m/%d/%Y %H:%M:%S')
    data.set_index('time', inplace=True)
    data = data.values
    data = data.astype('float32')
    # normalize the dataset
    def create_data(train,X,n_out=1):
    #data = np.reshape(train, (train.shape[0], train_shape[1], train_shape[2]))
    x,y=list(),list()
    start =0
    for _ in range(len(data)):
        in_end = start+X
        out_end= in_end + n_out
        if out_end < len(data):
            x_input = data[start:in_end]
            x.append(x_input)
            y.append(data[in_end:out_end,0])
        start +=1
    return np.array(x),np.array(y)
    scaler = MinMaxScaler()
    data = scaler.fit_transform(data)
    # split into train and test sets
    train = int(len(data) * 0.6)
    test = len(data) - train
    train, test = data[0:train,:], data[train:len(data),:]
    X=1
    x_train, y_train = create_data(train,X)
    x_test, y_test = create_data(test,X)
    x_train=x_train.reshape(x_train.shape +(1,))
    x_test=x_test.reshape(x_test.shape + (1,))

    n_timesteps, n_features, n_outputs = x_train.shape[1], x_train.shape[2], x_train.shape[1]

    model = Sequential()
    model.add(LSTM(8, activation='relu', input_shape=(n_timesteps, n_features)))
    model.add(Dense(8,activation='relu'))
    model.add(Dense(n_outputs))
    model.compile(loss='mse', optimizer='adam')
    # fit network
    model.fit(x_train,y_train, epochs=10,batch_size=1, verbose=0)

My csv file:

My csv file.

My error:


texts:
I'm trying to build a regression model, where I see which attributes are influencing the margin. My data set looks like something below.

UserId  |  [products_bought] | revenue |  [places_visited] | discount  --> Margin

In the above mentioned schema a single user might have bought a set of products maximum of 4000 and places visited can be in hundreds.

I tried to flatten the data but this is very sparse, are there any approaches to efficiently model this kind of problem


texts: Tidying Time Intervals for Plotting a Histogram in R


texts:
In order to ensemble a decision tree, let me explain the specific situation.
I have split the dataset in 5 sections, per protocol e.g. TCP, HTTP etc.
Now I've trained a decision tree for each one and run this against my test dataset.

How do I go about combining the 5 prediction models.
Do I....

a) combine the predicted output e.g. 1,0;1,1 for each section done separate and then run it against the test dataset to identify confusion matrix. This is actual class, predicted class.
or
b) do I take the tree build and then add the additional ones to the tree model, in effect combining them.

Which method should be suitable, would option a even be a good solution?


texts: Identity covariance matrix, decorrelated data?

texts:
Why would you want to decorrelated data?

As I am reading about PCA and whitening on image data for DNN, I wonder what is the purpose of achieving the identity covariance matrix in your data is?

Is it to remove interaction between variables, thus allow simpler models to express interactions without having to compute x1*x2?


texts: I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so?

texts:
Say I have n dimensional data samples. I want to check the integrity of the features, if they are good representation of the respective classes, i.e. these features are good or not.

My plan is:

I use PCA and convert this to 2 dim data. Plot this data. See if they're separable enough.

Does the above plan sound okay for testing to see if features are any good?


texts: Strange behavior with Adam optimizer when training for too long

texts:
I'm trying to train a single perceptron (1000 input units, 1 output, no hidden layers) on 64 randomly generated data points. I'm using Pytorch using the Adam optimizer:

import torch
from torch.autograd import Variable

torch.manual_seed(545345)
N, D_in, D_out = 64, 1000, 1

x = Variable(torch.randn(N, D_in))
y = Variable(torch.randn(N, D_out))

model = torch.nn.Linear(D_in, D_out)
loss_fn = torch.nn.MSELoss(size_average=False)

optimizer = torch.optim.Adam(model.parameters())
for t in xrange(5000):
  y_pred = model(x)
  loss = loss_fn(y_pred, y)

  print(t, loss.data[0])

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

Initially, the loss quickly decreases, as expected:

(0, 91.74887084960938)
(1, 76.85824584960938)
(2, 63.434078216552734)
(3, 51.46927261352539)
(4, 40.942893981933594)
(5, 31.819372177124023)

Around 300 iterations, the error reaches near zero:

(300, 2.1734419819452455e-12)
(301, 1.90354676465887e-12)
(302, 2.3347573874232808e-12)

This goes on for a few thousand iterations. However, after training for too long, the error starts to increase again:

(4997, 0.002102422062307596)
(4998, 0.0020302983466535807)
(4999, 0.0017039275262504816)

Why is this happening?


texts:
I have the following CNN:

I start with an input image of size 5x5
Then I apply convolution using 2x2 kernel and stride = 1, that produces feature map of size 4x4.
Then I apply 2x2 max-pooling with stride = 2, that reduces feature map to size 2x2.
Then I apply logistic sigmoid.
Then one fully connected layer with 2 neurons.
And an output layer.

For the sake of simplicity, let's assume I have already completed the forward pass and computed δH1=0.25 and  δH2=-0.15

So after the complete forward pass and partially completed backward pass my network looks like this:

Then I compute deltas for non-linear layer (logistic sigmoid):

$$
\begin{align}
&\delta_{11}=(0.25 * 0.61 + -0.15 * 0.02) * 0.58 * (1 - 0.58) = 0.0364182\\
&\delta_{12}=(0.25 * 0.82 + -0.15 * -0.50) * 0.57 * (1 - 0.57) = 0.068628\\
&\delta_{21}=(0.25 * 0.96 + -0.15 * 0.23) * 0.65 * (1 - 0.65) = 0.04675125\\
&\delta_{22}=(0.25 * -1.00 + -0.15 * 0.17) * 0.55 * (1 - 0.55) = -0.06818625\\
\end{align}
$$

Then, I propagate deltas to 4x4 layer and set all the values which were filtered out by max-pooling to 0 and gradient map look like this:

How do I update kernel weights from there? And if my network had another convolutional layer prior to 5x5, what values should I use to update it kernel weights? And overall, is my calculation correct?


texts: Fitting glm without explicit declaration of each covariate

texts:
When I fit a linear model with many predictor variables, I can avoid writing all of them by using . as follows:

model = lm(target_deathrate~., data = full_data)

But for models with higher complexity, I cannot make this work:

x = glm(target_deathrate~poly(., i),data = full_data)

In these cases I have to write all variables.

How to avoid writing all variable names and include all variables in my model?


texts: How to estimate GridSearchCV computing time?

texts: If I know the time of a given validation with set values, can I estimate the time GridSearchCV will take for n values I want to cross-validate ?


texts: Regression coefficient stability measurement in a multiple linear regression particularly in r?

texts:
I have a model with 13 independent variables (all of them are significant and 2 of them are categorical variables) and 678 observations. I was wondering if there is any way to check the coefficients stability. My approach is to start from the model with one variable and then find that coefficients. Then I’ll add another variable by subset selection method and find the new variables and coefficients. I’ll continue until that all variables are in the final model. Hence, I have 15 models (2 categorical variable have 3 levels each) which have different variables (1 to 15 variables including the categorical variables levels). I was wondering if I can show all the coefficients in one graph. For instance, for X1 which is in all models I’ll have 15 coefficients, for X2 which I’ll have 14 coefficients etc.

Any feedback about how can I have all the coefficients in one graph and check their variation for all models or any other approach that help me to check the Regression coefficient stability would be grateful. Thanks.


texts: Using a feature in prediction even if it gets zero as p-value?

texts:
I created two binary classification based logistic regression models and I got these results:
Model 1:

Accuracy:  67.51%

AUC:       65.21%

Model 2:

Accuracy:  67,99%

AUC:       65,70%

The second model is created based on an additional column (Col_x).
But, when I try to use the following code to show the correlation, I got the result: (0.16402709962654458, 0.0)
from scipy import stats
stats.pearsonr(data['y'] , data['Col_x'] )

I would like to know if I can considerate Model2 an improvement of Model1 even if I have a p-value = 0.0


texts: Need machine learning algorithm to fill in time-series data

texts: I am currently dealing with a time-series data set with cyclical gaps every 30 minutes (30 minutes of data, 30 minutes of no data). Is there a relatively simple way of using scikit-learn (or some other Python library) to predict the missing data using the available data to train it? I assume this would involve a 'supervised training' approach? I have provided a graph for visual reference, the orange line is a 20-minute centralized moving average of the gap data set, and I want to fill in the data to look more like the green 60-minute centralized moving average from a separate complete data set. Thanks!


texts:
In the ResNeSt paper they say on page 4:
"despite their great success in image classification, the meta network structures are distinct from each other, which makes it hard for downstream models to build upon."
What are downstream models in this context?
Paper can be found at https://arxiv.org/abs/2004.08955


texts: How to handle different input sizes of an NN when One-Hot-Encoding a categorical input?

texts:
let's assume an input dataset that is a mix of categorical values and real values. When preprocessing this data into an appropriate NN input, OHE is recommended because it doesn't assume any order of the categories. ["Man", "Woman", "Diverse"] has no order to it so having one input that represents them all within one dimension makes little sense.

When using cross validation, the dataset often gets split into a lot smaller subsets. These subsets may not hold all categories. When using OHE of sklearn, the input set is used to determine the dimensionality. This can lead to unpredictable column counts of the networks data input. It can also lead to different categories taking different positions in the NN.

How would one process this input to feed to the NN without hard-coding all possible categories and still be able to handle varying numbers of categories in the input set?

Two intuitive ideas that don't work:

determine the dataset size (after OHE) and set the input size of the NN based off of that: Doesn't work because each CV subset would potentially have a different model and each category position in the dimensions doesn't map to other subsets
0 pad to an arbitrarily high (but likely never reached) input size: dirty, because the ordering is still not ensured and if one category is missing, all inputs may be shifted by 1+ positions

Would autoencoders help?


texts: How to use reduced dimensions (of a PCA) for detection purposes?

texts:
A general question aiming at the application of a PCA: I want to detect abnormal data points and therefore I want to use a PCA for it at first. The next step is to try several distance functions or quantils of distributions or something similar to check if it is working.
My problem is: How do I know this when I operate on the dimensionally reduced data set? Or in other words: How do I transform it back to see if the certain marked data points match the ones from the original non-transformed data?

I expect something to do like here:

How do I know that this outlier in the transformed/reduced data is sensefull? I think I have to cross-check it somehow in the original data?


texts: [Keras][LSTM] error due to shape mismatch

texts:
I have following data. Where I have 2 samples. Each sample I have 3 time steps each with 2 features. I intend to have 2 batches (to updates weights after every sample)

X=np.array([[[0.54, 0.3], [0.11, 0.2], [0.37, 0.81]],[[0.55, 0.4], [0.12, 0.3], [0.38, 0.9]]])
X=X.reshape(2,3,2)
y = np.array([[[0.2],[0.3],[0.4]],[[0.3],[0.4],[0.5]]])
y = y.reshape(2,3)

Following is my Keras LSTM code

model = Sequential()
model.add(LSTM(50, activation='relu', batch_input_shape=(2, 3, 2)))
# model.add(LSTM(50, activation='relu', batch_input_shape=(1, n_steps, n_features)))
# model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

model.fit(X, y, epochs=10000, verbose=0)

Which gives me following error. According to my understanding it is due to the size mismatch but cannot figure out the issue in my sizes. Appreciate your input

ValueError: Error when checking target: expected dense_1 to have shape (1,) but got array with shape (3,)


texts: GridSearchCV() to fine tune outputs ValueError and FitFailedWarning

texts:
I want to fine tune some parameters for my linear SVM.
This is the code:
class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import LinearSVC
    from sklearn.pipeline import Pipeline
    self.pipeline = Pipeline(
        [
            #('vect', CountVectorizer()),
            #('tfidf', TfidfTransformer()),
            ('tfidf', TfidfVectorizer()),
            ('clf', LinearSVC(

            #('clf', SGDClassifier(
              #  loss='hinge',
               # penalty='l2',
                #alpha=1e-4,
                #random_state=42,
                #max_iter=100,
                #learning_rate='optimal',
                #tol=None,

            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    from sklearn.model_selection import GridSearchCV
    from sklearn.svm import SVC
    train_df = self.read_data(train_file, lower_case)
    parameters = {'C': [1, 10],
                  'gamma': [0.001, 0.01, 1]}
    model = SVC()
    grid = GridSearchCV(estimator=model, param_grid=parameters)
    grid.fit(train_df['text'], train_df['truth'])
    print(grid)
    # summarize the results of the grid search
    print(grid.best_params_)
    print(grid.best_estimator_)

    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)

    test_df['pred'] = learner.predict(test_df['text'])
    return test_df

However, I'm having some issues as it raised some ValueErrors: ValueError: could not convert string to floatand I don't know how to deal with this information. And also:  FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. How can I solve these?


texts: When is the sum of models the model of the sum?

texts:
The response variable in a regression problem, $Y$, is modeled using a data matrix $X$.

In notation, this means:

$Y$ ~ $X$

However, $Y$ can be separated out into different components that can be modeled independently.

$$Y = Y_1 + Y_2 + Y_3$$

Under what conditions would $M$, the overall prediction, have better or worse performance than $M_1 + M_2 + M_3$, a sum of individual models?

To provide more background, the model used is a GBM.  I was surprised to find that training a model for a specific $Y_i$ resulted in about equal performance than using the overall model $M$ to predict that $Y_i$.  The $Y_i$'s are highly correlated.  In hindsight, then this is not surprising because training a model for a vector correlated with the target also is correlated with the target.

For analogy, take the case with a linear model and independent response variables.

The overall model is

$Y = X\beta$

It is trivial to see that the sum of the models is the model of the sum.

$Y = X\beta = X\beta_1 + X\beta_2 + X\beta_3 = X(\beta_1 + \beta_2 + \beta_3)$

If the $Y$'s are independent then the $\beta$'s will be as well.  This implies that each of the model coefficients will be unchanged.  Take for example a two-dimensional case (where $X$ has two columns).

For $i \neq j$, $Y_i = X(\beta_i + \beta_j) = X\beta_j + 0$


texts: is the logit transform ever actually computed in modeling process of logistic regression?

texts:
i've been tying to wrap my head around logistic regression, the logit transform, and the sigmoid function.

logit transform:

from what i understand, in practice all we want to do is maximize the following log likelihood (to get the sigmoid parameters that best fit the data):

which involves some form of gradient descent to find the weights that maximize the above expression.

what i'm a bit hung up on is i don't see the logit transform anywhere here, ie, it looks like it isn't even used in the modeling process.

so is the logit transform actually ever computed in logistic regression? or is it just used to set up the problem, as a rationale for maximizing the log likelihood (rather than computing linear least squares)?

its a bit similar to the question below, i understand that the logit is the inverse of the sigmoid, and in that sense it is "used" in the modeling process, but i don't see the log odds anywhere in the modeling process. thanks,

What is the purpose of Logit function? At what stage of model building process this logit function is used?


texts: Is this Neo4j comparison to RDBMS execution time correct?

texts:
Background: Following is from the book Graph Databases, which covers a performance test mentioned in the book Neo4j in Action:

  Relationships in a graph naturally form paths. Querying, or
  traversing, the graph involves following paths. Because of the
  fundamentally path-oriented nature of the datamodel, the majority of
  path-based graph database operations are highly aligned with the way
  in which the data is laid out, making them extremely efficient. In
  their book Neo4j in Action, Partner and Vukotic perform an experiment
  using a relational store and Neo4j.

  The comparison shows that the graph database is substantially quicker
  for connected data than a relational store.Partner and Vukotic’s
  experiment seeks to find friends-of-friends in a social network, to a
  maximum depth of five. Given any two persons chosen at random, is
  there a path that connects them which is at most five relationships
  long? For a social network containing 1,000,000 people, each with
  approximately 50 friends, the results strongly suggest that graph
  databases are the best choice for connected data, as we see in Table
  2-1.

  Table 2-1. Finding extended friends in a relational database versus efficient finding in Neo4j

Depth   RDBMS Execution time (s)    Neo4j Execution time (s)     Records returned
2       0.016                       0.01                         ~2500
3       30.267                      0.168                        ~110,000
4       1543.505                    1.359                        ~600,000
5       Unfinished                  2.132                        ~800,000

  At depth two (friends-of-friends) both the relational database and the graph database perform well enough for us to consider using them in an online system. While the Neo4j query runs in two-thirds the time of the relational one, an end-user would barely notice the the difference in milliseconds between the two. By the time we reach depth three (friend-of-friend-of-friend), however, it’s clear that the relational database can no longer deal with the query in a reasonable timeframe: the thirty seconds it takes to complete would be completely unacceptable for an online system. In contrast, Neo4j’s response time remains relatively flat: just a fraction of a second to perform the query—definitely quick enough for an online system.

  At depth four the relational database exhibits crippling latency,
  making it practically useless for an online system. Neo4j’s timings
  have deteriorated a little too, but the latency here is at the
  periphery of being acceptable for a responsive online system. Finally,
  at depth five, the relational database simply takes too long to
  complete the query. Neo4j, in contrast, returns a result in around two
  seconds. At depth five, it transpires almost the entire network is our
  friend: for many real-world use cases, we’d likely trim the results,
  and the timings.

Questions are:

Is this a reasonable test to emulate what one might except to find in a social network? (Meaning do real social networks normally have nodes with approximately 50 friends for example; seems like the "rich get richer" model would be more natural for social networks, though might be wrong.)
Regardless of the naturalness of the emulation, is there any reason to believe the results are off, or unreproducible?


texts: Is it possible to update data and retrain just one of several data series in bigquery model

texts:
I am building something very similar to this BigQuery ML example project.
My system is different in two ways:
Firstly it will need several thousand time-series so I would prefer to use the multiple-series feature rather than having thousands of individual models.
Secondly is the data is more unpredictable in the long run (rather than periodic or seasonal) so needs retraining quite often, with only local trends being detected.
The data is actually monitoring voltages in battery-operated devices, which usually drops at a linear rate, but sometimes can drop much faster depending on usage, then get recharged randomly. I am forecasting the future voltage and predicting when a critical level will happen. I've tested one model and the predictions are impressively good, ARIMA seems to balance both long-term typical behaviour with the local recent changes even though it's aperiodic.
Concerning the data-pipeline, I receive new data from each device individually at random intervals. So I need to push this new data into the model, retrain it and refresh the forecast for that device's time-series.
I can limit the number of times this refresh happens, e.g. once per hour, rather than doing it in event-based in realtime. Near realtime would be great, but it's not essential.
If I had thousands of single models, this would be trivial to design, I'd update the data (BiqQuery table) and then re-create that specific model, which takes some 30 seconds.
However having all the series in one model, it seems (from the docs) that I'd have to throw away the entire model of all thousands of series, retrain the entire thing. I don't currently know how long this would take, how it scales, but I assume it will be significantly longer and presumably more expensive. All but one of of the time series are unchanged so this seems a very wasteful operation.
So is there a way to retrain just one data-series in a BigQuery ARIMA model?
I have read the interesting notes under the limitations section for the time-series specific CREATE MODEL syntax which discusses processing thousands of series.
I have read this question but I wasn't convinced by the assertion of using warm start as he leaves no doc references, and it doesn't seem available in the ARIMA model.
I have also read this question which is more about the general necessity of retraining.


texts: Training PCA on BERT word embedding: entire training dataset or each document?

texts:
I want to reduce the dimensionality of the BERT word embedding to, let's say, 50 dimensions. I am trying with PCA. I will use that for the document classification task.

Now for training  PCA, should I train on the entire dataset by using all the word vectors from the entire data set at once that is:

pca.fit_transform([all_the_word_vectors_of_the_dataset])

or word vectors per document, that is:

for document in train_dataset:
    pca.fit_transform([word_vectors_of_current_document])


texts:
The hyperbolic tangent is commonly used as an activation function:
$$
tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}
$$
Although, it is unclear how this function is implemented to be numerically stable in frameworks like pytorch or numpy. Since $e^x$ will explode for any $x \gtrapprox 1000$, is the function being approximated in some way? Or are there static checks for big input numbers?


texts: Excel file merge with different headers but same data

texts:
I need to merge data from 1000s of excel files provided by different operations managers on productivity and other reports. The excel files have similarity of data but the headers are all custom since being different manager and different clients.

For example manager A will have a.xlx and manager b will have a.xlx, but the headers for each will be different though the data inside will be usually the same. Each day 100 different excel files are updated by all team members via new files e.g a_todays_date.xlx and manager b using /a.xlx.

Is this something that can be handled via python ML libraries.

What is the best way to merge all these data and save to the DB on a per day basis. The average data would per day would be around 15GB. The end goal is to create dashboard.


texts:
Gradient descent in $n$ dimensions.

I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve.

My question is, does the slope go down a multi-dimensional curve when the characteristics of a data set > 3?

How to see the slope going down in a data set with many characteristics?


texts: How to find appropliate algorithm to bulid a model for natural language based two data

texts:
What I would like to do
I would like to create a model to infer nationality from name and created the below data frame combining two dataset from Kaggle.
Titanic: Machine Learning from Disaster （input/titanic/train.csv）
titanic-nationalities
    PassengerId Nationality Name
0   1   CelticEnglish   Braund
1   2   CelticEnglish   Cumings
2   3   Nordic  Heikkinen
3   4   CelticEnglish   Futrelle
....

Problem
How can I find algorithm to build a first model using these two data: Nationality and Name?
Since both natural language, so I can understand that it is essencial to make them vectors and this problem would be multi-value classification.
However, I have no idea how to find algorithm to train this dataset.


texts:
I am working to create a Risk score on data where I have variables -
Invested_amount, Profit Amount, Age of Account in days, Total Trading Transactions, Profit per Transaction & Investment per transaction. Basically, I want to derive a method to calculate the Risk score, where a person's profit is more (most of the time he is winning). Here the score should be higher so that I can classify them as High Risk (Mostly wins and create a big profit), medium Risk, and Low-Risk customers (Always makes Loss).
Basically, my problem is to assign a Risk score to each customer, the higher the score more the customer is Risky. Once this score is derived then using this score we will segment the customers into three classes.
Below are the variables currently I am using…
|Variable|               Definition|
|username|               UserId to identify the records|
|Total_Freq_transaction| Total Number of trades done|
|Total_Freq_win|         Total number of trade won|
|Account_age_days|       Account activated date - Last trade date |
|Invest_Amount|          Total Amount invested in trade|
|total_profit|           Profit made on trade (pay out amount - Investment Amount)|
|Trade_per_day|          Total Number of trades done / Account age|
|Win_prob%|              Wining probability from total number of trades|

Any help would be greatly appreciated.


texts: How to find year in a text using Spacy NER?

texts:
Sample Sentence:

1. We originally formed in October 2002 as company and later we established in California in 2004.

2. ABC pvt. ltd. was incorporated in 1988 in Colarado.

3. Since our inception in 1900, we worked hard and turned into a million dollar company in 1920.

Now I want to find the year founded in each sentence. i.e., 1. 2002, 2. 1988, 3. 1900.

I would like to achieve this using NER model. Previously, I've  trained a model using Spacy NER where I annotated only the years(1. 2002, 2. 1988, 3. 1900), but the model did not extract the year in any sentence.

content_block, content_block_lower = cleaned_content, cleaned_content.lower()
attributes_values = dict(zip(attr_Lst,[dataframe_rows[attributes] for attributes in attr_Lst]))
 for attribute_key, attribute_values in attributes_values.iteritems():
if(attribute_key == "Year_Founded"):

        try:
            content_index = content_block_lower.find(str(int(attribute_values)))
            #print content_index
            if(content_index!=-1):
                #print content_block [content_index : content_index+len(str(final_converted_date))+1]
                entity_lst.append((content_index,content_index+len(str(attribute_values)),attribute_key))
        except:
            pass

Is there any better way to annotate the year and achieve my desired result?


texts:
I'm new to ML and data science. I would really like high level advice how to approach the following problem. I need to predict if an engine will fail, what I've is a sensor that give a certain value between 1 to 4 so a record might look like:

1243412312431444123123234234232423

Each record is 150 readings in length. I've a few millions of this records but in no particular order (i.e. the database doesn't contain explicit timestamp although by using some quirk database artifact I can make some records appear in order)

I've 200 different engines and a label of "failed" and "active" for each one.

So I'm looking for a predictive model that can predict (or detect) using the above data whether an engine failed or not. Any advice?

Thanks,
Eden


texts: Clustering and getting threshold to classify data points

texts:
I have real world data set (120 data points) about enterprises, containing 4 features. I would like to put these enterprises in exactly 4 categories based on the values of these specific features (an official requirement). I can either find an equation with the features as parameters; or find the 3 threshold values for each feature, that would divide my data into 4 distinct categories. There are no other inputs. I tried KMeans, but I would like more insights.
How do I determine the which method to use, and how do I compute the thresholds anyway? Thank you.
My data looks like this (simplified):

Enterprise
Number of employees
Income (currency unit)
Expenditure (currency unit)
Investments (currency unit)

First
1200
120
110
20

Second
5
60
70
30

...
...
...
...
...

Last
125
50
55
70


texts: Measuring the uncertainty of predictions

texts:
Given a multiclass classification model, with n features, how can I measure the uncertainty of the model for that particular classification?

Let's say that for some class the model accuracy is amazing, but for another it's not. I would like to find a metric that will let me decide if, for a particular sample, I would like to "listen" to the model or not.

I thought about using prediction intervals, but am not sure how to calculate them.


texts: Creating similarity metric with Doc2Vec and additional features

texts:
I have a dataset which contains many features. Each record is company that has many features.

For example...

Company A:

Keywords - data, big data, tableau, dashboards, etc.
Industry - Information Technology
Sub-Industry - Data Visualization
Total Funding - $150,000,000

I want to create a similarity metric between multiple companies, incorporating both doc2vec embeddings trained on the keyword lists as well as the additional features listed. I had a hard time searching/finding papers that did something like this. Any ideas?


texts: semi supervised learning doubt only classify points with confidence above threshold

texts:
I currently have a dataset with approximately 5% labelled points and 95% unlabelled. I would like to label some of the unlabelled points only if I am very confident and leave the rest NaN. Personally I would like to use a random forest but I am not sure if that is possible - I assume I am going to have to use some generative model?

One of the reasons I would like to do this is because the known points do not contain all the labels therefore I would like to classify as many of the unknown points as possible before using unsupervised learning on the rest.

Is there a library I could use?


texts: I do not understand the prediction result of the CNN model


texts: How to interpret long equations in Deep Learning papers

texts:
For eg. I've been studying a paper on Recommender systems using collaborative deep learning and I've just started learning.

The paper revolves around the NN representation as shown below

The hierarchical models.

Having the main optimization problem

Need help interpreting it.
Here is a link to the original paper:
http://www.wanghao.in/paper/KDD15_CDL.pdf


texts: Why does the classic Neural Network perform better than LSTM in Sentiment Analysis

texts:
My goal is to predict the polarity of some reviews (negative, positive or neutral). I tried two different neural networks:

    left_branch = Input((7000, ))
    left_branch_dense = Dense(512, activation = 'relu')(left_branch)

    right_branch = Input((14012, ))
    right_branch_dense = Dense(512, activation = 'relu')(right_branch)
    merged = Concatenate()([left_branch_dense, right_branch_dense])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit([np.array(review_matrix), np.array(X_train)], labels,epochs=2, verbose=1)
    model.save('model.merged')

    #############################################################################

    #############################################################################

    #We will try to merge two different models in a different way: Accuracy: 70

    # Prepare the review column for embedding:
    review_matrix_for_embedding = prepare_for_encoding(train_set[4].tolist(), 7000) # Shape: (1503,100)

    second_matrix = np.array(pd.concat([onehot_category, aspect_matrix],axis=1))

    left_branch = Input(shape=(100,), dtype='int32')
    # input_dim: Size of maximum integer (7001 here); output dim: Size of embedded vector;
    # input_length: Size of the array
    left_branch_embedding = Embedding(7000, 300, input_length=100)(left_branch)
    lstm_out = LSTM(256)(left_branch_embedding)
    lstm_out = Dropout(0.7)(lstm_out)
    lstm_out = Dense(128, activation='sigmoid')(lstm_out)

    right_branch = Input((7012, ))
    merged = Concatenate()([lstm_out, right_branch])
    output_layer = Dense(3, activation = 'softmax')(merged)

    model = Model(inputs=[left_branch, right_branch], outputs=output_layer)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit([review_matrix_for_embedding, second_matrix], labels,epochs=5, verbose=1)

The first one does 80% accuracy while the second one does 70%, with embedding vectors and LSTM layer. How is it possible? Is there anything wrong in my architecture?


texts: Convert CSV from an api to a dataframe format in python

texts:
I am new to python, I have extracted some reviews from a website and I used the api  of the webscrapping tool to import my data in python and the format is in csv. I want to convert this csv to a dataframe in python. Can someone guide me on how to perform this please.

Below is the code for importing the api extraction in csv format.

import requests

params = {
  "api_key": "abc",
  "format": "csv"
}
r = requests.get('https://www.parsehub.com/api/v2/runs/ttx8PT-EL6Rf/data', params=params)
print(r.text)

My output for the above codes are as follows:

"selection1_name","selection1_url","selection1_CommentID_name","selection1_CommentID_Date","selection1_CommentID_comment"
"A","https://www..html","137","February 02, 2020","I enjoy the daily package from the start with the welcoming up to the end.
I recommend this hotel."
"A","https://www.e a lot. Relaxing moments with birds chirping, different swings to chill. Overall, I shall visit again. Thanks Azuri & Marideal."
"A","https://www.html","17","June 12, 2019","Had an amazing stay for 2 nights.
The cleanliness of the room is faultless"
"B","https://www.html","133","April 16, 2019","Had a good time. Food is good."

etc...
Can you please help me to convert this into a dataframe in python please please.


texts: How important is lookahead search in decision trees?

texts:
I am using random forests, and, in my data, I have a lot of situations where $X_1$ is a bad predictor, $X_2$ is a bad predictor, but the joint distribution would make a good predictor.

Say that $X1$, $X2$ and $Y$ are binary variables. We have that $P(X_1|Y)=0.5$ and $P(X_2|Y=y)=0.5$ for any $X_1,X_2,Y\in\{0,1\}$, but $P(X_1=1,X_2=1|Y=1)=1$.

Practical example: I want to predict whether text1 is the same as text2. I have added as a feature difference-in-usage-of-!. But, difference-in-usage-of-! can be zero in two very different cases: when neither text has ! or when both use ! in the same numbers. So, I have added another feature: both-texts-use-!.

The problem now is that the decision tree needs to be forward looking because it is only true that the text is the same if difference-in-usage-of-! is very small and both-texts-use-! is 1.

Now, I probably could and should combine these two features into a single, smarter feature (and suggestions are appreciated). But I have a lot of features such as this one, and I wonder whether the decision trees in my random forest, which are learnt using traditional local search are sub-optimal, relative to lookaheads kind of search.

I read in this thesis that,

  As empirically investigated in
  [Murthy and Salzberg, 1995], such an approach (using lookahead search) is however not only more computationally intensive, it is also not significantly better than greedily induced decision trees.

Comment in parenthesis is added by myself. The work being cited is another thesis, and it indeed says that, for instance,

  The counter-intuitive result that lookahead search does not improve over greedy search would stand to reason if the latter itself constructs near-optimal trees.

Both of these works are old and, from what I read, do not delve much into it (but I have not read them fully). I wonder if anyone knows if there are features for which only some kind of lookahead search would work, or whether regular the regular local search algorithm is good even for those cases, and whatever thoughts you have about local vs lookahead searches.


texts: How to find if any variable which is normally distributed in pandas dataframe

texts: Im having age column and how can i plot and say this is normally distributed or not


texts:
I have a dataset of certain user activity per week (e.g. purchasing an item or using a service per week) for the past 52 weeks and for 100K+ users. The matrix is very sparse (85% of the entries are zeros). so, it looks something like this:

          W52  W51 .... W01
User01     0    1        0
User02     0    0        0
...
User99     0    0        2
...

W01 is the most recent week.

My question is whether there are good techniques to cluster the users based on their corresponding time series.


texts:
I hope this doesn't come off as a silly question, but I am looking at SVMs and in principle I understand how they work. The idea is to maximize the margin between different classes of point (within any dimension) as much as possible.

So to understand the internal workings of the SVM classification algorithm, I decided to study the cost function, or the Hinge Loss,  first and get an understanding of it...

$$L=\frac{1}{N} \sum_{i} \sum_{j \neq y_{i}}\left[\max \left(0, f\left(x_{i} ; W\right)_{j}-f\left(x_{i} ; W\right)_{y_{i}}+\Delta\right)\right]+\lambda \sum_{k} \sum_{l} W_{k, l}^{2}$$

Interpreting what the equation means is not so bad. For all classes we find the difference between all classes and the class we want to interpret (with a difference of delta) and sum all of them that are greater than 0. By looking at this it seems we are trying to figure out how off a classification is for every point.

However, I do not actually understand why the Hinge Loss works, and why it is useful for SVMs. I cannot find any proofs of the Hinge Loss, nor any visual representation as to why it works.

I understand that we are trying to minimize the Hinge Loss during training, but I have no idea how the equation of the Hinge Loss came to be, and why it works. I am looking for some kind of mathematical proof of the Hinge Loss in relation to SVMs.


texts: Unsupervised Defect Detection on Any Images

texts:
Everyone. I want to ask that Is there any way to do unsupervised defect detection(without labeled data) or without knowing the possible defects that will arrive in future. Means that I train my model only on good samples and when any defective image comes it highlights the defective part only. Another issue is that the images are very few, maybe 10 or 20.
So is there any way to solve this problem with deep learning or any other approaches.
Thank you very much


texts: Learning words embedding for bigrams and unigrams in a corpus

texts:
I am working on a topic modeling for tweets projects. I have generated my topics using both unigrams and bigrams. Topics are defined with a mixture of both bigrams and unigrams.
Now I am planning to evaluate the coherence of my generated topics with the coherence measure I am trying to use is the sum of pair-wise similarity of terms that define the topics.
To compute that measure as you may see I need embeddings for my terms, and that is where the issue came from.
I have trained Fasttext on my corpus to learn the word embedding but it only gives me the embedding for unigrams and not bigrams.
My first question is how to train the embedding to learn both bigrams and unigrams embedding?
I found some research where they include n-grams to improve word embedding but I can't see any approach where they output embedding for bi_grams.
How should I process my text so that I can learn the embdeedings?
I found the Phrase and Phraser classes from Gensim but they are not returning all possibles embedding and I can't seem to understand how they are doing it.
Which another approach should I use?

Can I split each sentence in my sentence to included both unigrams and bigrams and learn the embedding from them?

Ex: These are sentences about words embeddings  === should be split into :
these, these_are, are,  are_sentence, sentence,  sentence_about, about, about_words, words words_enmbeddigns and learn the embedding from that?

Each sentence can be split into one for unigrams and another one for bigrams and combine both and train the model on the combination? to

Ex: These are sentences about words embeddings should be split into :

these, are, sentences, about, words, embeddings and

these_are, are_sentences, sentences_about, about_words ,words_embeddings

Or simpler why not considering the bigrams embedding as the average of the unigrams embeddings?

All ideas on how to tackle this are welcomed...
Thanks


texts: How can I transform (pre-process) pure count data for PCA analysis?

texts: If 12 covariates of data are all count data (looks like a Poisson dist with the highest peak at 0), what are reasonable pre-processing methods that might make PCA more effective?


texts:
My dataframe looks like this :

there are some rows ( example : 297) where the "Price" column has two values ( Plugs and Quarts) , I have filled the Nans with the previous row since it belongs to the same Latin Name. However I was thinking of splitting the Price Column further into two columns with Names "Quarts" and "Plugs" and fill the amount, 0 if there is no Plugs found and same with Quarts.

Example :

Plugs | Quarts
0 | 2
2 | 3
4 | 0

Can someone help me with this?


texts:
My data file is of 4 GB (json), I need to apply everything on this dataset (from applying clustering/ML algorithm to wrangling it) like the way people do with pandas/scikit. But it is difficult to work with them on my local machine. I tried using dask and blaze but could not find them efficient for my case because of their limited functionalities. I don't wanna use hadoop because it is not meant for such small datasets.

I wanted to know approaches for working with not so large datasets with python and applying ML algos on them. Just like the way you work on small datasets using pandas (if any exists).


texts: non-binary nominal variable in linear regression

texts:
For example, I have a variable Color with 3 different values. How can I transform it, so that I can use this variable in linear model?
I've read about dummy variables, but as these three will depend on each other (if the value is "red", then it's not "black"), can I use dummy variables? Shouldn't independent variables be not dependent on each other?


texts: Is this a good practice of feature engineering?

texts:
I have a practical question about feature engineering... say I want to predict house prices by using logistic regression and used a bunch of features including zip code. Then by checking the feature importance, I realize zip is a pretty good feature, so I decided to add some more features based on zip - for example, I go to census bureau and get the average income, population, number of schools, and number of hospitals of each zip. With these four new features, I find the model performances better now. So I add even more zip-related features... And this cycle goes on and on. Eventually the model will be dominated by these zip-related features, right?

My questions:

Does it make sense doing these in the first place?
If yes, how do I know when is a good time to stop this cycle?
If not, why not?


texts: Replace part column value with value from another column of same dataframe

texts:
I have a dataframe with two columns:
Name          DATE
Name1         20200126
Name2         20200127
Name#DATE#    20200210

I need to replace all the #DATE# with the data from the DATE column, and get something like this:
Name
Name1
Name2
Name20200210

How can I achieve this? I've tried things like this, without any good result..:
df_merged_tables["Name"].str.replace("#DATE#",merged_tables["DATE"])

Thanks!


texts: Can all statistical algorithms be parallelized using a Map Reduce framework

texts:
Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework?  Or are there restrictions?

I guess there may be some algorithms that is not possible to parallelize?


texts: Alternative to Flesch for a readability score algorithm

texts:
Someone I know has been using the Flesch-Kincaide readability score algorithm to try to gauge how well students write as a predictor of their academic success. They've been mostly unsuccessful.

While taking a UCSD Coursera course I recently learned about this algorithm and I believe that their lack of success is just a basic misunderstanding of how the algorithm works. A low score does not imply poor writing -- anything written for college level reading ability, such as U.S. law and classic literature, receives a low readability score. High readability scores imply more simplistic writing, but not that the writing is more or less correct. So it's not really measuring much in terms of proficiency.

All that is to ask this -- is there some other algorithm or metric to gauge writing quality, which would be more useful? Something that perhaps measures proper grammar and/or punctuation?

I have been and will continue to search for this myself, but if anyone has any knowledge they can share it would be greatly appreciated.


texts: Simple way to visualise word2vec vector space

texts:
I want to visualize my word2vec vector space (with zoom in and zoom out). I found a really interesting GitHub project named word2vec explorer. However, it does not seem to work in windows and mac.

Hence, I am interested in knowing if there are any other off-the-shelf project I can use for this?


texts: TF object detection - The total number of detected objects is not increasing

texts:
I'm building a model to recognize fishes in the aquarium (150 different fishes).
I'm using a faster_rcnn_inception_v2_coco_2018_01_28 model for transfer learning from TF object detection API. I have labelled around (80 fishes) and (55) get detected only.
when I labelled more (30 fishes) and added it to the model and retrained it again, the number of detected fishes reach to (66 fishes), but some of the fishes which detected in the previous model are not detected after adding more classes and retrain the model.
I don't know why that happened. and I have some concerns about the model.

While labelling the images, each image contains multiple fishes types, and I labelled the clear ones only.

Should I label the images that taken when I applied zoom in to the camera to focus on some fishes, in this case, the fish will appear in a bigger size than the real size.

Some of the fishes are big and some of them are small and some of them are too small, is it applicable to detect all of them in one model?

Are there any suggestions to deal with similar fishes?

Is there another benefit for training using GPUs other than reducing time-consuming for training?

Thanks in advance for helping.
Update:
I have added more class (20 classes more) and the count of detected fishes reach to ( 87) but not more, even I put it for more time in training.


texts: What is the difference between Freidman mse and mse?

texts: The original question was posted on StackOverflow. However, taking into account the recommendations of Sergey Bushmanov, I'm providing an answer through this medium.


texts:
I want to place a Call Marker on a plot. Call should be "buy" whenever the smaller moving average (21) crosses over longer moving average (34) AND the Call should be "sell" whenever smaller moving average crosses under longer moving average.

I have a column average price. I have calculated the rolling mean for 21 and 34 days using the rolling() function and plotted the line plot of all three columns: average price, sma_21 and sma_34 using matplotlib. I want to place a marker on the plot.

If the smaller moving average (21) crosses over longer moving average (34) I need to place this maker "^" AND if smaller moving average crosses under longer moving average I need to place this maker "v".

averageprice = [
    2352.6, 2410.26, 2443.31, 2525.78, 2506.58, 2530.69, 2530.49, 2545.01,
    2605.4, 2593, 2577.65, 2554.74, 2549.69, 2552.85, 2568.84, 2577.2,
    2693.18, 2624.95, 2543.44, 2513.28, 2487.48, 2464.89, 2469.41, 2427.94,
    2402.96, 2430.5, 2427.14, 2412.24, 2403.02, 2388.78, 2357.33, 2345.89,
    2342.52, 2361.01, 2368.46, 2366.9, 2354.42, 2348.75, 2343.49, 2426.54,
    2478.13, 2453.34, 2449.5, 2396.18, 2402.63
]
avg_p = df['Average Price']
sma21 = avg_p.rolling(window = 21).mean()
sma34 = avg_p.rolling(window = 34).mean()
avg_p.plot()
x = df.index
f = sma21
g = sma34
plt.plot(x, f)
plt.plot(x, g)
idx = np.argwhere(np.diff(np.sign(f - g))).flatten()
plt.plot(x[idx], f[idx], '^')
plt.show()

I used this code to get the intersection point and place the marker

idx = np.argwhere(np.diff(np.sign(f - g))).flatten()

and got a plot like this:

I need to get markers that look something like this:


texts: I've trained a CNN model based on landscape and pool images. The main purpose is to make model classify if the image contains a pool or not. The accuracy on the test set was about 94% but when I used my own image to test, sometimes model did wrong predictions then I implemented GradCam and realized that the model looks only at the edges of the pool when I would classify the pool by the water. How can I fix my model so it could make better predictions and look not only at the edges?


texts: Deep Reinforcent Learning Model, experience replay

texts:
So I am working on a deep RL model in OpenAI Gym. I have everything else working except for my function that allows my agent experience replay.

here is the code:

import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from collections import deque
from keras.optimizers import Adam
import numpy as np
import gym
import tensorflow as tf
import math
import random

env = gym.make('MountainCar-v0')

#hyperparams
gamma = .99 #discount factor for reward
alpha = .01 #learning rate
beta  = .01 #decay
batch_size = 32
memory = deque(maxlen=2000)
epsilon = 1.0

#building model
model = Sequential()
model.add(Dense(12, activation='sigmoid', input_dim=2))
model.add(Dense(12, activation='tanh'))
model.add(Dense(2, activation='linear'))
model.compile(loss='mse', optimizer=Adam(lr=alpha, decay=beta))

def preprocess_state(state):
    return np.reshape(state, [1, 2])

def experience(state, action, reward, next_state, done):
    # memory for replay
    memory.append((state, action, reward, next_state, done))

class agent():

def choose_action(self, state):
    rand_number = np.random.randint(2)
# choosing random action 50%, acting greedily with respect to value func/policy (naive e-greedy)
    return env.action_space.sample() if (np.random.random() <= epsilon) else np.argmax(model.predict(state))

def replay(self):
    x_batch, y_batch = [], []
    minibatch = random.sample(memory, batch_size)
    for state, action, reward, next_state, done in minibatch:
        y_target = model.predict(state)
        y_target[0][action] = reward if done else reward + gamma * np.max(model.predict(next_state)[0])
        x_batch.append(state[0])
        y_batch.append(y_target[0])

agent = agent()

def run():

for episodes in range(1000):
    state = preprocess_state(env.reset())
    tot_reward = 0
    done = False
    while not done:
        env.render()
        action = agent.choose_action(state)
        next_state, reward, done, t = env.step(action)
        next_state = preprocess_state(next_state)
        #record results

        experience(state,action,reward,next_state,done)

        tot_reward += reward

        #replay for agent
        if batch_size < len(memory):
            agent.replay()

        print("episode: {} | score: {} |".format(
       episodes,tot_reward))

        if done:
            print "You did it"
            break

run()

The issue is with this line:

  y_target[0][action] = reward if done else reward + gamma * np.max(model.predict(next_state)[0])

and the error:


texts: How can we manually change some TF slim weights and use them to re-train, fine-tune, or predict

texts:
I am interested in changing some weights values in certain conv layers of a tf-slim model. Say Mobilenet_v1. Is there any straightforward way to do that while keeping the whole architecture and the data the same as before?

E.g.:
Suppose we have got this numpy array of weights from the last level layer of trained MobileNet_v1:

>>> conv=sess.run('MobilenetV1/Conv2d_13_depthwise/depthwise_weights_2:0')

which gives me the whole array:

...

[ -7.38371849e-01],
[  1.39969051e+00],
[  1.34190738e+00],
[ -5.66634417e-01],
[ -2.53899932e-01],
[  1.18222773e-01],
[  2.57152438e-01],
[  2.43642002e-01],
[ -5.11337161e-01],
[ -5.80285192e-01],
[  3.62841874e-01],
[ -4.72957313e-01],
[  7.34595060e-01],
[ -7.35578299e-01],
[  6.73987687e-01]]]], dtype=float32)

for instance, I would like to search for all values within a specific range:

>>> numpy.count_nonzero((conv>=XX) & (conv<=YY))

and then set those values to zero. That means that I overwrite the new variables values, so then I use the new pre-trained model for further training, or, fine-tuning, or inference. How can I do that?


texts:
I tried creating a simple linear regression model on just 30 rows of data. I got this error while trying to fit the model:

dataset = pd.read_csv('Salary_Data.csv')
x=dataset.iloc[:, :-1]
y=dataset.iloc[:, 1]

from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 1/3, random_state = 0)
regressor = LinearRegression()
regressor.fit(x_train, y_test)

Here is the error message I got:

ValueError                                Traceback (most recent call last)
<ipython-input-53-8dc82dc6fe8b> in <module>()
----> 1 regressor.fit(x_train, y_test)

~\Anaconda3\lib\site-packages\sklearn\linear_model\base.py in fit(self, X, y, sample_weight)
    480         n_jobs_ = self.n_jobs
    481         X, y = check_X_y(X, y, accept_sparse=['csr', 'csc', 'coo'],
--> 482                          y_numeric=True, multi_output=True)
    483
    484         if sample_weight is not None and np.atleast_1d(sample_weight).ndim > 1:

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    581         y = y.astype(np.float64)
    582
--> 583     check_consistent_length(X, y)
    584
    585     return X, y

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_consistent_length(*arrays)
    202     if len(uniques) > 1:
    203         raise ValueError("Found input variables with inconsistent numbers of"
--> 204                          " samples: %r" % [int(l) for l in lengths])
    205
    206

ValueError: Found input variables with inconsistent numbers of samples: [20, 10]


texts:
I'm currently training a CNN to do a binary classification.
I'm getting fairly good results, but unfortunately the training is very unstable.
Just by changing the seed the relative error changes by 20-30%.

What can be the cause of this and how can I prevent it?

Other info:

The amount of data is rather small.
I'm starting from imagenet snapshot.


texts: I want to know feature names that a LogisticRegression() Model has used along with their corresponding weights in scikit-learn. I can access to weights using coef_, but i did not know how can pair them with their corresponding weights.


texts: How to specify scale_pos_weight value at runtime in Hyperopt?

texts:
I want to use LighgbmClassifier for a binary Classification. for Hyper Parameter tuning I want to use Hyperopt. The Dataset is imbalanced. Using Sklearns class_weight.compute_class_weight as shown below
        clas_wts_arr = class_weight.compute_class_weight('balanced',np.unique(y_trn),y_trn)
        self.scale_pos_wt = clas_wts_arr[0] / clas_wts_arr[1]

The following is the space parameter that I am passing to the objective function
        space = {'objective' : hp.choice('objective', objective_list),
                 'boosting' : hp.choice('boosting', boosting_list),
                 'metric' : hp.choice('metric', metric_list),
                 "max_depth": hp.quniform("max_depth", 1, 15,2),
                 'min_data_in_leaf': hp.quniform('min_data_in_leaf', 1, 256, 1),
                 'num_leaves': hp.quniform('num_leaves', 7, 150, 1),
                 'feature_fraction' : hp.quniform('feature_fraction', 0.5, 1, 0.01),
                 'min_gain_to_split' : hp.quniform('min_gain_to_split', 0.1, 5, 0.01),
                 'lambda_l1' : hp.uniform('lambda_l1', 0, 5),
                 'lambda_l2' : hp.uniform('lambda_l2', 0, 5),
                 'feature_pre_filter': False}

My question will the following set scale_pos_weight properly in the space dictionary
        #set scale pos weight explicitly
        space['scale_pos_weight'] = self.scale_pos_wt

If that is wrong then what would be the correct way to set scale_pos_weight at runtime in the space dictionary that is passed to the Objective fn that is in turn passed to the fmin of Hyperopt.
Thanks for your help and answers.


texts:
I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations.

When I wanna' use the standard deviation as an outlier detection, I struggle with this definition as there will always be outlier. But I probably have a misunderstanding somewhere, somehow..

In other words: 3 standard deviations contain always about 99 % of the data, hence there will always be $ outlier > 3 std $, no matter if they are outlier.

What is the issue I don't see?


texts: Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson)

texts:
I'd like to ask what is the main reason why we find the roots in logistic regression (i.e. why we use Newton Raphson method on logistic regression ). I understand the basics of Newton Raphson method, but I just can't understand what is the importance of finding the roots or using second derivatives.

P.S. I know the idea of Newton Raphson, I am wondering why do we need to use this method and find the zero's or roots of function? What does it want to tell us for logistic regression for instance?


texts: Quick question on basic Basic concept of experience replay

texts:
Due to my admitted newbie's understanding on the field, I'm about to ask a dummy question.
While sampling batches, for example experience replay buffer which contains number of samples, after getting n (size of a batch) of loss values through forward propagation, what way will the weights updates follow:

Update all weights based on all losses separately, like iterating through that n losses to update
Update all weights based on the mean (or maybe standard deviation) of all losses


texts: XGBoost: how to adjust the probabilities of a binary classifier to match training data?

texts:
Training and testing data have around 1% positives, but the model predicts only around 0.1% as positives.

The model is an xgboost classifier.

I’ve tried calibration but it didn’t improve much. I also don’t want to pick thresholds since the final goal is to output probabilities.

What I want is for the model to have a number of classified positives similar to the number of positives in the actual data.


texts: 80-20 or 80-10-10 for training machine learning models?

texts:
I have a very basic question.

1) When is it recommended to hold part of the data for validation and when is it unnecessary? For example, when can we say it is better to have 80% training, 10% validating and 10% testing split and when can we say it is enough to have a simple 80% training and 20% testing split?

2) Also, does using K-Cross Validation go with the simple split (training-testing)?


texts: How to apply only 3 layers of a network to a data

texts:
I want to use the 3rd layer's output of the VGG16 network. The error is like below:

UserWarning: Model inputs must come from `keras.layers.Input` (thus holding past layer metadata), they cannot be the output of a previous non-Input layer. Here, a tensor specified as input to your model was not an Input tensor, it was generated by layer input_1.
Note that input tensors are instantiated via `tensor = keras.layers.Input(shape)`.
The tensor that caused the issue was: input_1:0
  str(x.name))
Traceback (most recent call last):

The code I'm using is below:

from keras import Model
from keras import applications

vgg_model = applications.VGG16(include_top=True, weights='imagenet')
vgg_model.summary()
layers = [l for l in vgg_model.layers]

first_layers = layers[0:3]

result_model = Model(input=layers[0].input, output=first_layers[2](first_layers[1](first_layers[0](layers[0].input))))
print("success")
result_model.summary()

My eventual goal is to take this output and send it to another process and it will continue from 4th layer.

How can I split the neural network into two like this?


texts:
Let's say that I have a set of users in my database, that have GUIDs as their IDs. I use xxhash to generate fixed-length hashes for each value, so that I can then proceed to "bucketizing" them and being able to do random sampling with the help of the modulo function.

That said, if I have a hash such as 367b50760441849e, I want to be able to use hash % 20 == 0 to randomly pick 5% of the population (hence, 20 "buckets"). This is the approach that is used in Kusto hash() with a modulo argument.

With this in mind, what is the approach that should be used to calculate an integer value from the hash, so that I can calculate the modulo?


texts:
I'm in the beginning to learn and understand recurrent neural networks. As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time. Not sure if it's a right way to think about it, but so far it's my first impression.
Some things are unclear though.

As far as I understood the final output of each timestep is supposed to predict the input of the next timestep. Is this true? What if I would just like to show the network two images of for example a horse, and depending on them, predict what distance did it go, and in which direction? Is this possible?

In the illustration above there's $A_0$. From where? I would assume at least two timesteps are needed to make a prediction, so in my understanding an $x_0$ is missing from the left side of the diagram. Am I right?

I've been reading through an article which says "Lets train a 2-layer LSTM with 512 hidden nodes". Does it mean two layer of activations, and 512 timesteps?


texts: How to access the data of the column on which some groupby operation has been carried out?

texts:
Suppose there is a pandas dataframe which has one column consisting of names of something, and there are multiple entries respective to each entry in the first column. To count the number of entries in the second column for each entry in the first, applying count method grouped by entries on the first is the classical solution. And it works as intended.
Here's something concrete-

There are multiple entries for each of the county e.g. California. I found the number of entries for each county by executing this code-

Now I want to have something like this-
|               county|             number_of_venues|
|---------------------|-----------------------------|
|Alameda              |                  some number|
|San Diego            |                  some number|
|Yolo                 |                  some number|

and so on.
I tried achieving this by-
df = pd.DataFrame(california_venues.groupby('county').count())

Which gives me this weird dataframe-

Where county is not a key. It just sits on top of it. Trying to access the entries of the column through county raises KeyError.
My plan was to extract the two relevant columns once the desired dataframe had been formed.And finally rename the column name(s).
What are my options to get the form described by me in the markdown table?


texts: How do you use KS-test in a data science report?

texts:
I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result  , but when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model.
I mean, whatever what distribution/model you use to fit the sample, you cannot expect to have a perfect result, especially working with huge amount of data. So what does KS-test do in a data science report? Does it means only if we got high p-value in KS-test then the model is correct?


texts: Why are discriminative models denoted as $P\left(Y|X\right)$?

texts:
In general, we can speak we know deterministic and probabilistic models. Discriminative ones are deterministic, while generative one is probabilistic.

But I have been reading about the difference between discriminative and generative models where the difference was expressed as:

$P\left(Y|X\right)$ for D. models

and $P\left(X|Y\right) \cdot P\left(Y\right)$ for G. models

My question is: Why do we denote $P\left(Y|X\right)$ for Discriminative models, since these models are completely deterministic, build on top of composite functions with no stochastic processes? Isn't this confusing?


texts: How to predict out-of-sample observations with depmixS4 package in R?

texts:
I have a series of univariate data and I want to fit a Hidden Markov Model on it using the depmixS4 package on R. My final goal is to predict the next k observations (let's say k = 10) for the data series. I am not really interested in predicting the new state (that is important, but not my final goal), but I want to predict the next values for the data series.
It is a snippet of code:
# My series
data = rnorm(10000)
df_1_col = data.frame(data)
colnames(df_1_col) <- c('obs')

mod <- depmix(obs ~ 1, data = draws, nstates = n_state)
fit.mod <- fit(mod)

At this point I don't know how to predict the next out-of-samples values. I would like something similar to the forecast function.
I try using the following code:
state_ests <- posterior(fit.mod)
pred_resp <- matrix(0, ncol = n_state, nrow = 10)

for(i in 1:n_state) {
  pred_resp[,i] <- predict(fit.mod@response[[i]][[1]])
}

Using this code the predict function generates a number of predicted values that is equal to the number of observations into data, so it is not right.
How can I do this quite basic operations? I am new to HMM, but I already tried to look into many resources and I did not find any information. Thanks :)


texts: Interpreting in higher dimensions with maximal spacing

texts:
I have data that I would like to interpret in higher dimension (sort of the opposite of an auto-encoder). What I intent to achieve is, an m dimension data being represented in n dimensions (n>m) with the constraint that the representation in the n dimensions is maximally separated.
For clarity, I will assume the binary field. My data has m=20 dimensions. I would like to represent this data in a n=50 dimensional space. There are 2^m number of data combinations that are possible and 2^n number of points in the larger dimension. My goal is to select 2^m points from 2^n points, which are as far away from each other as possible.
I am versed with neural networks. What I want to know is how do I tackle this problem from a neural network perspective? Thanks!


texts: Why Matlab neural network classification returns decimal values on prediction dataset?

texts:
I have an input dataset (matrix 25x2275) which is normalized to values between 0 and 1. I also have a binary formatted output matrix (3x2275) like 0 0 0 0 0 0 0 0 1, 1 0 0 1 1 1 0 0 1 ...

I imported both files in matlab nntool and it automatically created a network with 25 input and 3 output nodes as I wanted.

After I trained this network using feed-forward backProp, I tested the model in its training data and each output nodes returns a decimal value like (0.9999 0.978 1 0 0.99 0.59368 0.38359 0.31435 1.0604).

Why it doesn't return discrete values like 1 0 0 1 1 1 0 0 1? Is there any thing that I must set in nntool to get such values?


texts: How to properly train batch-normalization networks with gradient accumulation?

texts:
Using gradient accumulation efficiently replicate training with larger batch sizes for networks that are independent on the batch size. However, networks with BN layers rely on a batch size -- running statistics are calculated on mini-batches instead of full-batch and it could harm the training process.
My question is: could we modify gradient accumulation training procedure to allow BN layers (as well as the other layers) to be updated as if we use full-batch? Training step described below seems to be a legit approach.
Note: grad_accum here is a number of mini-batches to accumulate gradients before doing optimization step.
Training step

Freeze all the network parameters except for BN layers'.
Concatenate grad_accum mini-batches into full-batch, calculate loss and accumulate gradients. Here gradients are calculated only for BN layers and BN layers running statistics are updated for full-batch.
Unfreeze all the network, freeze BN layers. Set BN layers to evaluation mode not to update running stats.
Iterate over grad_accum mini-batches, calculate and scale loss and accumulate gradients as we do in a usual gradient accumulation procedure. Here gradients are calculated for all the layers except BN and full-batch BN running stats calculated at step 2 are used.
Perform optimization step.

Doing so, we efficiently update the layers except BN layers as usual for gradient accumulation, calculating running stats and update BN layers on a full-batch. Memory overhead is minor so as it is only BN parameters are updated in a full-batch mode.
What am I missing here?


texts: Newbie question on restricted boltzmann machine

texts: I’m quite a newbie to RBMs so I’m trying to understand how do you feed real valued data to it given that all the visible and hidden units are binary?