texts: How to extract crucial features to create an image

texts:
Imagine, you have a dataset containing pictures of (example only, just to explain the task) cats and dogs. The data set is labeled, so we can train using supervised learning algorithms.

My goal is to make a cat from a dog. How to do it? For now, I have a couple of ideas which I can share:

Use a convolutional autoencoder and train on cats, then give a
picture of a dog and see the result (I suppose, it will show the
most "similar" cat, so the goal is reached)
Use an algorithm like GAN to transfer "style". I have no idea
whether it is possible or not, but looks like a working idea

Which approaches could you recommend to try out?


texts: Maximum Entropy modelling - likelihood equation

texts:
I am trying to understand maximum entropy modelling and I came across log likelihood equation of the empirical distribution, which I did not quite understand, which also eventually turns out to be equal to the dual function we get when trying to maximise the entropy with constraints using Lagrange's multipliers

$$ L_\widetilde{p}(p) \equiv log \prod\limits_{x,y}p(y|x)^{\widetilde{p}(x,y)} = \sum\limits_{x,y}\widetilde{p}(x,y)\log{p(y|x)}$$

Where,

$y$ is the outcome produced by a random process
$x$ is the feature influencing the outcome $y$
$L_\widetilde{p}(p)$ is the log likelihood
$\widetilde{p}$ is the empirical distribution of training data
$p(y|x)$ is the model
$\widetilde{p}(x,y) \equiv \frac{1}{N} \times \text{number of times that } (x,y) \text{ occurs in the sample} $

Can someone please explain how $p(y|x)$ is raised to the power of $\widetilde{p}(x,y)$ in the log likelihood equation mentioned above? Instead, shouldn't $p(x|y)$ be raised to the power of number of times that $(x,y)$ occurs in the sample.

I went through this reference tutorial on max entropy.


texts:
I am new to data science, I have clustered some data using Scipy agglomerative clustering. how can I fit new data into the learned clusters?

dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))
z = hac.linkage(dm, method='average')
cluster = hac.fcluster(z, t=100, criterion='distance')
leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='distance')

I would like to cluster new data into the same clusters, How can I do it?


texts: SelectKBest for regression `f_regression` behaves weird when changing the random_state parameter when splitting

texts:
I am working on a regression project using the Audi dataset from Kaggle.
I have looked at other notebooks and i saw that people use SelectKbest. I tried using the same thing, but when I was splitting my data to train-test I used random_state = 42 and when I tried using the SelectKBest I had a lot of warnings:
minmax_scaler = MinMaxScaler()

#cars_df_with_dummies -> this is a dataframe I created with pd.get_dummies(cars_df)
cars_scaled = minmax_scaler.fit_transform(cars_df_with_dummies)
cars_scaled = pd.DataFrame(cars_scaled, columns = cars_df_with_dummies.columns)

scaled_price_label = cars_scaled['price']
scaled_cars_without_price = cars_scaled.drop(['price'],axis=1)

X_train,X_test,y_train,y_test = train_test_split(scaled_cars_without_price,scaled_price_label,test_size=0.2,random_state=42)

The code to select the best features (was taken from the top rated notebook):
column_names = cars_df_with_dummies.drop(columns = ['price']).columns

no_of_features = []
r_squared_train = []
r_squared_test = []

for k in range(3, 35, 2): # From 3 to 35 variables (every single one)
    selector = SelectKBest(f_regression, k = k)
    X_train_transformed = selector.fit_transform(X_train, y_train)
    X_test_transformed = selector.transform(X_test)
    regressor = LinearRegression()
    regressor.fit(X_train_transformed, y_train)
    no_of_features.append(k)
    r_squared_train.append(regressor.score(X_train_transformed, y_train))
    r_squared_test.append(regressor.score(X_test_transformed, y_test))

sns.lineplot(x = no_of_features, y = r_squared_train, legend = 'full')
sns.lineplot(x = no_of_features, y = r_squared_test, legend = 'full')
plt.show()

/anaconda3/lib/python3.8/site-packages/sklearn/feature_selection/_univariate_selection.py:302: RuntimeWarning: invalid value encountered in true_divide
  corr /= X_norms
/anaconda3/lib/python3.8/site-packages/scipy/stats/_distn_infrastructure.py:1932: RuntimeWarning: invalid value encountered in less_equal
  cond2 = cond0 & (x <= _a)

As soon as I changed to random_state=0 in the code above it worked!, but I have no idea why.
I have read about the random_state parameter and people said it doesn't matter which value is assigned to it.
From the Docs:

An integer
Use a new random number generator seeded by the given integer. Using an int will produce the same results across different calls. However, it may be worthwhile checking that your results are stable across a number of different distinct random seeds. Popular integer random seeds are 0 and 42.

why did the random_state parameter affect the SelectKbest ?


texts: how can I replicate working of Multi Label Binarizer from sklearn package in R?

texts:
I want to achieve same working of MultiLabelBinarizer from sklearn.preprocessing package in R. I have list of labels for each example (for Predicated and Actual) like below.

[[1]][1] "a"      "b"      "d"    "f"
[[2]][1] "g"      "b"      "f"    "d"
....

what I want is to get this in -

[[1 1 0 1 0 1 ...]
[0 1 0 1 0 1 1 ..]]...

So that I can get it in the form for multilabel_confusion_matrix.


texts: how to create multiple subplots( scatterplot) in for loop

texts:
I want to create scatterplot of target variable with all the other variables in dataframe. Just like we create pairplot but as pairplot with large number of variables will take lot of time and computing memory, it is required to draw scatter plot containing all the variables with target variable.
So tried implementing it in for loop, but only last one got rendered properly.
count=1
for i in df.columns:
    plt.subplot(n,1,count)
    plt.figure(figsize=[5,5])
    sns.scatterplot(df["cnt"],df[i])
    count+=1

output: only last plot is rendered properly


texts: Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

texts:
According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments:

tgt – the sequence to the decoder (required).
memory – the sequence from the last layer of the encoder (required).
tgt_mask – the mask for the tgt sequence (optional).
memory_mask – the mask for the memory sequence (optional).
tgt_key_padding_mask – the mask for the tgt keys per batch (optional).
memory_key_padding_mask – the mask for the memory keys per batch (optional).

Unfortunately, Pytorch's official documentation on the function isn't exactly very thorough at this point (April 2021), in terms of the expected dimensions of each tensor and when it does or doesn't make sense to use each of the optional arguments.
For example, in previous conversations it was explained to me that tgt_mask is usually a square matrix used for self attention masking to prevent future tokens from leaking into the prediction of past tokens. Similarly, tgt_key_padding_mask is used for masking padding tokens (which happens when you pad a batch of sequences of different lengths so that they can fit into a single tensor). In light of this, it makes total sense to use tgt_mask in the decoder, but I wouldn't be so sure about tgt_key_padding_mask. What would be the point of masking target padding tokens? Isn't it enough to simply ignore the predictions associated to padding tokens during training (say, you could do something like nn.CrossEntropyLoss(ignore_index=PADDING_INDEX) and that's it)?
More generally, and considering that the current documentation is not as thorough as one would like it to be, I would like to know what the purpose is of each argument of nn.TransformerDecoder's forward function, when it makes sense to use each of the optional arguments, and if there are nuances in the usage one should keep in mind when switching between training and inference modes.


texts: Time series prediction of discontinuous data

texts:
In the context of time series prediction, I have read that time series is a series of data that taken at successive equally spaced points in time (which means its in order). What if I have a discontinuous time series data, for example:

If I have data that represnt a room temperature within the working hours, specifically from 7:00 am - 3:00 pm. And this data is repeated on the day of the week. So we have something similar to this:

Saturday: [ Time series data from 7:00 am - 3:00 pm]

Sunday: [ Time series data from 7:00 am - 3:00 pm]

Monday: [ Time series data from 7:00 am - 12:00 pm]  -- Only up to 12 pm

The first question:

Is that considered as time series data
Now if I want to predict the room temperature on Monday @ 1:00 pm. How can I do this using given this type of datasets?


texts: Regression model performance with noisy dependent variable

texts:
I'm doing a support vector regression with the dependent variable representing measurements from an uncalibrated sensor (measurement error between 2% and 20%) and I want to study the effect of this error on the model performance.
What's the best method to inject error on the dependent variable samples ? is it sufficient to do this by adding white noise?
Any help or suggestion is appreciated


texts: Variable Importance for NN's with Olden's Method

texts:
Background:
I've built a neural network for predicting categorical outcomes and wanted to test variable importance with the variable importance method proposed by Olden et al. 2004.
Problem:
However, I've noticed that the more 'important' variables, i.e., variables with greater absolute value on the output from Olden's algorithm, tend to have far fewer samples in the training set than the 'less important' variables. For example, a variable $x$ with variable importance value 5000 might only have 5 observations in the training set that have $x$ encoded as true (I've used one-hot encoding to encode observations here) whilst variable $y$ with importance value 72 might have 750 observations in the training set that have $y$ encoded as true.
Questions

Has anybody come across this issue?
How would you resolve this?
Is it reasonable to say that variable $x$ has more variable importance than variable $y$, even though it has a much smaller sample size?


texts: Predict_proba for Binary classifier in Tensorflow

texts:
I'm working in binary classifier problem, where I have used Tensorflow low level API's.
Last layer wrapped with Sigmoidal Function and just returning a  single value.

For my prediction I just set a standard threshold value of 0.5 and hence if it's >0.5, I'm predicting as 1, otherwise 0.

I'm also looking for a confidence level of each prediction. Is it possible to get the same with this architecture, or does it need some modification?


texts: Numerical example of Confusion in understanding learning rate in xgboost

texts:
I fail to understand as to how learning rate is used in XGBoost?
Can anyone explain using a numerical example?


texts: Hierarchical Clustering customized Linkage function

texts:
In my clustering project, I need to customize the linkage function, so that after each cluster merging I can update the inter-cluster distance in my own way.

Currently I'm using scikit-learn AgglomerativeClustering, which seems not having this customizable feature. After a quick glance in scipy, no luck there either. Does anyone know any python hierarchical clustering toolkit that has customizable linkage?


texts:
Which of the below set of steps options is the correct one when creating a predictive model?

Option 1:

First eliminate the most obviously bad predictors, and preprocess the remaining if needed, then train various models with cross-validation, pick the few best ones, identify the top predictors each one has used, then retrain those models with those predictors only and evaluate accuracy again with cross-validation, then pick the best one and train it on the full training set using its key predictors and then use it to predict the test set.

Option 2:

First eliminate the most obviously bad predictors, then preprocess the remaining if needed, then use a feature selection technique like recursive feature selection (eg. RFE with rf ) with cross-validation for example to identify the ideal number of key predictors and what these predictors are, then train different model types with cross-validation and see which one gives the best accuracy with those top predictors identified earlier. Then train the best one of those models again with those predictors on the full training set and then use it to predict the test set.


texts: Regression and Machine learning models for panel data

texts:
I have a panel data which consists of the sales of different companies across different years. Can someone tell me what kinds of regressiona and ML methods we use for forecasting the sales?

Thanks


texts:
I've been trying to find an answer to this question for a long time.

What are the relationships/differences between Bias, Variance and
Residuals?

I think I do understand Bias, Variance and Residuals as separate concepts. Correct me if I'm wrong -
Bias is the difference between the average expected results from different runs of the model and the true values from data.
Variance is the variability in the expected results (predictions) of a given data point between different runs of the model.
Residual is the difference between the expected results from a model and the true values from data. y - y^ Residual seems somewhat similar to Bias. But is that my misunderstanding?
Please explain to me in plain words and also simple equations the relationships or differences between these three.


texts: neural network probability output and loss function (example: dice loss)

texts:
A commonly loss function used for semantic segmentation is the dice loss function. (see the image below. It resume how I understand it)

Using it with a neural network, the output layer can yield label with a softmax or probability with a sigmoid.
But how the dice loss works with a probility output ?

The numerator multiply each label (1 or 0) of the predicted and ground truth. Both pixels need to be set to 1 to be in the green area. What is the result with a probability like 0.7 ? Does the numerator result in a floating number (ie with ground truth = [1, 1] and predicted = [0.7, 0], the "green" area of the numerator would be 0.7) ?


texts: Fitting data with Fourier Series coefficients

texts:
Suppose I have some data, given by $\boldsymbol{x}$ and $\boldsymbol{y}$ pairs, for instance$\{(\boldsymbol{x_1}, \boldsymbol{y_1}), (\boldsymbol{x_2},\boldsymbol{y_2}),...,(\boldsymbol{x_n},\boldsymbol{y_n})\}$ for $n\in\mathbb{N}$.

I assume that the data comes from some function $f(\boldsymbol{x})$ with some noise, I don't know the function, so we don't know whether it is periodic (very likely, it is NOT periodic).

This means $\boldsymbol{y_i} = f(\boldsymbol{x_i}) + \boldsymbol{\epsilon_i}$ where $\boldsymbol{\epsilon_i} \sim \textbf{N}(\textbf{0}, \boldsymbol{\sigma}^2))$.

Now I know that the Fourier Series is defined only for periodic functions. However (here) it is possible to "extend" a non-periodic function defined on an interval, to a periodic function.

The amount of data that I have is finite, so will lie in an interval.

So my question is:

  Does it make sense (is it even possible) to try and fit a Fourier Series to the data, by considering the data (which is multi-dimensional, both x and y are vectors) in a multi-dimensional interval , by finding the coefficients that minimize some loss function? (for instance Mean Squared Error)
  How would someone define this fourier series for a multi-dimensional, non-periodic, noisy function?

(by using only a finite number of fourier terms!)


texts: How to set batch_size, steps_per epoch and validation steps

texts:
I am starting to learn CNNs using Keras. I am using the theano backend.

I don't understand how to set values to:

batch_size,
steps per epoch,
validation_steps.

What should be the value set to batch_size, steps per epoch, and validation steps if I have 240,000 samples in the training set and 80,000 in the test set?


texts:
i want to use KFold from mode_selection instead of cross_validation ut it didn't work for the pobject Kfold

 from sklearn.model_selection import KFold
    import xgboost as xgb
    # Some useful parameters which will come in handy later on
    ntrain = X_train.shape[0]
    ntest = X_test.shape[0]
    SEED = 123 # for reproducibility
    NFOLDS = 10 # set folds for out-of-fold prediction
    kf = KFold(shuffle=False, n_splits= NFOLDS, random_state=SEED)
    def get_oof(clf, x_train, y_train, x_test):
        oof_train = np.zeros((ntrain,))
        oof_test = np.zeros((ntest,))
        oof_test_skf = np.empty((NFOLDS, ntest))

        for i, (train_index, test_index) in enumerate(kf):
            x_tr = x_train[train_index]
            y_tr = y_train[train_index]
            x_te = x_train[test_index]

            clf.train(x_tr, y_tr)

            oof_train[test_index] = clf.predict(x_te)
            oof_test_skf[i, :] = clf.predict(x_test)

        oof_test[:] = oof_test_skf.mean(axis=0)
        return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)
    xgb_oof_train, xgb_oof_test = get_oof(xgb,x_train, y_train, x_test)

i got this error TypeError: 'KFold' object is not iterable


texts: Algorithm suggestion for anomaly detection in multivariate time series data

texts:
I have time series data containing user actions at certain time intervals
eg

Date                 UserId   Directory  operation      Result
01/01/2017 99:00     user1    dir1       created_file   success
01/01/2017 99:00     user3    dir10      deleted_file   permission_denied

unique userIds > 10K
10 distinct operations

and 4 distinct Results

I need to perform anomaly detection on user behavior in real time. Any suggestions on which method I should use?

The anomaly needs to flag whether some user operations are outliers

A very small subset of input data will be labelled. But most of the data will be unlabelled.


texts: What is "data scaling" regarding StandardScaler()?

texts:
I'm trying to figure out the purpose of StandardScaler() in sklearn.

The tutorial I am following says

  "Remember that you also need to perform the scaling again because you had a lot of differences in some of the values for your red and white [wines]"

So I looked up the function in the sklearn docs.

  "Standardize features by removing the mean and scaling to unit variance"

  https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html

What good would removing the mean do?
What is scaling the data? Hard to google that.

# Scale the data with `StandardScaler`
X = StandardScaler().fit_transform(X)


texts:
In RapidMiner I want to create a k-NN model in order to create a classifier. To generate the test sets and the training sets I use the the cross-validation.

If I choose 10 as the number of folds, the cross-validation algorithm will do 10 iterations with different test sets and training sets.

At the end, what training set will be used to create my k-NN model ?


texts:
I wasn't sure where to ask this question so forgive me if the question seems out of place (and please guide me where to ask it !)

I have an archive of 9GAG images and I want to Cluster them based on their content and their similarity... 9GAG images are mostly memes so it's natural that you'd find many of them pretty similar to each other...

I couldn't find any application which does this out of the box (if there is, could you please refer me to it ?) and what I found was a vast number of Papers about Image Clustering but no real application based on them...

I was wondering if there is a Ruby, Python or Java Program which could simply get the directory of the Images and Cluster them into groups (folders) based on their Similarity to each other ?

Thank you very much...


texts: What is the difference between TDNN and CNN?

texts:
I read about time delay neural network (TDNN) and I am not sure I understood it. From what I read it seems that tdnn works just like  one dimensional convolutional neural network (CNN).
What are the differences between the architectures, if they exist?


texts: How to handle different categorical embedding sizes in hold out data set

texts:
I have a pytorch tabular dataset with zip code as a categorical embedding.  I'm getting great results on the test set.  When I go to run my hold out sample through, it errors out because I have more zip codes in the hold out then what the model was trained on.

How do I handle this?  In production, the likelihood of seeing a new zip code is high so I need to learn something I can transfer into production.

Thank you.


texts:
I am wondering how random forests are exactly implemented in Weka. This paper is very specific about RFs in Weka, but the description of its learning process in chapter 2 seems strange to me.
They say:

Bootstrap samples $B_i$ for every tree $t_i$
A random subset of features is selected for each $t_i$
Information gain is used to grow unpruned trees $t_i$

My questions:

Shouldn't step 2 be repeated on all levels of the decision tree? Otherwise each tree will never see some of the features
Whats the default when setting numFeatures=0. I think this is the number of features that is available for each split. Is it the square root of the number of all features?
Is really information gain used for determining the best split attribute?

I am using Weka 3.8.3 - not sure if this matters.

Thanks for all hints :)


texts: Choosing a distance metric and a clustering algorithm for time series

texts:
For every entity I have a corresponding time series which is built by a sliding window (win_size=7d, win_shift=3d, so we have overlapped windows)

With every win-shift, we count how many users are currently using the entity.

So we might see a time series that looks like this:
$$[128, 105, 87, 20, 20, 20, 20, 20, 19, ...]$$

Or a time series that looks like this:
$$[5, 5, 5, 5, 8, 6, 2, 1, 1, 1, 1, 1, 1, -1, -1, -1, ...]$$

Or
$$[1, 1, 1, 1, 1, 1, 1, 1, ...]$$

(By the way, I padded with $(-1)$ to indicate that no information is available from that point and on)

I've encountered with some clustering algorithm that are dedicated for time series, and some time series similarity metrics like DTW.

I tried to use TimeSeriesKMeans with metric='dtw' but for some reason very "far" time series clustered together. I believe it has something to do with dtw which might be indifferent for the "intensity" of time the series? For the same reason, Pearson correlation might not be a good choice, right?

Also, should I normalize the time series? I thought about min-max normalization where max is global. What do you think?

EDIT:
Interestingly, increasing the number of clusters helped, but I still see weird stuff like $$[4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 3, 2, 3, 3, 4, 4, 3, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 477, 477, 477, 477, 450]$$ is clustered with time series like $$[98, 96, 96, 95, 94, 94, 94, 94, 94, 94, 93, 93, 92, 89, 89, 89, 89, 89, 88, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 84, 84, 84, 84, 82, 82, 82, 82, 82, 82, 81, 81, 78, 77, 75, 73, 72]$$


texts:
I have a question on the backpropagation in a simple neural network (I am trying to derive the derivative for the backpropagation).

Suppose that the network is simple like so (forward pass):

$$\begin{aligned}z_1 &= xW_1 + b_1  & (1) \\
  a_1 &= \tanh{(z_1)} & (2) \\
z_2 & = a_1W_2 + b_2 & (3)\\
a_2 & = \text{softmax}(z_2)=\hat{y}_n & (4) \\
l_n & = \log{(\hat{y}_n)} & (5) \\
PL_n & = l_n\dot{y}_n^T & (6) \\
L & = -\frac{1}{N} \sum_{n \in N} PL_n & (7)
\end{aligned}
$$

So only one hidden layer. Let the dimensions be the following:

$x$ is $1 \times 2$, $W_1$ is $2 \times 500$, $b_1$ is $1 \times 500$, $a_1$ is $1 \times 500$, $W_2$ is $500 \times 2$, $b_2$ is $1 \times 2$, then $\log{(\hat{y_n})}$ is $1 \times 2$, then $\dot{y}_n^T$ is $2 \times 1$.

Then the backpropagation algorithm (for a single sample), for this staged computation would proceed as follows (equipped with the knowledge that the derivative of $X$ whose dimensions are $N \times M$ will preserve those dimensions):

$$\begin{aligned}
dPL_n & = -1/N & (7) \\
d\dot{y}^T_n & = l_n^T  dPL_n & (6) \\
dl_n & = \dot{y}^T_n dPL_n & (6) \\
d\hat{y}_n & = 1/\hat{y}_n & (5)
\end{aligned}
$$

Backpropagation equation $(5)$ above is a bit of an abuse of notation, but what I am trying to say that it is a vector, whose values are $[1/\hat{y}_n^{(1)}, 1/\hat{y}_n^{(2)}]$. But I am stuck at this step. Because I am not sure about the softmax. I have found this. Which tells me that if $a_2$ is the resulting softmax vector, then I can take following four derivatives: $\frac{\partial a_2^{(1)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(1)}}$,$\frac{\partial a_2^{(2)}}{\partial a_2^{(2)}}$. However, the dimensions of $z_2$ are $1 \times 2$, thus there should only be two partial derivatives. It makes conceptual sense to me, that $dz_2$ should equal to $\left[\frac{\partial a_2^{(1)}}{\partial a_2^{(2)}}, \frac{\partial a_2^{(2)}}{\partial a_2^{(1)}} \right]$, but I cannot motivate my intuition. Also, if I generalize this. Suppose $z_2$ is $1 \times 10$, what symbolic expression for derivative: $dz_2$ would we have then?

(I was motivated to implement my neural network from scratch after reading this post by Stanford; hence this question).

Another resource I am using. I am inclined to believe that $dz_2 = \hat{y}_n - \dot{y}_n$, but why, I do not know ;(

I am also missing $dl_n$ in backprop in $(5)$, the only way to end up at $1 \times 2$, is if we do: $d\hat{y}_n = 1/\hat{y}_n \circ dl_n$.

I have just looked at the porblem in the following way, which did not help me much. We need to find $dz_2  = \frac{\partial L}{ \partial z_2}$, we can rewrite this using backpropagation chain rule $$\begin{align} \frac{\partial L}{ \partial z_2} &= \frac{\partial l_n}{\partial z_2} \frac{\partial L}{\partial l_n} \\ &= \frac{\partial l_n}{\partial z_2} dl_n \\ &= \frac{\partial}{\partial z_2} \log{(\text{softmax}(z_2))} dl_n \\ &=\frac{\partial}{\partial u} \log{u} \frac{\partial u}{\partial z_2} dl_n \\ &= \frac{1}{\text{softmax}(z_2)} \frac{\partial}{\partial z_2} \text{softmax}(z_2) dl_n \end{align}$$

Also, same question on Maths.SE.


texts:
I have a simple Question:

I am using XGBoost to classify some data:
1.) With 100 estimators I have following scores(roc_score):
    train_data : 98.5
    validation_data : 97.2

2.) With 500 estimators I have following scores(roc_score):
    train_data : 99.4
    validation_data : 97.7

Acco. to above can we say model with 500 estimators works better. Or should I change the validation data few times and see if similar kinds of increment happens by shifting from 100 to 500 estimators.


texts:
I have stored the some products and there respective category in sql.

Like this. Now I want fetch all product from table with respective category.
output should be=>


texts: R and Python, memory differences yielding performance differences

texts:
I think there are numerous posts regarding which one to use: R or Python. However, I'm curious about how their architecture differences yield differences in speed performance, not which one to use.

This blog post performs a small test between R and python to show that the (optimized) python code was 2x faster than R code.* And I've read in this post that R tends to put everything in memory, which is why computations on large datasets is generally slow.

But what makes python's low level memory management so much different than R, which helps it yield these benchmarks?

*Though python was 2x faster in this test than R, I'm not saying that python is generally 2x faster than R.


texts:
Most heatmap tutorials look at discrete data, where each cell has a well-defined boundary and a single value. How do you create a heatmap of continuous data, where individual points may be very close together without actually being identical?
Example: I have a set of 1000 map coordinates, all in the range (-8192, -8192) to (8192, 8192). They might look like this:
745.2218627929688,808.1373901367188
741.4943237304688,775.325927734375
757.3713989257812,806.7517700195312
-300.1320495605469,1425.9727783203125
-1845.8055419921875,2091.180419921875
-289.85308837890625,1429.4222412109375
-306.8935241699219,1434.9525146484375
-287.78253173828125,1439.9078369140625

The locations will tend to be clustered, and the clusters are the most interesting, but outliers are still of some value. What I'd like would be to show colours on an image that indicate this density.
It's highly likely that all I need is the right term to search for, but "heatmap" keeps finding discrete data tutorials only :)


texts: Unbalanced data set - how to optimize hyperparams via grid search?

texts:
I would like to optimize the hyperparameters C and Gamma of an SVC by using grid search for an unbalanced data set. So far I have used class_weights='balanced' and selected the best hyperparameters based on the average of the f1-scores.
However, the data set is very unbalanced, i.e. if I chose GridSearchCV with cv=10, then some minority classes are not represented in the validation data.
I'm thinking of using SMOTE, but I see the problem here that I would have to set k_neighbors=1 because in some minority classes there are often only 1-2 samples.
Does anyone have a tip how to optimized the hyperparameters in this case? Are there any alternatives?

Many thanks for every hint


texts:
I have a very large dataset with timestamp data. Till now, I loaded the whole dataset in order to train some models using python (statsmodels.api,  statsmodels.formula.api, keras.models.Sequential).
Now, I would like to train models with all historical data just one time, and then update the model just with new data (each day I have new samples).
Is it possible and how with these python libraries?
Thanks!


texts: How does the cost function of LSTM works?

texts:
I am searching to understand how does LSTM network work, but I couldn't find any good sources that explains how it's cost function works?

I mean I know we have a sequence of inputs x<1> to x<t>, and a sequence of y_hat<1> to y_hat<t>, but how do we calculate loss or cost function? Should we calculate  y_label<i> - y_hat<i> for each unit? If so, where do we accumulate them and when we use it? Where? How?

Is LSTM network supervised learning or I am confused?


texts: Oversampling only balances the training set, what about the testing set?

texts:
In a case of imbalanced data classification, I know that we only oversample the training set (to prevent data leakage from training to testing subsets), but what if there are no positive data points in my testing set? The testing set is still highly skewed and has only 1% of my positive class. I am using XGBoost, Random Forest, Logistic Regression, and KNN for the classification task.

Also, I have tried SMOTE, SMOTE-NC, and Class_weight to oversample my training set. To increase the chance of having more data from the minority class, I changed the 10-fold to 5-fold cross-validation (when developing the models), no improvement!

PS: I have >100K data points in my dataset.


texts: How to understand Bionomial Theroem and the Recursion Rule?

texts:
In this video from EDX, the instructor explains the binomial theorem as:
Binomial Theorem:
When you calculate $(a + b)^n = a^n + C(1)a^{(n-1)b} + C(2)a^{(n-2)b^2} + ... + C(n-1)ab^{(n-1) + b^n}$
The coefficients = $C1, C2, ..., C(n-1)$
Coefficient of $(a + b)^4 = 1, 4, 6, 4, 1$
From the formula above, you get after some manipulation:
$(a + b)^n = a^n + (n choose 1) a^{(n-1) b^1} + (n choose 2) a^{(n-2) b^2} + ... + (n choose n-2) a^2 b^{(n-2)} + (n choose n-1) a^1 n^{(n-1)} + b^n$
The coefficients will be the same as pascal's triangle
Is it necessary to know this? I still do not understand how he went from $(a + b)^n$ formula to plugging in $(n choose 1) (n choose 2)...$
Additionally, for the recursion theory:
Recursion Rule:
If we have 52 cards and we select 5 cards we have a collection of 5 cards taken from 52 cards
The order the cards are dealt does not matter
To get the recursion, we will split the hands into hands that contain the ace of spaces with hands that do not contain the ace of spades
For the hand that does not contain the ace of spades, there are 51 cards to choose from
So it is 51 choose 5
For the hand that does contain the ace of spades, then that hand is made from
4 cards in the remaining 51 cards in the deck
So it is 51 choose 4
So since every hand either has ace of spades or does not
We get our recursion
52 choose 5 is = 51 choose 5 + 51 choose 4
But if you actually calculate 52 choose 5 you get 311875200
And if you calculate 51 choose 5 + 51 choose 4 you get 287884800
To me it seems mathematically inaccurate?


texts:
I'm an R language programmer. I'm also in the group of people who are considered Data Scientists but who come from academic disciplines other than CS.

This works out well in my role as a Data Scientist, however, by starting my career in R and only having basic knowledge of other scripting/web languages, I've felt somewhat inadequate in 2 key areas:

Lack of a solid knowledge of programming theory.
Lack of a competitive level of skill in faster and more widely used languages like C, C++ and Java, which could be utilized to increase the speed of the pipeline and Big Data computations as well as to create DS/data products which can be more readily developed into fast back-end scripts or standalone applications.

The solution is simple of course -- go learn about programming, which is what I've been doing by enrolling in some classes (currently C programming).

However, now that I'm starting to address problems #1 and #2 above, I'm left asking myself "Just how viable are languages like C and C++ for Data Science?".

For instance, I can move data around very quickly and interact with users just fine, but what about advanced regression, Machine Learning, text mining and other more advanced statistical operations?

So. can C do the job -- what tools are available for advanced statistics, ML, AI, and other areas of Data Science? Or must I lose most of the efficiency gained by programming in C by calling on R scripts or other languages?

The best resource I've found thus far in C is a library called Shark, which gives C/C++ the ability to use Support Vector Machines, linear regression (not non-linear and other advanced regression like multinomial probit, etc) and a shortlist of other (great but) statistical functions.


texts: KNN custom transformer shows same accuracy for every k i set

texts:
I built custom trasformer for KNN and i can't figure why my k-number, when i set it, always shows same accuracy...

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import StandardScaler
from scipy.spatial import distance

def minkowski_dist(a, b):
    return distance.minkowski(a, b)

class MyKNN(BaseEstimator, TransformerMixin):

    def __init__(self, k, scale_data=False):
        super().__init__()
        self.k = k
        if scale_data is True:
            self.ss = StandardScaler()
        else:
            self.ss = None
        self.X_train = None

    def fit(self, X_train, Y_train):
        self.X_train = X_train
        self.Y_train = Y_train

    def predict(self, X_test):
        prediction = []
        for row in X_test:
            label =  self.closest_n(row)
            prediction.append(label)
        return prediction

    def closest_n(self, row):
        best_dist = minkowski_dist(row, self.X_train[0])
        best_index = 0
        for i in range(1, len(self.X_train)):
            dist = minkowski_dist(row, self.X_train[i])
            if dist < best_dist:
                best_dist = dist
                best_index = i
        return self.Y_train[best_index]

Been doing KNN on iris dataset

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris_dataset=load_iris()
X=iris_dataset.data
Y=iris_dataset.target

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=42, stratify=Y)

print(X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)

And this is what i get:

for k in range(1,21,2):
    knn = MyKNN(k)
    knn.fit(X_train, Y_train)
    prediction = knn.predict(X_test)
    from sklearn.metrics import accuracy_score
    print("Test accuracy for k={}:".format(k),accuracy_score(Y_test, prediction))


texts: Influence of trend on (supposedly) correlated time series

texts:
TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated?

I'm currently trying to reconstruct/cross-validate an analysis delivered by one of my companies contractors.

The data is based on time series of sensor data (approx. 3.5m timestamps). Goal was to find the signals with the highest correlation with one specific signal.

Despite not being an expert in data science I was able to reproduce their data cleaning (drop columns with zero variance, interpolate linearly over smaller gaps, drop remaining columns containing NaN-values). But after that I'm not sure if I can confirm their findings.

Seemingly they did a simple pearson-correlation like

corr = df.corrwith(df['DesiredSignal'])

Yet looking at the data the signals seem definitely trended.

When I then apply a detrend-function like

from scipy import signal

df_d = signal.detrend(df[column])
df_n = pd.DataFrame(data=df_d)

and apply the corrwith-function to this new dataframe I get totally different results (e.g. a significant higher amount of highly negativ correlations).

My Question now is: Can I trust the findings of the contractor or are they rendered invaild by not considering the influence of trends on correlation or am I getting something completly wrong?


texts: Many things behave differently in high dimensional space

texts:
It turns out that many things behave very differently in high dimensional space. The below paragraph is picked from a book. I need extra help to understand.
The book says,

  if you pick a random point in a unit square (a $1\times 1$ square), it will have only about a $0.4\%$  chance of being located less than $0.001$ from a border( In other words, it is very unlikely that a random point will be extreme along any dimension). But in a $10,000$-dimensional unit hypercube (a $1\times 1\times 1\times 1\times 1\times 1\ldots \times 1$ cube with ten thousand 1s), this probability is greater than $99.99999\%$. Most points in a high dimensional hypercube are very close to the border.

Q) What is the author trying to convey from the above paragraph?

If you pick two points randomly in a unit square, the distance between these two points will be, on average roughly $0.52$. If you pick two points in a unit 3D cube, the average distance will be roughly $0.66$. Now If you two points picked randomly in a $1,000,000$ dimensional hypercube, the average distance will be about approx $408$?

How is it possible?


texts: Using class weight in decision trees with Information Gain

texts:
How are weights considered in a decision tree when we want to maximize Information Gain? In other words, what would the entropy calculation become when weights are involved?
I can guess either
$$ e_1 = \sum_{i} w_iP(X=i)log(P(X=i))$$
or
$$ e_2 = \sum_{i} w_iP(X=i)log(w_iP(X=i))$$
should be used as the new weighted entropy but I'm not sure.
And is the rest of the algorithm same as before, using $e_1$ or $e_2$ instead of normal entropy, or are there other changes as well?


texts: Do weights of keywords for each topic add up to 1 in topic modeling?

texts:
When you run a topic modeling (say LDA), you can get outputs for some number of topics with corresponding keywords and their weights. Based on my understanding, people usually output top 10 or top 20 keywords for each topic. For these keywords, they also have weights which represent how important each keyword is for a certain topic.
For example, if I decided to draw out top 10 keywords for each topic, then the example output will be shown below.
topic 0: 0.2*keyword1 + 0.15*keyword2 + 0.09*keyword3 +... + 0.005*keyword10

topic 1: ...
...

topic n

I'm not sure how many maximum keywords I can draw out for each topic but do these weights for each topic add up to 1?


texts:
EDITED: See below for additional information..

TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset.

ORIGINAL:

I have a question about the mice package.

I am looking at a dataset like the example code:

set.seed(1337)
df <- data.frame(x = c(seq(100, 10, -4), seq(100, 20, -3), seq(100, 60, -3),
seq(80, 40, -3), seq(100, 10, -3))) %>% data.table()
df[sample(seq_len(nrow(df)), size = nrow(df) * 0.3), x := NA]
df$y <- seq(1,nrow(df),1)

Tmp <- mice(df, m = 1, maxit = 30, seed = 1337, print = FALSE)
Completed <- mice::complete(Tmp, 1)

plot(df$y, df$x, col = "blue")
points(Completed[is.na(df$x), ]$y, Completed[is.na(df$x), ]$x, col = "red")
legend(70, 30, legend = c("actual", "pred"), col = c("blue", "red"), lty = 1, cex = 0.5)

I am trying to fit the missing data points, but as you can see in the image below, the fit (red) isn't like I want it to be, because I would expect it to appear in between the blue line. Would anyone know how to 'fix' this?

For clearance: The data on the x-axis is interpreted as a datetime object while the value on the right is the current battery level (in percentage) of ONE device. The points where the blue dotted line stops and starts at a higher number is when the battery is charged.

UPDATE:
I have looked at the DMwR-package and especially at knnImputation. I was able to fit it to the sample set without any issue, see image below.

However it gave an issue when I tried to implement it with the real data set, because it threw an error:

Error in scale.default(xcomplete, dm[i, ], FALSE) : length of 'center' must equal the number of columns of 'x'

See sessionInfo below:

> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS release 6.9 (Final)

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
 [1] lubridate_1.3.3     zoo_1.7-12          tidyr_0.6.0         pROC_1.13.0         randomForest_4.6-14 DMwR_0.4.1          lattice_0.20-35     stringr_1.0.0
 [9] openxlsx_3.0.0      tictoc_1.0          data.table_1.10.4-3 wlhive_1.1.9003     ggplot2_1.0.1       dplyr_0.5.0


texts: xgboost calibration kde plots (isotonic) not smooth

texts:
i am training my xgboost model on an imbalanced binary classification problem. It is important to me to have well calibrated probabilities so i have chosen to optimize the brier score. I then plot the kde and reliability curve of my models where i try isotonic and platts.

e.g my grid search is:
gscv = GridSearchCV(pipeline, param_grid=params['xgboost'],scoring='neg_brier_score',
                    cv=kfolds.split(x_train, y_train),return_train_score=True)

the kde plot on the left corresponds to the uncalibrated probabilities , to me this looks good since the RED CURVE which is class 1  shows a bump towards lower end of curve , note the probabilites are the probability of default.
When i view the kde of the isotonic probabilities however it is not smooth:

my question is, are the above results satisfactory? I am not sure how i can improve this, for it to a) be smoother b) and be well calibrated.


texts:
Say I have an fMRI experiment where I have a task that is suppose to measure response time, and the corresponding "activation" in the brain. I then have participants complete this task. Then I measure some variables through tests or assessments, and questionaries such as IQ, working memory, age, and years of education.

In fMRI experiments, as I am aware typically voxels (the name for the 3d fmri representation of a pixel) that remain "active" after statistical tests, while the participant completes a task, say something about how that brain region where those voxels are modulates that cognitive task.

Now, what if I want to use the variables that I measured, IQ, age, working memory, and years of education in a multiple linear regression test, with the brain activation to this task as the response variable, or dependent variable. What does this tell me?


texts: What would be a good way to use clustering for outlier detection?

texts: For simplicity let's assume the feature space is the XY plane.


texts: In which cases shouldn't we drop the first level of categorical variables?

texts:
Beginner in machine learning, I'm looking into the one-hot encoding concept.

Unlike in statistics when you always want to drop the first level to have k-1 dummies (as discussed here on SE), it seems that some models needs to keep it and have k dummies.

I know that having k levels could lead to collinearity problems, but I'm not aware of any problem caused by having k-1 levels.

But since pandas.get_dummies() has its drop_first argument to false by default, this definitely has to be useful sometimes.

In which cases (algorithms, parameters...) would I want to keep the 1st level and fit with k levels for each categorical variable?

EDIT: @EliasStrehle's comment on above-mentioned link states that this is only true if the model has an intercept. Is this rule generalizable? What about algorithms like KNN or trees which are not exactly models in the statistic definition?


texts: flexibility vs complexity vs number of predictors in machine learning

texts: I'm new to machine learning so am quite confused with the above concepts. It seems to me both flexibility and complexity measures how well the model fit the data (in terms of the curvy-ness), so what's the difference between the two? In addition, I saw some websites say complexity can be indicated by the number of features in the model; but does 'feature' in this context mean variable, e.g. x and y (two variables), or term, e.g. x and x^2 (two terms), and is it different from 'predictor'? Also, I know flexibility can affect the generalization error, but except that, does flexibility affect any other aspects of model performance? Thanks for answering.


texts: What makes a graph algorithm a good candidate for concurrency?

texts: GraphX is the Apache Spark library for handling graph data. I was able to find a list of 'graph-parallel' algorithms on these slides (see slide 23). However, I am curious what characteristics of these algorithms make them parallelizable.


texts: Unable to open application master UI in spark1.6.1 in cluster mode

texts:
Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2.

when i submit the application in cluster mode. the driver Container can be in any of the 8 nodes. If driver Container goto the node 3 (where Active ResourceManager is service is running) then i am able to open application master UI, but in other cases it is not opening, after some time ambari will show critical alert with message Connection failed to resource manager host url.

if i check the resource manager logs. It shows some access exception for spark user while calling getServiceState.

Here is the full stack strace :

2018-08-25 05:02:30,209 WARN  resourcemanager.AdminService (RMServerUtils.java:verifyAdminAccess(185)) - User spark doesn't have permission to call 'getServiceState'
2018-08-25 05:02:30,210 WARN  resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(345)) - USER=spark IP=11.111.1.11  OPERATION=getServiceState TARGET=AdminService RESULT=FAILURE  DESCRIPTION=Unauthorized user PERMISSIONS=
2018-08-25 05:02:30,210 INFO  ipc.Server (Server.java:logException(2294)) - IPC Server handler 0 on 8033, call org.apache.hadoop.ha.HAServiceProtocol.getServiceStatus from 11.111.1.11:40169 Call#51845 Retry#0
org.apache.hadoop.security.AccessControlException: User spark doesn't have permission to call 'getServiceState'
  at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:191)
  at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:157)
  at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAccess(AdminService.java:232)
  at org.apache.hadoop.yarn.server.resourcemanager.AdminService.getServiceStatus(AdminService.java:365)
  at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.getServiceStatus(HAServiceProtocolServerSideTranslatorPB.java:131)
  at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4464)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
2018-08-25 05:05:43,300 INFO  client.DefaultHttpClient (DefaultRequestDirector.java:tryExecute(726)) - I/O exception (org.apache.http.NoHttpResponseException) caught when processing request: The target server failed to respond


texts: R: saving/printing all intermediate results

texts:
currently, we have developed a model at work and it is at the verification step. the goal is for another person to receive output from each and every step of calculation to verify.
The issue is that the model is quite complex and large due to Montecarlo simulation and also being assembled from various scripts (ca. 20 scripts with further for/if test within).
it also has too many intermediate and temporary variables (like temporary row index or col index) to use write.csv() after every line.
Hence, I wonder if anybody faces the same issue before or having a systematic package or method to implement these features? Many thanks. for example:
f <- function(x) { #toy function doing rowSums
output <- rowSums(x); return(output) }

set.seed(12345)
# generate a temporary matrix
randomMatrix  <- matrix(rnorm(1e6*10),ncol = 10, nrow = 1e6) #this need to be saved/stored/printed
y             <- 1*2 + 3 #this need to be saved/stored/printed
t             <- f(x)    #this need to be saved/stored/printed

finalvariable <- t^y     #this need to be saved/stored/printed

# many thanks


texts: Python: How to plot time interval from a Dataframe in Pandas

texts:
I have the a dataframe(df) which has the data of a Job being executed at different time intervals. It includes the following details about the execution of a job:

Job Start Time (START)
Job End Time (END)
Time Interval (interval) i.e., END - START.

A small part of dataframe is shown below.
Dataframe(df):
  END    |  START   |  interval
1423.0   |  1357.0  |    66.0
33277.0  |  33325.0 |   -48.0
42284.0  |  42250.0 |    34.0
53466.0  |  53218.0 |   248.0
62158.0  |  62073.0 |    85.0

I want to plot a graph with the x-axis as the timestamp and the y-axis with the interval. I tried to do it with the START time but it is not giving the correct result. How can we do this?
Code
fig_dims = (12, 10)
fig, ax = plt.subplots(figsize=fig_dims)

sns.lineplot(x = 'START', y = 'interval', data = df, ax = ax)

Output

Required Output
x-axis - Timestamp
y-axis - Interval


texts: Keras - Error when using HDF5Matrix to fit the model

texts:
So I define my data like this:

x_train = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'x_train')
y_train = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'y_train')
x_test = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'x_test')
y_test = keras.utils.io_utils.HDF5Matrix('dataset.h5', 'y_test')

But then when I try to fit the model like this:

model.fit(x_train, y_train, epochs=epochs,
          validation_data=(x_test, y_test),
          callbacks=[tensorboard, modelcheckpoint],
          batch_size=batch_size, shuffle=False)

I get this error:

File "G:\CryptoForecast\cryptomodel.py", line 34, in train_model
    model.fit(x_train, y_train, epochs=epochs, validation_data=(x_test, y_test), callbacks=[tensorboard, modelcheckpoint], batch_size=batch_size, shuffle=False)
  File "C:\ProgramData\Anaconda3\envs\TradingBot\lib\site-packages\keras\engine\training.py", line 952, in fit
    batch_size=batch_size)
  File "C:\ProgramData\Anaconda3\envs\TradingBot\lib\site-packages\keras\engine\training.py", line 670, in _standardize_user_data
    'You passed: x=' + str(x))
ValueError: Please provide as model inputs either a single array or a list of arrays. You passed: x=<keras.utils.io_utils.HDF5Matrix object at 0x000002342575E4E0>

But it should be working, no?


texts: (Crowdsourced) Dataset with label/annotation metadata like duration/quality

texts:
I'm looking for a research project for datasets (the more the better), potentially crowdsourced, which go beyond basic feature vectors + labels, and include additionally some metadata about the labels. Specifically I'd like the annotation time, or some other cost metric, and if the dataset was crowdsourced, the individual labels per annotator. On top, some measure of the quality of the labels by different annotators would be helpful as well. The Dataset domain is not relevant.
Also I'm not sure if https://datascience.stackexchange.com is the right place to ask for that, but maybe you can help me.
So far I could only find this dataset.


texts: How to handle Memory issues in training Word Embeddings on Large Datasets?

texts:
I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the dataset and need ideas on how to train word embeddings on such a large dataset which is a collection of 243 thousand full article documents. The research computing resource is timed so get short access to GPU node and so opted for Incremental model training:

Incremental Model training: One way to train on entire dataset is to use incremental model training that is train the model on one chunk of the data and save it. Later on pick up the same pre-trained model and start training on it next chunk of data. The problem that I am facing in this approach is that how do I maintain the vocabulary/dictionary of words. In word embedding methods dictionary/vocab plays an important role. We sweep over all documents and create vocab of words that have count greater than a minimum set frequency. Now, actually this vocab is a hash map which has index associated to each word and in training samples we replace words with their indices in the vocab for the simplicity in the model. In the case of incremental training how do I create dictionary incrementally? Do I have to create vocab/dictionary on entire documents initially and then train incrementally? Or is the a way to extend vocab also in incremental training?
Another problem is memory limit on the size of the vocab data structure. I am implementing my model in Torch which is LUA based. So, LUA puts a limit on tables size, I cannot load vocab for entire documents in a single table. How to overcome such memory issues?
Taking inspiration from Glove vectors. In their paper they say that they “We trained our model on five corpora of varying sizes: a 2010 Wikipedia dump with 1 billion tokens; a 2014 Wikipedia dump with 1.6 billion to- kens; Gigaword 5 which has 4.3 billion tokens; the combination Gigaword5 + Wikipedia2014, which has 6 billion tokens; and on 42 billion tokens of web data, from Common Crawl5. We tokenize and lowercase each corpus with the Stanford tokenizer, build a vocabulary of the 400,000 most frequent words6, and then construct a matrix of co- occurrence counts X” . Any idea on how Glove vectors trained on such a big corpus and big vocabulary and how memory restrictions in their case might have got handled?  Paper reference - http://nlp.stanford.edu/pubs/glove.pdf
Any ideas on how to limit the size of the dataset for generating word embeddings? How would it affect the performance or coverage of word embeddings with the increase or decrease in number of documents? Is it a good idea to use sampling techniques to sample documents from dataset? If yes, then please suggest some of the sampling techniques.


texts:
I'm trying to analyze some data I have but there is a lot of inconsistencies in my data.

I have a SQL table that I'm trying to analyze.

The table is a table of universities with the following structure: name:string, city:string, state:string, country:string
Name is always present however city, state, country can be missing.
My main issue is that there are a ton of typos and different declination of a university name.
For example here are the declination of Standford Unversity I find when I do  SELECT "universities".* FROM "perm_universities" WHERE (name like '%stanford%'):

stanford university - stanford - ca - united states of america
the leland stanford junior university - stanford - ca - united states of america
leland stanford jr. university - stanford - ca - united states of america
stanford university graduate school of business - stanford - ca - united states of america
the leland stanford junior university (stanford university) - stanford - ca - united states of america
leland stanford junior university - stanford - ca - united states of america
stanford university - stanford -  -
leland stanford jr. university, graduate school of business - stanford - ca - united states of america
stanford law school - stanford - ca - united states of america
stanford - stanford - ca - united states of america
stanford university, graduate school of business - stanford - ca - united states of america
stanford graduate school of business - stanford - ca - united states of america
stanford univerity - stanford - ca - united states of america
stanford university (the leland stanford junior university) - stanford - ca - united states of america
the leland stanford jr. university - palo alto - ca - united states of america
leland stanford junior university, school of law - stanford - ca / n/a - united states of america
stanford universit - stanford - ca - united states of america
the leland stanford university - stanford - ca - united states of america
leland standford stanford junior university - stanford - ca - united states of america
stanford university - cambridge - ma - united states of america
the leland stanford junior university 'stanford university' - stanford - ca - united states of america
stanford university school of law - stanford - ca - united states of america
stanford univresity - stanford - ca - united states of america
the leland stanford jr. university (stanford university) - stanford - ca - united states of america
leeland stanford junior university - stanford - ca - united states of america
leland stanford junion university -  - ca - united states of america
leland stanford junior university (stanford university) - stanford - ca - united states of america
the leland stanford junior university - stanford -  -
stanford university - graduate school of business - stanford - ca - united states of america
graduate school of business, stanford university - stanford - ca - united states of america
stanford universoty - stanford - ca - united states of america
leland stanford junior university - stanford -  -
stanford univeristy - palo alto - ca - united states of america
leland stanford university - palo alto - ca - united states of america
stanford university - stanford - ca / n/a - united states of america
the leland stanford junior university, stanford university - stanford - ca - united states of america
the leland stanford junior university graduate school of business - stanford - ca - united states of america
stanford universtiy - stanford - ca - united states of america
stanford univerisity - stanford - ca - united states of america
stanford university - stanford - ct - united states of america
stanford law scool - stanford - ca - united states of america
mba: stanford university - stanford - ca - united states of america

They are all the same university, but some have typos, some have different names, some have no cities, some have the wrong cities, ... The data isn't great.

So I'm trying to fix it. How can I consolidate this data?


texts: Python: multiplication of sparse matrices slower in csr_matrix than numpy

texts:
I am trying to multiply a sparse matrix with itself using numpy and scipy.sparse.csr_matrix. The size of matrix is 128x256. Its 93% values are 0. Ironically the multiplication using numpy is faster than scipy.sparse. I do not know why? The code I am using is:

import numpy,time
W=numpy.random.choice([0, 1], size=(128,256), p=[0.93,0.07])
start=time.time()
W1=numpy.matmul(W,numpy.transpose(W))
end=time.time()
print(end-start)

from scipy.sparse import csr_matrix
start=time.time()
W1=csr_matrix(W).dot(csr_matrix(W).transpose())
end=time.time()
print(end-start)

Numpy gives time 0.0006 and scipy gives 0.004. Why. Comparing times for dense matrix, numpy gives smaller time on dense matrix as well and scipy takes more time. Why is the time for scipy.sparse not less than numpy for sparse matrix


texts: How smaller does the input data get reduced in a LSTM autoencoder

texts:
Question

In a LSTM autoencder, how smaller does my input data(59 features) get reduced in a latent vector, which is usually located in the middle between an encoder and a decoder?
Why did the author increase the feature number from 5 to 16 in the middle of encoding stage. This question is described in more detail below after the picture of LSTM autoencoder structure.

My questions are based on the article LSTM Autoencoder for Extreme Rare Event Classification in Keras. You can take a look at the codes from this github repository. Please refer to these resources to get to know my questions better.

 Details of the Question

My Autoencoder model is as follows:

lstm_autoencoder = Sequential()

# Encoder
lstm_autoencoder.add(LSTM(timesteps, activation='relu', input_shape=(timesteps, n_features), return_sequences=True))
lstm_autoencoder.add(LSTM(16, activation='relu', return_sequences=True))
lstm_autoencoder.add(LSTM(1, activation='relu'))
lstm_autoencoder.add(RepeatVector(timesteps))

# Decoder
lstm_autoencoder.add(LSTM(timesteps, activation='relu', return_sequences=True))
lstm_autoencoder.add(LSTM(16, activation='relu', return_sequences=True))
lstm_autoencoder.add(TimeDistributed(Dense(n_features)))

lstm_autoencoder.summary()

The shape of the input data is X_train_y0_scaled.shape = (11692,5,59). It means that we have 11692 batches. Each batch is comprised of 5 rows and 59 columns, and since the data is time series data, it means that 59 features are gathered for each of 5 days.
The summary of the autoencoder model is as follows:

- Side question : I cannot understand why the author of this codes increased the feature number from 5 (in the layer 'lstm_16') to 16 (in the layer 'lstm_17').
- The original number of feature is 59, so in the first layer the feature number got reduced from 59 to 5. But after this dimension reduction, why would someone want to increase the dimension in the encoding stage?!

It is easy to see how smaller the shape of the input data gets reduced in a latent vector, if it is a fully-connected autoencoder. For example, in the picture below 10-node long input data gets reduced to 3-node long latent vector.

However in the LSTM autoencoder, it is not clear to me how long the 59 feature-long vector got reduced to. The layer lstm_18 is only 1-node long while the repeat vector is (5,1) long. Does that mean that 59-feature long vector got reduced to 1-node long vector?


texts: I want to implement ML to monitor log file, classify them as normal and abnormal. The model must learn via this process of classification and after time be able to classify the log files itself. This classification of the log files is based on time of login and place of login. Any hint / advice of ready made model for the scenario would be of great help.


texts:
From my machine learning class,it seems L1 norm regularization is the standard way to obtain sparse and probably better fit for machine learning problems. But from pybrain, it seems L1 norm regularization is not implemented, rather L2 regularization is implemented which is quite weird to me...

It is implemented as followed:

trainer = BackpropTrainer(net, ds, weightdecay=0.01)

Is it possible for me to implement L1 norm regularization in pybrain? Even if I need to modify the source code? Thank you.


texts: How to evaluate a model based on the effect of one important variable?

texts:
We want a well-calibrated classifier that tells us the probability of an event. The model has multiple inputs, but we are interested in how the probability of an event changes as we vary one input. This is the primary use of the model so it is very important.

Ex. What is the probability that you buy my sandwich?

Inputs: meat type, cheese type, sandwich weight, price

The real question of interest is, how does the probability of buying the sandwich change with the price?

I am a sandwich shop using this model to set my sandwich prices, so I really mostly care about how you respond to changes in price. But I still need the other variables because you have sandwich preferences that impact how much you are willing to pay.

Suppose we have several candidate models with similar performance in AUC and calibration, but each one shows a different effect of price on probability. How do we select which model to use?

For the same sandwich, model 1 vs model 2 give different probability of buying the sandwich as you vary the price input:


texts: Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time

texts:
First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., is anything unclear?).

My main concern is whether I have adequately displayed the data and
whether I can do anything useful to improve (e.g., moving average)?
Additionally, whether I have interpreted this time series data appropriately:

"The relative frequency of affect-related tokens (counts per 10,000 tokens in psychology abstracts) increased from 3.51% in 1980 to 4.87% in 2017—-an overall relative increase of 39%. The relative frequency of affect-related tokens shows an increase at a rate of approximately 0.037 units per year (over 37 years). Overall, this displays a rapid growth trend in academics' use of emotion-related terms in psychology abstracts over time."


texts: XGBoost multiclassification interpreting predicted probabilities

texts:
Let's consider an example. I have patient level data, their symptoms, reading from various medical tests. Based on that, I have built a binary classifier given patient data to classify if they are likely to have a disease, and if so, I'd want to manually run medical tests on them. For that case, we look at the disease class and rank order the predicted probabilities and select the riskiest patients to have the disease.

A few months later, say the disease evolves into a different strain. So we already have an imbalanced class of healthy vs infected patients. Not the new strain will be still a smaller population. Apart from that, the symptoms, various test readings will also be different. So translating this to a multi-classification problem makes sense. When you build a multi-classifier, and you have three probabilities, how would you go about rank ordering the same set of patients?

If you're interested in the risk of patients likely to be infected, taking the max of the either of the strain of disease a good way to proceed or is there a method to choose them based on their individual probabilities?

Edit: To bring a little more clarity.
The multi:softprob gives the probability for each of the classes. If you have three classes, it will give three probabilities for each class summing up to 1. More details are here.

Given that we will get multiple probabilities for each row/patient from the example above, how do you go about choosing the final probability to rank order risky patients.


texts: Sampling methods for Text datasets (NLP)

texts:
I am working on two text datasets, one is having 68k text samples and other is having 100k text samples. I have encoded the text datasets into bert embedding.
Text sample > 'I am working on NLP' ==> bert encoding ==> [0.98, 0.11, 0.12....nth]
               # raw text 68k                              # bert encoding [68000, 1024]

I want to try different custom NLP models on these embeddings, but dataset large to test the model's performance quickly.
To check different models quickly, the best way is to take a small subset of dataset from the entire population and feed it to different algorithms. At last, choose the top algorithms to fit the entire dataset.
I am planning to sample at least 10k samples subset from 68k dataset and 10k subset from 100k dataset. I could select randomly 10k from 68k but that method is not the best way to sample.
Any advice on how to sample embeddings(text) from 68k samples while maintaining the probability distribution of the original population and how many samples would be enough for one sample subset?
Thank you!


texts: Why decision tree needs categorical variable to be encoded?

texts: As per my intuition, decision trees should work better with categorical variables than with continuous variables. If this is the case, why is encoding needed on categorical variables? Can someone give me the intuition behind this?


texts:
I wrote a simple CNN with a maxpool, a dense layer and a drop layer. Unfortunately there is two messages why it doesn't work.
In a normal session codes complaines about a misshape between y_conv and y_

nsorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1400] vs. [100]
 [[Node: eval/Equal = Equal[T=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](eval/ArgMax, eval/ArgMax_1)]]

If I use the invoke stepper from the tensorflow debugger I get the message "Attempting to use uniniatialized variables". I am very confused. Does anyone have a clue?

x = tf.placeholder(tf.float32, [None, 784], name="entry")

#layer conv 1

x_image = tf.reshape(x, [-1, 28, 28, 1])

W_conv1 = weight_variable([5, 5, 1, 32], "Wconv1") # 5 x 5
b_conv1 = bias_variable([32], "bconv1")

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)

#layer pool1
h_pool1 = max_pool_2x2(h_conv1)
h_pool1 = tf.Print(h_pool1, [tf.shape(h_pool1)], 'shape=')

# Dense Layer

W_fc1 = weight_variable([ 7 * 64, 1024], "Wconv2") # 7 x 7
b_fc1 = bias_variable([1024], "bconv2")

h_pool2_flat = tf.reshape(h_pool1, [-1, 7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

#drop out

keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout (h_fc1, keep_prob)

W_fc2 = weight_variable([1024, 10], "WconvD")
b_fc2 = bias_variable([10], "bconv2")

# softmax

y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2, name="y_conv")

with tf.name_scope('train'):
    y_ = tf.placeholder(tf.float32, shape=[None, 10], name="y_")
    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
    train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

with tf.name_scope('eval'):
    correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

These are the definitions of some functions:

def weight_variable(shape, name):
    initial = tf.truncated_normal(shape, stddev=0.1, name=name) #normalverteilung
    return tf.Variable(initial)

def bias_variable(shape, name):
    initial = tf.constant(0.1, shape=shape, name=name)
    return tf.Variable(initial)

def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name="maxpool")


texts: Methods to analyze time series data when only small amount of it is labeled

texts: I have time series data out of which only 10% is labeled into 10 classes. What should be the methods or models that I should look for to analyze this problem? I know the question is a bit vague but I am not looking for exact answers. I would appreciate any pointers or resources on how to approach these types of problems.


texts: Confidence Score For Trained Sentiment Analyser Model

texts:
I have trained a text based sentiment analysis model, using SciKit-learn and custom data. I have the model ready and it works fine in predicting a text to a class (Positive or Negative or Neutral). I have achieved over 85% testing accuracy and around 80% cross validation accuracy.

But I want to get the confidence score attached to each of my prediction to a new example data/text I feed to the classifier. This is just an extra parameter I want to show/output apart from just the predicted class.

I have no idea how to achieve this, I shall be really thankful if anyone can provide some helpful insights.


texts: When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

texts:
In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer.
What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations when dropout layer is better?


texts: What is going on with this kind of validation loss graph?

texts: I am using stock prices and a whole bunch of indicators values to try to get a tensorflow model to predict to buy,sell, or hold.  I think im going about this right but when i train the model, first i set a learning rate scheduler to increase the learning rate until the model converges and i use the training rate from the graph where the train loss and val loss first make their steeppest slope down for the next training run.  so then the losses are smoothly going down and eventually the training loss will keep going down but the validation loss starts to move back up again.  I dont know what that means, if its bad or good haha. Ive been at this all day.


texts: what is correct way to perform normalization on data in Auto encoder?

texts:
working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of input is essential for my dataset.

problem with normalization is that when noise value is very high compare to entire dataset. then prediction follows noise. for example if I add  noise (delta=300) to 80% of the data and perform normalization on the dataset which mean value is 250 and standard deviation is 79. here noisy data points(80% of the total dataset) are greater than 300. after normalization, I feed this dataset to the model, prediction follows the noise and giving wrong output. this happens because of feature scaling. when I add noise to most of data points, model consider this points as normal data points and rest of as anomalous data points.

In inverse scaling process, I can not use min-max values of my input to perform inverse scaling on prediction otherwise its follow noise in dataset.

so what is right way to perform feature scaling in denoise kind of problem?


texts: What is the difference between fully observed and partially observed state features in Reinforcement learning ?

texts: When we use a function to approximate Q values or policy to state-action pairs we hear the above two terms fully and partially observed features. what is the difference . Normally in Atari Games they generate Q values by taking the current pixels distribution of the frame .


texts: I don't know local spatial correlation meaning

texts: In GoogLeNet paper, 'local sparse correlation' is mentioned. But, I don't know the meaning. Also, I don't understand that statement " In the lower layers (the ones close to the input) correlated units would concentrate in local regions. This means,we would end up with a lot of clusters concentrated in a single region and they can be covered by a layer of 1×1 convolutions in the next layer"


texts: Is there any existing algorithm or formula that can calculate the complexity of a dataset?

texts:
Suppose that we have a dataset of n elements each element is composed of m sub elements, is there any existing algorithm that we can use to calculate the complexity of this dataset?

I mean by the complexity anything that includes the following elements:

Diversity of dataset (number of distinct elements)
Distance between each element
Correlation between the elements of this dataset
Size of the dataset


texts:
I am trying to predict a car price from a given dataset. When I am trying to encode the categorical data I am getting an error. I am attaching an image with the error below. Can someone help me know what I am doing wrong ? I am new to this domain.
Above this code block I imported libraries and dataset and divided it into X and y.


texts: Classifying one particular class of documents from the rest

texts:
I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I would sample 'not a sports document' category as there can be anything - book, news article, resume, invoice etc. How would approach this problem?

I already tried training One class SVM classifier with my sample of sport documents, however the accuracy turned out to be awful - around 6%.

I also read about PU learning, do you think this is the way to go? Are there any other options?

Thank you.


texts: Correlation as an evaluation metric for regression

texts:
I'm dealing with a regression prediction challenge where the evaluation metric is (pearson) correlation.
However, I have the impression that this metric is kind of arbitrary.
While I can keep the RMSE stable the correlation can have great variety.

Could someone please explain this metric and how to optimise for it?


texts: If I have to recommend 10 movies to the users

texts:
Let's say I have some information about a user and movie data similar to the following:

user: age, gender, height, nationality, etc
movie : actor, genre , director info
userwatch : userA watch movieA at 2009,,.......
userComment : userA comments about movie

In this case, if I would like to recommend 10 movies to each user, which machine learning technique would I use?

I was thinking about making a data frame and running RandomForestClassifier, but in this case, it's hard to recommend exactly 10 movies to each user.

I was also thinking about using K-means clustering, but when the movie number is over 200000, i'm not sure k-means is the best choice.

Any help is greatly appreciated.


texts: Assigning numerical IDs to variable values in a data file

texts:
I have the problem that I got a huge source data file which is showing text for all variable values instead of numerical IDs. So for example, I would like to have the variable gender coded as 1 and 2 instead of "female" and "male" written out. And equally the same for 200 other variables of which some have up to hundreds of distinct variable values.

Therefore, doing this manually is not really an option here.

Could anybody please point me to a solution or hint within R, SPSS or Python how I can assign numerical IDs to each distinct variable value?

I thought this would be a problem other people face more commonly as well, but I have found nothing of this kind at all.

Thank you for any help!


texts:
I am modeling an Independent cascade model in graph diffusion, by simulating a disease.
I have an undirected graph, and I need to choose 50 nodes to vaccinate before the disease starts spreading, and my goal is to get to the least amount of infected nodes.
I am trying to figure out a way to choose those 50 nodes wisely in order to achieve the beat outcome.
I thought about choosing the 50 nodes with the highest degree, but I am looking for alternatives and better ways.


texts: Dictionary learning for image classification

texts:
I'm wondering if the approach I'm thinking of could even work.
I want to use dictionary learning for image classification. The first step would be to learn the dictionary from a set of similar yet different images to be able to extract background from an image. For example, I have a set (e.g. 500 photos) of images of the same object, but the scenes differ (light, the angle the photo was taken at etc.) Basically, the main object is the same, or more precisely, the objects are very similar, they all show the main object. Is it possible to use dictionary learning to model the main object? My next step would be to model the main object (the background) with sparse coding techniques and detect "unexpected" objects. This is so called a contrario approach in object detection problems.
I'm inspired by these publications:

https://ijpam.eu/contents/2013-89-2/5/5.pdf
https://arxiv.org/pdf/2007.15757.pdf

Next, what software would let me do this? I found the mlpack package (for R and Python). It has a function sparse_coding that performs dictionary learning. I tried it on MNIST dataset and I can model digits, denoise them etc. But haven't tried on images of my interest so far. The learning process will take a lot of time, I guess. I took a while for MNIST 28 x 28px images.
Does my experiment has sense? What do you recommend? Do You know any tutorials with code samples I can take a look?


texts: Is PCA considered a machine learning algorithm

texts:
I've understood that principal component analysis is a dimensionality reduction technique i.e. given 10 input features, it will produce a smaller number of independent features that are orthogonal and linear transformation of original features.

Is PCA by itself considered as a learning algorithm or it is a data pre-processing step.


texts: How does word2vec handle the input word being in the context?

texts: If word2vec encounters the same word multiple times in the same window, what occurs?  Obviously it is meaningless to decrease the distance between the vectors for the input word and the target word.  But will the repetition strengthen the relationship between the repeated word and the context words?


texts: Reading Date Time in R (01.01.2019 00:00)

texts:
How do you read a date in R that is given in the following way:

01.01.2019 00:00

While reading it with read.csv("my_file.csv", sep = ";", dec = ",") I am getting the date column as a "factor" class. I am tring to figure how to read this data time data as a date. Any thoughts?

Thank you!


texts: multivariate LSTM for varying sequence length and varying Batch size

texts:
I receive data from a sensor and after some preprocessing, I get data points (1-18) with 8 features in every for-loop. That is, in one for-loop I get data of size 10x8 points while in the 2nd run, I get 18x8. where [10,18] represents the number of data-points while 8 are the number of features and so on. The number of data-points differs in every for-loop run. To make it stable I padded zeros where it is less than 15 and ignore the data points if the count is more than 15. (for above two runs, data became 15x8 and 15x8)
Training Data set: I have a data set from 20 measurements file and each file has the different batch size(number of training data). Every File has a different time duration. Let's say if one cycle is 1 sec. One file ran for 384 sec while another for 500 sec.
What I did:
I am using multivariate LSTM with labels 0 and 1 to do binary classification and padding zeros to get equal sequence data points i.e, from one file (386,15,8) while from another file (1380,15,8). In the end, I am giving Keras input_shape=(15,8). The model performance is really bad, I changed many hyper-parameters. Should I avoid zero-padding and give input_shape=(None,8) or should I involve batch_size somehow? Can anyone guide me on this?


texts: Do I Need Pretrained Weights For Keras VGG16?

texts:
As context, I am relatively new to the world of machine learning and I am attempting a project with a goal of classifying plays in an NBA game. My inputs are a sequence of 40 frames from each play in an NBA game and my labels are 11 all-encompassing classifications for a given play.

The plan is to take each sequence of frames and pass each frame into a CNN to extract a set of features. Then each sequence of features from a given video would be passed onto an RNN.

I am currently using Keras for most of my implementation and I chose to use a VGG16 model for my CNN. Here is some of the relevant code below:

video = keras.Input(shape = (None, 255, 255, 3), name = 'video')
cnn = keras.applications.VGG16(include_top=False, weights = None, input_shape=
(255,255,3), pooling = 'avg', classes=11)
cnn.trainable = True

My question is - would it still be beneficial for me to initialize the weights of the VGG16 ConvNet to 'imagenet' if my goal is to classify video clips of NBA games? If so, why? If not, how can I train the VGG16 ConvNet to get my own set of weights and then how can I insert them into this function? I have had little luck finding any tutorials where someone included their own set of weights when using the VGG16 model.

I apologize if my questions seem naive but I would really appreciate any help in clearing this up.


texts: Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text?

texts: I have a movie transcript without commas, punctuation, or newlines. Is there any NLP technique that can help to implement this?


texts: Normal vs Uniform Distribution for machine learning

texts:
I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is would introduce a bias, and thus I was thinking of restructuring the data to fall into buckets. Thus my model would be a multi-class classification model, rather than a regression model (I am training a NN).
My question is whether I should draw up the buckets such that the distribution of the items is uniform, or normal. A uniform distribution would ensure that the NN has the same amount of examples of each bucket, however, some say that normal distributions work better for machine learning. Which one should I use?
Thanks


texts: How to filter data samples which do not improve classifier?

texts:
I have a text dataset with noisy labels and an unbalanced shape.
There are various ways to find features which do not drive improvement in some metric, and help to prune those from the pipeline.
I cannot however find a way to throw away noisy / confusing samples.
One naive approach might be to run the model removing a single sample at a time and see if the metric improves or does not change - if that happens, throw away that sample.
Is there a name for such a technique? Or does it not exist for good reason :)


texts: Choosing best methods for estimating the unknown parameters in a linear regression model

texts:
Given some dataset for prediction,

for eg say I have different housing price prediction dataset:

dataset  1 : 100 training and 100 testing sample, 50 feature

dataset  2 : 100 training and 100 testing sample, 120 feature

dataset  3 : 1000 training and 1000 testing sample, 50 feature

dataset  4 : 1000 training and 1000 testing sample, 5000 feature

how should I choose the best methods for estimating the unknown parameters ( predict price) in a linear regression model from the following for each of these dataset?

Ordinary least squares
Stepwise regression
Principal component regression
Partial least squares regression

Should I experiment with each of these one by one and compare the results or is there any rule of thump on when to use each of them based on the dataset ?

Please help


texts:
I posted the question also on ai.stackexchange but it didn't get any answers so I though I could try here.
Here is a copy paste:

Let's say you are training a NN in a RL setting where the state (i.e. features/input data) does not change in every step, but rather in a few (~typically around 8 steps).
For example, a state might consist of the following values:
30, 0.2, 0.5, 1, 0
And then again the same values for 6-7 times more resulting in ultimately the following input arrays:
[[30, 0.2, 0.5, 1, 0], [30, 0.2, 0.5, 1, 0], ..., [30, 0.2, 0.5, 1, 0]]
I know that the value 0 in the feature set depicts that the weight for this feature results in insignificant value.
But what about repetition of values?
How does that affect learning, if it does at all?
Any ideas?
Best, Marios.
Edit: I did not know how to search for this particular topic, it might have been answered >before, maybe you can point me there. Thanks.

Edit: I was advised to delete this question and leave it only on AI SE. However, and since I have been upvoted already, people might want to trace it back here for an answer.
Only for that reason, I leave the question here with the link that traces back to the original updated question and accepted answer.


texts:
Problem
I have data that includes multiple different text inputs as well as floats, categories, etc. Therefore I need to pass several different data types as features, including text which is an int array when tokenized.

Question
Say I tokenize the several text inputs; can I pass the tokenized text array as a feature alongside my floats and categories? If not, how is this done?

Background
When I've done NLP models, my code looks similar to this:

...
tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(df['Stem'])
list_tokenized_train = tokenizer.texts_to_sequences(df['Stem'])
X_train = pad_sequences(list_tokenized_train, maxlen=10)
y_train = df['TotalPValue']
...

So, the text input becomes an array of int tokens padded with zeroes, e.g. [0 0 0 0 0 0 0 1 12 52].

This is not enough to solve my problem. I want to instead use multiple tokenized string inputs and floats as features. I want to first tokenize and pad each text input like above and put them in the same input array, like this: X_train = [[0 0 0 0 0 0 0 1 12 52], [0 0 0 0 0 0 0 42 12 23], 0.0425672].

I want to then start my model like this:

model = Sequential()
model.add(Embedding(max_features, embedding_vector_length, input_length=3))

Will it work if implemented like this?

My attempts
I searched for a while but couldn't find anyone else doing it like this. Surprising to me that I couldn't find anything since it seems like a basic problem.

Just wanted to know if I have the right idea, since - as a beginner -  implementation will cost a lot of time if this isn't the right way of doing it. Thanks so much for the insight!


texts: difference in between CNN and Inception v3

texts: What is the difference in between the inception v3 and Convolutional neural network?


texts: Linear SVM with slack variables: Will it find a perfect decision boundary if it exists?

texts:
Suppose I use a linear Support Vector Machine with slack variables on a dataset that is linearly separable.
Could it happen that the Support Vector Machine reports a solution that does not perfectly separate the classes?

As an illustration: Is the situation in the picture possible for a Support Vector Machine with slack variables? Although there is a "better" boundary that allows perfect classification, the Support Vector Machine goes for a sub-optimal solution that misclassifies two samples.


texts: Finding Outliers in Resource Forecast Data

texts:
Unsure if this is the correct place to place this, please close if so.

I'm a workforce analyst at a large retail company, I own and maintain all the forecasting for our retail stores. This is based on Product Sales Forecast which I run through a model which has various tasks (such as changing items on a shelve or selling a car alarm for example) binned into timed categories, e.g

Item A takes 30 seconds, so 500 product forecasts will allow you (0.5 * 500) or 4.1 hours which are then further binned into Weekly allocations.

Now, as, for each iteration of this model I've run, there are always outliers (read mistakes), such as Store Y has no product volume to Product Group X in Week 22, which is a mistake from the finance team.

My senior has said this has always been the case for the 8 or so years he's been with the company.

Now, in my head, I assume I would be able to find outliers based on historical data using some sort of statistical method, however, I have no idea where to start, my data is as follows and is around 10 million rows of data.

import pandas as pd
import numpy as np

data =  [19,21,24,18,3]
pg = ['PG','ZF','AA','GG','ZF']
location = ['AA_1','AA_1','AA_2','AA_2','AA_2']
weeks = [1,1,2,2,2]

df = pd.DataFrame({'Location' : location,
            'productGroup' : pg,
            'Week' : weeks,
            'productVolumes' : data })
print(df)

  Location  productGroup    Week    productVolumes
0   AA_1    PG  1   19
1   AA_1    ZF  1   21
2   AA_2    AA  2   24
3   AA_2    GG  2   18
4   AA_2    ZF  2   3

in your expert opinions, what would be the best method to use?


texts:
I've built an artificial neural network in python using the scipy.optimize.minimize (Conjugate gradient) optimization function.

I've implemented gradient checking, double checked everything etc and I'm pretty certain it's working correctly.

I've run it a few times and it reaches 'Optimization terminated successfully' however when I increase the number of hidden layers, the cost of the hypothesis increases (everything else is kept the same) after it has successfully terminated.

Intuitively it feels as if the cost should decrease when the number of hidden layers is increased, as it is able to generate a more complex hypothesis which can fit the data better, however this appears not to be the case.

I'd be interested to understand what's going on here, or if I've implemented neural net incorrectly?


texts: What is affine transformation in regards to Neural Networks?

texts:
I have been reading a paper recently on Highway Neural Networks and found the following:

$y=H(x,W_H)$
$H$ is usually an affine transform followed by a non-linear activation function, but in general it may take other forms.

After googling about affine transform I can't say I understand fully what it means. Can somebody please elaborate?


texts: Why am I getting misaligned values while plotting heatmaps using seaborn?

texts:
While plotting heatmaps using seaborn, the numbers are getting printed on the boundary of the heatmap (Please see the image attached). I have already plotted heatmaps many times using the same code. It used to work well but since a few days, I'm facing this issue of misalignment.

Code:

#Defining a custom function
def get_confusion_matrix(clf,X_te,y_test):
y_pred = clf.predict(X_te)
df_cm = pd.DataFrame(confusion_matrix(y_test, y_pred), range(2),range(2))
df_cm.columns = ['Predicted NO','Predicted YES']
df_cm = df_cm.rename({0: 'Actual NO', 1: 'Actual YES'})
sns.set(font_scale=1.4)#for label size
sns.heatmap(df_cm, annot=True,annot_kws={"size": 16}, fmt='d',cmap="YlGnBu", linewidths=5)

#Using the function to plot the heatmap
get_confusion_matrix(model_s1,X_tr,y_train_1)

Kindly help.


texts: WEKA Random Forest J48 Attribute Importance

texts:
I have been using WEKA to classify very long duration audio recordings. The best performing classifiers have been Random Forest and J48. The attributes used to classify the audio are acoustic indices. This process of generating these indices is quite resource intensive.

I would like to determine the importance of the various attributes. Is there a way for these classifiers to report this?

I see J48 produces a decision tree, is it safe to day the attributes used at the root of the tree are most important?


texts: Time-series data pipeline for Kafka + Spark?

texts: I'm working on a project related to predicting syscall sequences' malice. The data, which are syscall sequences that each belongs to a program,  will be stored temporary to Kafka (which I have no idea which form it will be stored), and are processed by Spark (currently I tend to use sparkdl to apply deep learning solutions on Spark). I'm stuck at the process of creating pipeline to load and process data from Kafka to Spark. Any suggestion on any aspect of this project is appreciated!


texts: feature extraction for a pretrained model in keras

texts:
Keras has a way to extract the features of a pretrained model, described here https://keras.io/applications/

from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
import numpy as np

model = VGG16(weights='imagenet', include_top=False)

img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

features = model.predict(x)

I tried this with some sample images. My vector x is of shape (100, 3, 224, 224) for 100 observations, 3 for RGB and 224x224 pixel size.
the preprocess_input reshapes this for the VGG model (it expects a different order).

However, the output shape of features is (100, 512, 7, 7). What is this shape?
I want to use the features as input for a logistic regression. So I need a shape like (100, n): one row for each observation and the features in the columns.
How do I reshape the output to this dimension?

Say I now want to build my own simple Convnet:

from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(1, 299, 299)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

This model expects grayscale images as input, hence the shape.

What kind of layer do I have to add to get features of this model (something I can input in a logistic regression or random forest).

Thanks