texts: Model comparison with CV using standard error

texts:
Discovering the ML world with sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model to be as accurate as possible, while I can progress in my understanding of ML.

I've separated my dataset (16k rows) into 80% training and 20% testing, and I'm testing at least KNN, Logistic, DecisionTree, RandomForest, NaivesBayes and maybe SVC (if my computer can handle it), with some Bagging and Boosting when I find out how to.

My training sample comes in 4 ways: 2 sets of features (95 and 11), standartised (with a StandardScaler) or not.

My outcome is binary and I'm using a custom scorer "amelioration" which maximalise the number of positives for the 30th percentile (easier to get it with the code at the end of my post), along with specificity and roc_auc.

For each dataset, I cross validate (stratified and shuffled with a random state, 5 folds, with repetition when possible) all hyperparameters I find relevant and repeat this for each training sample.

For each crossvalidation, I refit using my scorer so I can compare the results amongst models and datasets. For models I've tested so far, my mean_test_amelioration range from +42% to +114%.

Finally, I'll measure the performances of the selected model on the testing sample and report results.

I guess this can feel quite cumbersome to a pro (and I'm probably building a tank to kill a fly), but I've already learned so much in this manner.

I'm only comparing all these models on the mean_test_amelioration and do not take into account standard error (for instance). Could this lead to overfitting so that my final model won't generalize well? If yes, how could I take variability into account ?

Any educationnal link is also very welcome.

PS: As this could be relevant, here is my custom scorer code:

def get_amelioration(y_true, y_pred, **kwargs):
    """
       If I select 30% of my sample with this algorithm, I will have
       `amelioration`% more positives in my selection than without

       :use as: make_scorer(get_amelioration, needs_proba=True, N=30)
    """

    N = kwargs.pop('N', False)
    if kwargs: raise TypeError('Unexpected **kwargs: %r' % kwargs)

    decisions = (y_pred > np.percentile(y_pred, 100-N)).astype(int)

    tn, fp, fn, tp = metrics.confusion_matrix(y_true, decisions).ravel()
    v = (fp+tp)/(tn+fp+fn+tp)
    r = tp/(fp+tp)
    r_base = np.mean(y_true) #around 15% in my sample, expected to be stable
    amelioration = 100*(r/r_base-1)
#    print("N=%i, v=%0.3f, amelioration=%0.3f" %(N,v, amelioration))
    if v<0.75*N/100: return 0
    return amelioration


texts: Deep Learning for high contrast images with small differences

texts:
I am trying to make a deep learning classification on the gear part you can see in the images. The contrast is high and almost binary. I want to classify the images where the circle located closest to the centre circle is left of the circle located furthest from the centre circle as the good part.

When this is mirrored (the circle located closest to the centre circle is right of the circle located furthest from the centre circle) the part needs to be classified as bad.

The parts can be rotated over a range of 360 degrees. The parts are found beforehand and cropped with the centre of the part as the centre of the image, which makes translation not a problem. The parts do not vary in scale.

I am using Keras with TensorFlow as Backend. I have 3600 input images, with 1800 good and 1800 bad images, where the parts are rotated. I have build some simple CNNs with three convolutional layers up to the VGG16 model, without getting past 0.5 accuracy.

My knowledge on different kind of models is limited and therefore I don't know which arcitecture would suit my problem best. Can anyone help me with some guidance on the kind of model I could use or to tell me if this is at all possible like this?

If more information is needed please let me know.


texts:
When I train my model the loss increases over each epoch. I feel like this is a simple solve and I am missing something obvious but I cannot figure out what is it. Any help would be greatly appreciated.

The neural network:

def neural_network(data):
    hidden_L1 = {'weights': tf.Variable(tf.random_normal([784, neurons_L1])),
                'biases': tf.Variable(tf.random_normal([neurons_L1]))}

    hidden_L2 = {'weights': tf.Variable(tf.random_normal([neurons_L1, neurons_L2])),
                'biases': tf.Variable(tf.random_normal([neurons_L2]))}

    output_L = {'weights': tf.Variable(tf.random_normal([neurons_L2, num_of_classes])),
                'biases': tf.Variable(tf.random_normal([num_of_classes]))}

    L1 = tf.add(tf.matmul(data, hidden_L1['weights']), hidden_L1['biases']) #matrix multiplication
    L1 = tf.nn.relu(L1)

    L2 = tf.add(tf.matmul(L1, hidden_L2['weights']), hidden_L2['biases']) #matrix multiplication
    L2 = tf.nn.relu(L2)

    output = tf.add(tf.matmul(L2, output_L['weights']), output_L['biases']) #matrix multiplication
    output = tf.nn.softmax(output)

    return output

My loss, optimiser and loop for each epoch:

output = neural_network(x)
loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y) )
optimiser = tf.train.AdamOptimizer().minimize(loss)

init = tf.global_variables_initializer()

epochs = 5
total_batch_count = 60000//batch_size

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(epochs):

        avg_loss = 0

        for i in range(total_batch_count):

            batch_x, batch_y = next_batch(batch_size, x_train, y_train)

            _, c = sess.run([optimiser, loss], feed_dict = {x:batch_x, y:batch_y})

            avg_loss +=c/total_batch_count

            print("epoch = ", epoch + 1, "loss =", avg_loss)

    sess.close()

I have a feeling my problems lies in the either the loss function or the loop I wrote for each epoch, however I am new to TensorFlow and cannot figure this out.


texts: Building a Keras text embedding model with cosine proximity

texts:
I am trying to build a word embedding keras model wherein I give as input a text that is converted to its corresponding input ids and masks (like input to an Albert model) and it gives me back a 768 dimensional vector as output. What I am planning to do here is use an Albert layer followed by a LSTM layer and a dense layer to give back a vector. As for target variables, there are 768 dimensional vectors. I want to use something like a cosine proximity function in order to determine the weights of the model. My general architecture is like this. However after training the model for some epochs, when I test the model, I get almost the same output vectors for all inputs. Is there something I need to change to make the model work?
        max_seq_length = 400
        in_id = Input(shape=(max_seq_length,), name="input_ids")
        in_mask = Input(shape=(max_seq_length,), name="input_masks")
        in_segment = Input(shape=(max_seq_length,), name="segment_ids")

        albert_inputs = [in_id, in_mask, in_segment]
        albert_output = AlbertLayer(n_fine_tune_layers=3, pooling="first")(albert_inputs)
        x = RepeatVector(1)(albert_output)
        x = LSTM(units=512, return_sequences=False,
                 recurrent_dropout=0.3, dropout=0.3)(x)
        x = Flatten()(x)
        embedding_output = Dense(768)(x)

        model = Model(inputs=albert_inputs, outputs=embedding_output)
        model.compile(loss=cosine_proximity, optimizer='adam')

For the target variables, I have corresponding vectors for each training instance. For one input vector, its possible there might be multiple target vectors. I am using separate training instances for the input vector in this case.


texts: How to define a multi-dimensional neural network with keras

texts:
I have implemented a simple neural network with keras that takes an input of 50 values and returns a classification of '0' or '1'.  I believe the model is expecting an input shape of (50, 1).  I'd like to add another 50 data values for each input, but I'd like them to be associated with the original 50 respective inputs.  So instead of making the input of shape (100, 1), I guess I'd like to make it of shape (50, 2).  I would like the neural network to know from the start that each input feature has two values associated with it, instead of it thinking there are 100 separate input features.  Here's what I have so far:

model = Sequential()
model.add(Dense(50, input_dim=50, kernel_initializer='normal', activation='relu'))
model.add(Dense(100, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Can anyone show me the way the alter this structure to accept my new input shape?


texts:
I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to anonymize the original data first before implementing the machine learning algorithms. `The data anonymization includes:

changing the names of persons,
places,
geographical locations, etc.

I would like to know what are the best practices for anonymizing datasets? Ideally, I should be able to get the original data back after performing analysis on the anonymized dataset.

I went through the literature and looked over some answered questions. They all are based on cybersecurity aspects like encryption and decryption algorithms. I am not familiar with cybersecurity algorithms. Is there any way to slightly change the data without digging into cybersecurity algorithms?


texts: I am trying to build a resume parser (from PDF to JSON). After extracting text from a pdf as one long string, how would you split the string into different sections like the red lines show. Resumes have different formats and people use different labels for these sections. Is there any machine learning technique that I could look into? Thanks! .


texts: Odd results on bias-variance tradeoff assessment

texts:
I am running a bias-variance tradeoff assessment on ten regression models of increasing complexity (linear to x^10), but my results do not satisfy the MSE = Bias^2 + Variance + True Error relationship.
I noticed two oddities with results from my large actual dataset (true pattern is roughly cubic, with a flat left tail): as model complexity increases, (1) variance increases sharply then plateaus (2) bias is almost invariant. This does not look correct to me.

As it is unwieldy to upload the dataset I am working on, I use simulated data for reprex. I managed to recreate the bias issue, though the variance curve looks normal here. However, I am still unable to get the relationship:
# Generate 101 datasets, 100 obs each
# One test set, the rest are training sets.
library(tidyverse)
data_sets <- lapply(1:101, function(...) {
  tibble(
    X = runif(100, min = -pi / 2, max = pi / 2), # predictor
    y.true = X^3, # true outcome
    e = rnorm(100, mean = 0, sd = 3), # true error
    Y = y.true + e # observed outcome
  )
}) %>% {
  list(test_set = .[[1]],
       training_sets = .[-1])
}

# convert test_set to dataframe for predict() compatibility
testset <- as.data.frame(data_sets$test_set)

## Begin train/predict --------------

# Set necessary stuff
timesiter = 100 # iterations
polynom = 10 # model complexity

predict.df <- data.frame(matrix(NA,
                                nrow = nrow(testset)*timesiter,
                                ncol = polynom + 1)) # + 1 iteration col
colnames(predict.df) <- c(paste0("x",1:polynom), "iteration")
predict.df$iteration <- rep(1:timesiter, each = nrow(testset))

set.seed(696969)
for(p in 1:timesiter){

  # Take a new training set for training
  train.sample <- data_sets$training_sets[[p]]

  # Train models
  for (i in 1:polynom){
    lm.model <- lm(Y ~ poly(X, i), data = train.sample)
    # Get model predictions on testset & store it
    predict.df[predict.df$iteration == p, i] <- predict(lm.model, newdata = testset)

  }

}

## ----------------------------------
# Using manual calculations for understanding
# Also, I realise the for-loops are unwieldy

## VARIANCE for each model
var.df <- data.frame(matrix(NA,
                            nrow = 1,
                            ncol = polynom))
for (i in 1:polynom){
  var.df[, i] <- mean((predict.df[, i] - mean(predict.df[, i]))^2)
}
var.df

## BIAS^2
biasquare.df <- data.frame(matrix(NA,
                                  nrow = 1,
                                  ncol = polynom))
for (i in 1:polynom){
  biasquare.df[, i] <- mean( (mean(predict.df[, i]) - testset$y.true)^2 )
}
biasquare.df

## MSE (Mean Squared Error)
mse.df <- data.frame(matrix(NA,
                            nrow = 1,
                            ncol = polynom))
for (i in 1:polynom){
  mse.df[, i] <- mean((testset$Y - predict.df[, i])^2)
}
mse.df

## TRUE ERROR
true_error <- mean(((testset$y.true - testset$Y)^2))

# Check if MSE = Bias^2 + Variance + Error (they are not)
mse.df - (var.df + biasquare.df + true_error)

The plot for this looks like this:

Appreciate if anyone can point out my mistakes. Thanks!


texts: RL Policy Gradient: How to deal with rewards that are strictly positive?

texts:
In short:

In the policy gradient method, if the reward is always positive (never negative), the policy gradient will always be positive, hence it will keep making our parameters larger. This makes the learning algorithm meaningless. How do we get around this problem?

In detail:

In "RL Course by David Silver" lecture 7 (on YouTube), he introduced the REINFORCE algorithm for policy gradient (here just showing 1 step):

The actual policy update is:

Note that $v_t$ here stands for the reward we get. Let's say we're playing a game where the reward is always positive (eg. accumulating a score), and there are never any negative rewards, the gradient will always be positive, hence $\theta$ will keep increasing! So how do we deal with rewards that never change sign?


texts: deriving the gradient of batch normalization

texts:
I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'.

I present to you the triangle of death (in red), in the context of a computation graph for the batch norm equation:

The problem is that applying chain rule to the triangle results in 1 - 1 = 0, which kills the gradient.

My guess is I am messing up one or more of:

the derivatives.
the relationships I think the graph implies with respect to the nodes.
the graph representation I chose for BN as a composition of functions.

but I'm not sure where my error(s) is/are.


texts:
I'm working with sklearn SVM and I have a problem.

When I run the method sklearn.SVM.SVC.fit() using a database with only a few features (< 10) it takes a very long time. This is weird because, when I run the same method with the same database using all of the features (> 100) it takes just a few seconds.

I'm using a polynomial kernel and this problem only appears when the degree is >= 3. And it is also weird that if I use only a few features, but I add 50 features with 0s in all the samples, it works fine!

I have tried the following experiments:

fit() with all the features and any kernel degree. Works fine.
fit() with a few features and kernel degree <= 2. Works fine.
fit() with a few features and kernel degree >= 3. Very slow.
fit() with kernel degree = 3, a few features but adding 50 features with 0s in all the samples. Works fine!!

My question is, how can I solve this? Does my trick of adding features with 0s affect the classification?


texts: What does the one function $\mathbf{1}_{i,y^{(t)}}$ exactly mean in backward propagation of RNN in the book "Deep learning" of Bengio

texts:
It confused me for a long time what is $\mathbf{1}_{i,y^{(t)}}$ exactly mean in (10.18) below.

It is in the Chapter 10 on RNN of the book LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436.

Because the ${i,y^{(t)}}$ doesn't seem to be a condition while the author has introduced a condition function as $\mathbf{1}_\text{condition} = 1$ if condition is true or $=0$ if the condition is false.

It is computing the gradient of the log-likelihood with backward propagation in recurrent neural network.

Instead of $\hat{y}_i^{(t)}− \mathbf{1}_{i,y^{(t)}}$, I found my solution might be
$$
(\hat{y}^{(t)}_i-y_i^{(t)})\dfrac{\exp(o_i^{(t)})\sum_j\exp(z_j)-(\exp(z_i))^2}{(\sum_j\exp(z_j))^2}
$$
It is based on the prior assumption of Gaussian distribution with $\sigma=1$

Below is the equation I am asking about:

  The gradient $\nabla_{o^{(t)}}L$ on the outputs at time step $t$, for all $i$, $t$, is as
  follows:
  $$
(\nabla_{o(t)}L)_i
= \dfrac{\partial L}{\partial o_i^{(t)}}
= \dfrac{\partial L}{\partial L^{(t)}}\dfrac{\partial L^{(t)}}{\partial o_i^{(t)}}
= \hat{y}_i^{(t)}− \mathbf{1}_{i,y^{(t)}}.   \qquad\qquad (10.18)
$$

While, the corresponding notes are:

  $$
\begin{align*}
\boldsymbol{a}^{(t)} &= \boldsymbol{b} +\boldsymbol{Wh}^{(t−1)} + \boldsymbol{Ux}^{(t)} &(10.8)\\
\boldsymbol{h}^{(t)} &= \tanh(\boldsymbol{a}^{(t)}) &(10.9)\\
\boldsymbol{o}^{(t)} &= \boldsymbol{c} + \boldsymbol{V h}^{(t)} &(10.10)\\
\hat{\boldsymbol{y}}^{(t)} &= \text{softmax}(\boldsymbol{o}^{(t)}) &(10.11)
\end{align*}
$$

And I wonder which version of softmax function is used here?
$$
\text{softmax}(\boldsymbol{z})=\dfrac{\exp(z_i)}{\sum_j\exp(z_j)}
$$
or
$$
\text{softmax}(\boldsymbol{z})=\dfrac{\exp(z_i-\max_kz_k)}{\sum_j\exp(z_j-\max_kz_k)}
$$
?

Thank you very much!


texts: KL Divergence between Predictions and Ground truth

texts:
I've got four (non-linear, tree-based) models in production and using the average of them as the served prediction. We get ground truth data immediately.
During training the optimized candidate models had very similar performance, so I decided to deploy all of them and take the average and served that as the prediction. With the intention of figuring which one would really be best at a later point.
That later point has come.
Out of these four models two seem fit the ground truth distribution quite well, at least by examining the KDE plot of the predicted values against the ground truth distribution
I was initially thinking of doing a pairwise 2-sample KS Test (each model against ground truth) to see if the predicted values from the models come from the same distribution as the ground truth.
But I talked myself out of it. Mostly because I have 50,000+ predictions and I figured that a sample size that large will result in a small p-value anyway.
I then turned my attention to KL divergence. Something I have never used before.
Would comparing the KL-Divergence between each model's prediction and the ground truth be a good way to assess model fit? Can I compare just the raw values or does it need to be converted to a probability?
If so, how would I go about doing this? scipy.stats.kl_div outputs an array which contains (I assume) the divergence between the prediction and ground truth.
Would I just take the sum of the array and call that the Divergence between the model and ground truth?
Hopefully I don't sound crazy or too much like I don't know what I'm doing or talking about. Because I kinda don't, but not too much.
Thanks


texts:
I need to do coreference resolution for German texts and I plan to use OpenNLP to perform this task.

As far as I know OpenNLP coreference resolution does not support the German language.

Which components/data do I need to adapt the code such that it is possible to perform coreference resolution for German texts?


texts:
I am trying to use transfer learning in medical (ultrasound pictures). The problem is - I have very limited picture database = 400 (360+40). I am using resnet50 (I don't think this is important but maybe I'm wrong). Resnet as feature extractor + SVM is not great but normalized confusion matrix is somewhat about:

1.0  0
0.4 0.6

Now, I wanted to fine-tune resnet. And the problem is that CM at the beginning looks like:

0.8  0.2
0.6 0.4

is something like this:

1.0  0
0.8 0.2

Below you can see training + test loss/accuracy.

Now I thought it is overfitting (due to too large rate capacity / database) but someone pointed that network might not be learning. What is the case?


texts:
I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python.

What are people using in academia and industry, and what is the recommendation?


texts: Algorithm for rectifying incorrect text from OCR

texts:
I have an OCR (tesseract) engine running on processing "chyron" text from a live stream news feed.
However, the OCR is often inaccurate and ends up with some noisy words:
- Supreme court says states can punish faithless euxsx898
- Supreme 5$^* says states can sdsd sdsd electors
- Supreme court sdsds sdsds

the noisy text inside is an artifact of the OCR process and varies how much of it turns out to be correct. How can I reconcile or merge these detections into one headline:
Supreme court says states can punish faithless electors

Are there some accepted approaches or algorithms for such problems?


texts: What tools are available for semi-automated matching of dirty columnar data

texts:
Are there any automated or semi-automated tools for finding matching "similar" or data in two columnar data sets?  The data I'm working with was collected (and handled) by different organizations.  Some rows describe the same events and even carry the same unique identifier (about 70% of the smaller data set can be matched using the unique ids).  For the unmatched cases, there are slight differences such as typos in the unique id, date differences (+/- 1-2 days), and slight differences in value variables... 997.5 v. 1000, etc.

If, by inspection, the value variables, date, and location are close, I call it a match.  Sometimes it is even more obvious such when a one or more digits in the unique ID digit have been transposed or the wrong one has been duplicated.

Thus far, I've been doing this by hand.  I've been using R for data cleaning and Excel for comparison and matching (I export comparable subsets, make a crosswalk of old-new unique IDs by hand, and re-import the crosswalk).

What I'd really like is a tool that automates this process by presenting me with information such as: "record 1 in data set A is really similar (for the variables that matter) to records 10, 20, and 25 in data set B."  Then I could quickly examine the differences, make a choice (say record 20), and then move on.

I'd like it to feel very similar to the way Ancestry.com helps you match up birth, death, marriage, and census records.  But, anything that simplifies and automates this process would be welcome.

Thoughts?


texts: Keras: How to connect a CNN model with a decision tree

texts:
I want to train a model to predict one's emotion from the physical signals. I have a physical signal and using it as input feature;

  ecg(Electrocardiography)

I want to use the CNN architecture to extract features from the data, and then use these extracted features to feed a classical "Decision Tree Classifier". Below, you can see my CNN aproach without the decision tree;

model = Sequential()
model.add(Conv1D(15,60,padding='valid', activation='relu',input_shape=(18000,1), strides = 1,  kernel_regularizer=regularizers.l1_l2(l1=0.1, l2=0.1)))
model.add(MaxPooling1D(2,data_format='channels_last'))
model.add(Dropout(0.6))
model.add(BatchNormalization())
model.add(Conv1D(30, 60, padding='valid', activation='relu',kernel_regularizer = regularizers.l1_l2(l1=0.1, l2=0.1), strides=1))
model.add(MaxPooling1D(4,data_format='channels_last'))
model.add(Dropout(0.6))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dense(3, activation = 'softmax'))

I want to edit this code so that, in the output layer there will be working decision tree instead of model.add(Dense(3, activation = 'softmax')). I have tried to save the outputs of the last convolutional layer like this;

output = model.layers[-6].output

And when I printed out the output variable, result was this;

  THE OUTPUT:  Tensor("conv1d_56/Relu:0", shape=(?, 8971, 30),
  dtype=float32)

I guess, the output variable holds the extracted features. Now, how can I feed my decision tree classifier model  with this data which is stored in the output variable? Here is the decision tree from scikit learn;

from sklearn.tree import DecisionTreeClassifier

dtc = DecisionTreeClassifier(criterion = 'entropy')
dtc.fit()

How should I feed the fit() method? Thanks in advance.


texts: Which is the fastest image pretrained model?

texts:
I had been working with pre-trained models and was just curious to know the fastest forward propagating model of all the computer vision pre-trained models. I have been trying to achieve faster processing in one-shot learning and have tried the forward propagation with few models over a single image and the results are as follows:

VGG16: 4.857 seconds
ResNet50: 0.227 seconds
Inception: 0.135 seconds

Can you tell the fastest pre-trained model available out there and the drastic time consumption difference amongst the above-mentioned models.


texts: state transition classification on terminal state

texts:
I have data on a unit $i$ which enters an entry state $S_0$.
This unit has some covariates $x_i$
I would like to predict the probability the unit will reach the terminal state $S_{pos}$ or $S_{neg}$. The unit can spend time on each state, which affects the probabilities as well. In some time the unit can jump to an intermediate state $S_j$ which is not terminal which also changes the probability to get to one of the terminal states.
I have "dimension" data of the unit $x_i$ in one table and the state transition and time in a "fact" table.
I would like to build a model $M(x, s, t)$ which will predict probability to get to $S_{pos}$ given the covariate $x$ the state $s$ and the time spent in that state $t$.
My idea was to replicate the data so that each unit $i$ I will create $T$ rows, one for each of the timestamps it lived until it was terminated and feed this to a general-purpose classifier.
The question is, is this redundant? Is there some representation of the data where I will not have to duplicate rows? Maybe feed this to an RNN? but there I will still need to feed a "nothing happened" time stamp every day until a state has changed.
How would you model this?
Survival analysis seems less relevant as I do not want to predict the duration, just the terminal state.


texts: Random Forest Model Giving Same Accuracy for different feature sets after tuning

texts:
I am having this weird issue and cannot seem to find a solution. I am trying to tune a different random forest model for every different feature-set. Basically from a given data set, I have created 3 different feature sets. In first case you have all features, in second case you have slightly less number of features, and finally for the 3rd feature set you just have 5 features.
I am trying to tune the model with RandomizedSearchCV.

param_grid = {'n_estimators' : range(1,100),
'max_depth' : np.linspace(1, 50, 5, endpoint=True),
'min_samples_split' : np.linspace(0.1, 1.0, 10, endpoint=True),
'min_samples_leaf' : np.linspace(0.1, 0.5, 5, endpoint=True),
'max_features' : range(1,X.shape[1]),
"criterion": ["gini","entropy"],}

grid = RandomizedSearchCV(clf_rf, param_distributions = param_grid, cv = 5, scoring = 'roc_auc', verbose=True, n_jobs=-1, random_state=1, return_train_score=False)
grid.fit(X,y)

Next, I use the grid.best_estimator_, the best model found by RandomizedSearchCV,   to find  to find the cross validated score.

estimator_cv = grid.best_estimator_
cv_results = cross_validate(estimator_cv, X, y, cv=10)
cv_accuracy = cv_results['test_score'].mean()
cv_accuracy

The issue I am facing is that after following the above steps for the the 3 different feature-sets, I am getting the same cross validated accuracy score of 0.8873372903702229. The roc_auc however is changing.


texts: How do I implement stochastic gradient descent correctly?

texts:
I'm trying to implement stochastic gradient descent in MATLAB however I am not seeing any convergence. Mini-batch gradient descent worked as expected so I think that the cost function and gradient steps are correct.

The two main issues I am having are:

Randomly shuffling the data in the training set before the
for-loop
Selecting one example at a time

Here is my MATLAB code:

Generating Data

alpha = 0.001;
num_iters = 10;

xrange =(-10:0.1:10); % data lenght
ydata  = 5*(xrange)+30; % data with gradient 2, intercept 5

% plot(xrange,ydata); grid on;
noise  = (2*randn(1,length(xrange))); % generating noise
target = ydata + noise; % adding noise to data

f1 = figure
subplot(2,2,1);
scatter(xrange,target); grid on; hold on; % plot a scttaer
title('Linear Regression')
xlabel('xrange')
ylabel('ydata')

tita0 = randn(1,1); %intercept (randomised)
tita1 = randn(1,1); %gradient  (randomised)

% Initialize Objective Function History
J_history = zeros(num_iters, 1);

% Number of training examples
m = (length(xrange));

Shuffling data, Gradient Descent and Cost Function

% STEP1 : we shuffle the data
data = [ xrange, ydata];
data = data(randperm(size(data,1)),:);
y = data(:,1);
X = data(:,2:end);

for iter = 1:num_iters

    for i = 1:m

        x = X(:,i); % STEP2 Select one example

        h = tita0 + tita1.*x; % building the estimated     %Changed to xrange in BGD

        %c = (1/(2*length(xrange)))*sum((h-target).^2)

        temp0 = tita0 - alpha*((1/m)*sum((h-target)));
        temp1 = tita1 - alpha*((1/m)*sum((h-target).*x));  %Changed to xrange in BGD
        tita0 = temp0;
        tita1 = temp1;

        fprintf("here\n %d; %d", i, x)

    end

        J_history(iter) = (1/(2*m))*sum((h-target).^2); % Calculating cost from data to estimate

        fprintf('Iteration #%d - Cost = %d... \r\n',iter, J_history(iter));

end

On plotting the cost vs iterations and linear regression graphs, the MSE settles (local minimum?) at around 420 which is wrong.

On the other hand if I re-run the exact same code however using batch gradient descent I get acceptable results. In batch gradient descent I am changing x to xrange:

Any suggestions on what I am doing wrong?

EDIT:

I also tried selecting random indexes using:

f = round(1+rand(1,1)*201);        %generating random indexes

and then selecting one example:

x = xrange(f); % STEP2 Select one example

Proceeding to use x in the hypothesis and GD steps also yield a cost of 420.


texts:
Have you heard of the "Data Science Association"?
URL: http://www.datascienceassn.org/
Do you expect it to become a professional body like the Actuaries Institute?
If yes, then why?
If no, then why not and do you see anyone else becoming the professional body?

Lastly, is this question "on-topic" ?


texts: Comparison between DBSCAN and single linkage hierarchical clustering

texts: I am studying clustering algorithms and I want to find a good example where single link hierarchical clustering algorithm returns better cluster results than DBSCAN (having provided valid parameters). Is there a good dataset either real world or generated?


texts: Retaining past learning with Incremental Learning

texts:
I have built an ML model with the goal of making predictions for targets of the following week. In general, new data will come in and be processed at the end of each week and be in the same data structure as before.

Instead of re-training the model from scratch for each week's predictions, I am considering applying an incremental learning approach so that past learning is not entirely discarded and the model would (presumably) increase in performance over time. I'm working with sklearn, keras on Python 3.

Couple of questions:

Suppose we performed incremental learning on Dataset 1 and subsequently Dataset 2. If both datasets were actually available, would the performance be exactly the same as if the model was trained on both datasets without any incremental learning? (Assuming no data shuffling)
Are there any advantages to dedicated incremental learning packages such as creme instead of simply saving/loading the models and working with sklearn?
As time increases, there may be drift or pattern changes in the data (i.e. some features may not be as predictive, while others become more so). Is it possible to force the incremental learning model to more heavily weigh the current dataset while discounting some of the learning from the past?
The documentation here is a bit confusing for the actual implementation. When warm_start=True, calling .fit() changes model parameters but data should be "more-or-less constant", whereas calling .partial_fit() keeps model parameters fixed with data being different. My goal is for both model parameters to change (while retaining some past learning) and fitting on new datasets - which one should I be calling?


texts: What are the simplest methods for the label noise problem?

texts:
If I have enough low quality data from unsupervised methods or rule-based methods.

I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex for me.

In detail, I deal with a multi-label classification task. First I crawl web page such as wiki and use regex-based rule to mark the label. The model input is the wiki title and the model output is the rule-matched labels from wiki content. My task is to predict the labels for the wiki title.

Do you think removing the wrong data predicted by trained model is a simple but effective method?


texts: Classification on time series items - choose not constant threshold

texts:
I'm dealing with a classification problem on a time series {time: t, value: y(t)} where, for each time t,
my classification algorithm gives as ouput the probability that the y(t) belongs to class 0 or 1. It looks like a constant threshold, used for the whole time-series length, gives bad results. Instead, from a graphical inspection it looks that lowering the threshold in a very slow way would lead to better results.

The idea is, given the real value of the time series y(t), to lower the threshold in a real-time way after some false positives happen, and to increase it after some of false negative happen.

Does anything like this exist in the literature?

Thanks!


texts: When can AUC and accuracy rate be equal?

texts:
The terms AUC and accuracy has been confusing for me.
In which circumstance AUC rate and accuracy rate become exactly equal?


texts:
I'm having trouble finding a good reward function for the pendulum problem, the function I'm using: $-x^2 - 0.25*(\text{xdot}^2)$
which is the quadratic error from the top. with $x$ representing the current location of the pendulum and $\text{xdot}$ the angular velocity.

It takes a lot of time with this function and sometimes doesn't work.
Any one have some other suggestions?
I've been looking in google but didn't find anything i could use


texts: When is z-normalization not needed when using DTW?

texts:
I'm hoping to get some answers to a question I have regarding normalization of DTW datasets, in particular datasets in which two time-series shapes with similar shapes but differences in magnitude are misclassified under z-normalization.

One reason is simply empirical. When I z-normalize my data it doesn't cluster well. When I don't z-normalize it works quite well. But that isn't really enough to ignore the normalization advice.

The second is in reading the links in this thread which does a good job summarizing the other threads related to this, reading the literature, and thinking about my data. I cannot find a good discussion of the case where magnitude matters within a subset of similar clusters and z-normalization negates that important magnitude difference. In cases where the shape of some data can generally be the same (the lengths are the same, but the amplitude differs) z-normalization makes them cluster as identical when they aren't. Does this mean that DTW just isn't the right tool, that I should be normalizing differently, or is my data a special case because it is already normalized the way it is collected?

Briefly, my data is isotopic ratios recorded in fish "ear stones" which change as a fish moves from one location to another and are recorded in the growing layers of the stone (called an otolith). The isotope ratio in each river is very stable, and that ratio is known to the third or fourth decimal place. There is no biological fractionation, the information is recorded as it exists in the river when the fish enters the river. All data is normalized at collection with regard to the global ocean value which is known to the 5th decimal place. What we care about is classifying the data according to the shape of the isotopic curve, which is characteristic of where and when fish moved between rivers, the different decisions fish make in their movements. All the curves are interpolated to the same length prior to analysis.

The problem I see in my data is illustrated by the conceptual plot below.

The warping of DTW helps to better classify fish with mild differences in their movement timing,  like the two black lines in the plot. It also does a good job of classifying patterns with really different shapes (the red line). Without z-normalization, DTW correctly classifies the blue line as a different group. But, using z-normalization, the two black lines classify with the blue line because the magnitude difference is erased.

To take the example from Section 1.2.1 of Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping from the linked question above, the movement of fish in two different habitats is analogous to the action of drawing a gun or pointing a finger. The reason for z-normalization in the gun-point dataset, according to the author, is to account for offsets in scaling of the video. But in this case all of the data is normalized already to the global marine signature to account for machine drift, and the isotopic values are extremely precise and accurate. The error in amplitude of the curves is less than 1%. Is z-normalization required here, or is it just erasing meaningful differences in amplitude that are correctly classified without z-normalization?

I'm curious not just to find an answer regarding my own data, but also because it has been really difficult for me to find a straightforward discussion of this type of amplitude problem in the literature. Most of the advice is a blanket, "In 99% of cases, you must z-normalize." as eamonn stated in the other question...an opinion I respect given his work on this subject. But, I'm curious about what those 1% of examples are. Perhaps I'm just missing it, or perhaps I'm missing the boat completely. Your thoughts, and explanations, would be greatly appreciated.


texts:
I'm currently building a binary classifier.
My input is a sequence of 32 time-steps.
Certain time-steps of the input will be constant (ex: t-0 will always be 0, t-5 will always be 9, etc)
Does it make sense to add these time-steps as features into the model? Im thinking its not since: The model will have to pay attention to these features and they will add a type of noise/bias into the model - since there isnt any new information to be gained from them. Am i thinking about this correctly?


texts: Kmeans clustering with multiple columns containing strings

texts:
I have the following dataset:
https://www.kaggle.com/carolzhangdc/imdb-5000-movie-dataset
What I want to find is clusters based on imdb score per genre per country. I have created a pandas data frame that contains per country for every unique genre the average imdb rating.
The dataframe looks like this:
country       object
genre         object
avgRating    float64
dtype: object

Since the columns country and genre contain strings, I can't use Kmeans for this.
Is there anyway I can achieve what I want?


texts: Is there any solution for modified t-test in python for spatial data?

texts:
I am searching for a solution to conduct a modified t-test or any other possibility in order to compute significant or non-significant relations in a spatial dataset (which can be a stack of rasters or a larger list of (irr-)regular points with several attached attributes).

The problem of faulty t-test is for example described in
Dutilleul, P. 1993a. Modifying the t test for assessing the correlation between two spatial processes. Biometrics 49:305-314.

The only thing I found is a solution in R.

The test was implemented in the software PASSAGE, which is not available anymore.

Kind regards


texts:
By default this script will run 4,000 training steps. Each step chooses ten images at random from the training set, finds their bottlenecks from the cache, and feeds them into the final layer to get predictions. Those predictions are then compared against the actual labels to update the final layer's weights through the back-propagation process.

According to this paragraph from this article from tensorflow , once the bottlenecks features are calculated for newer classes, backpropagation algorithm is used to retrain the last layer given the output labels. My question is, are the bottleneck features of the 10,000 classes of imagenet present while this retraining progresses - or the retraining is done for the newer class labels only.


texts: Lower training accuracy than testing accuracy (MLP/Dropout)

texts:
I am working on a problem of multi-class classification by MLP. I have set dropout to each middle layer. Now I observe the training accuracy is around 10% less than the testing accuracy.
My guess is, dropout is active only during training but inactive during testing. So part of the neurons are reset at training (leading to low accuracy), but it is not happening for testing.
My questions:

Is my understanding correct? In other words, if I remove the dropout part, will the training accuracy increase but the testing accuracy will remain the same?
When reporting the MLP accuracy, should I report training acuracy or testing accuracy?


texts:
I refer to the example given at the Keras website here:

from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np

data_dim = 16
timesteps = 8
num_classes = 10

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
       input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

# Generate dummy training data
x_train = np.random.random((1000, timesteps, data_dim))
y_train = np.random.random((1000, num_classes))

# Generate dummy validation data
x_val = np.random.random((100, timesteps, data_dim))
y_val = np.random.random((100, num_classes))

model.fit(x_train, y_train, batch_size=64, epochs=5, validation_data=(x_val, y_val))

For a real-world example, what should be y_train and y_val? Should they be the same as x_train and x_val respectively, since they come from the same sequence?

Also, how should I understand data_dim and num_classes?


texts: SVM SMOTE fit_resample() function runs forever with no result

texts:
Problem

fit_resample(X,y)

is taking too long to complete execution for 2million rows.

Dataset specifications

I have a labeled dataset about network features, where the X(features) and Y(labels) are of shape (2M, 24) and (2M,11) respectively.

i.e. there are over 2million rows in the dataset. there are 24 features, and 11 different classes/labels.

Both X and Y are numpy arrays of float dtype.

Motivation for using SVM SMOTE

Due to class imbalance, I realized SVM SMOTE is a good technique balance it, thereby, giving better classification.

Testing with smaller sub-datasets

To test the performance of my classifier, I started small. I made small subdatasets out of the big 2million row dataset.

It took the following code:-

%%time
sm = SVMSMOTE(random_state=42)
X_res, Y_res = sm.fit_resample(X, Y)

1st dataset contains only 7.5k rows. It took about 800ms to run the cell.
2nd dataset contains 115k rows. It took 20min to execute the cell.

Solution Attempts

My system crashes after running continuously for more than 48hrs, running out of memory.

I've tried some ideas, such as

1. splitting it to run on multiple CPU cores using %%px

No improvement in quicker execution

2. using NVIDIA GPU's

Same as above. Which is more understandable since the _smote.py library functions aren't built with parallel programming for CUDA in mind.

I'm pretty frustrated by the lack of results, and a warm PC. What should I do?


texts: Approaches to Bag-Of-Words Information Retrieval

texts:
I'm interested in an overview of the modern/state-of-the-art approaches to bag-of-words information retrieval, where you have a single query $q$ and a set of documents which you hope to rank by relevance $d_1,...,d_n$.

I'm specifically interested in approaches which require absolutely no linguistic knowledge and rely on no outside linguistic or lexical resources for boosting performance (such as thesauri or prebuilt word-nets and the like).  Thus where a ranking is produced entirely by evaluating query-document similarity and where the problems of synonymy and polysemy are overcome by exploiting inter-document word co-occurence.

I've spent some amount of time in the literature (reading papers/tutorials), however there is so much information out there it's hard to get a bird's eye view.  The best I can make out is that modern approaches involve some combination of a weighted vector space model (such as a generalized vector space model, LSI, or a topic-based vector space model using LDA), in conjunction with pseudo-relevance feedback (using either Rocchio or some more advanced approach).

All these vector space models tend to use cosine-similarity as the similarity function, however I have seen some literature discussing similarity functions which are more exotic.

Is this basically where we currently are in attacking this particular type of problem?


texts: Matching content item to a persons profile

texts:
I'm working on an experiment which is essentially a content recommendation service.

I have a set of content items in the form of articles, tweets, blog posts etc which have had a set of tags associated with them based on their contents.

I also have a set of user profiles with information on the users personality, interests, dislikes, activities etc.

Each user profile then has a column for liked and disliked tags to show which content items they would enjoy reading.

I would like to be able build a service that when passed a new user profile, returns a set of tags (generated from similar existing user profiles) that can be used to find content items that the new user would enjoy reading.

This is my first experiment with machine learning so I was wondering if someone could give me some advice on how to achieve this, or an approach to get started.

Many thanks in advanced!
Harry


texts: Binary Classification Comparing two time series of variable length

texts:
Is there a machine learning model (something like LSTM or 1D-CNN) that takes two time series of variable length as input and outputs a binary classification (True/False whether time series are of same label)?

So the data would look something like the following

date        value label
2020-01-01  2     0     # first input time series
2020-01-02  1     0     # first input time series
2020-01-03  1     0     # first input time series
2020-01-01  3     1     # second input time series
2020-01-03  1     1     # second input time series

Is there something like that available out of the box, and if not how would you build a minimal working example model in Keras?

My best guess is to use a shared LSTM layer for both inputs and Concatenate both resulting vectors before feeding to the final Dense layer.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

n_lstm_blocks = 50

input_1 = keras.Input(shape=(None, 1)) # unknown timespan, fixed feature size 1
input_2 = keras.Input(shape=(None, 1))
shared_lstm = layers.LSTM(n_lstm_blocks)
encode_1 = shared_lstm(input_1)
encode_2 = shared_lstm(input_2)
concat = layers.concatenate([encode_1,encode_2])
output = layers.Dense(1, activation='sigmoid')(concat)
model = keras.Model(inputs=[input_1,input_2],outputs=output)
model.compile(optimizer='adam', loss='binary_crossentropy')

A comparable task would be Siamese Networks / One-Shot learning which is used for face recognition. But in this case the task is to compare to time series and detect if they are of the same label, but knowing each label is NOT task of the network!


texts: NoSQL vs SQL backend for semi structured data

texts:
I have a corpus of job descriptions and another corpus of CVs of applicants. I plan to implement a matching system using machine learning algorithms, to find top 5 or top 10 applicants for each job description. Should I store the data in a document oriented NoSQL db (MongoDB) or stick to SQL.

Given that the data I have is semi-structured at best, I feel a NoSQL db will offer more flexibility. I would appreciate opinions on this.


texts: How to build a machine translation system for a new language

texts:
I am trying to figure out what are the options for building a natural language translation model for a language that is not yet supported by existing machine translations.
The project is to build a system for translating a very limited subset of a small east african language into English (only one-way needed).
The language is to my knowledge not yet supported by any machine translation systems, but it is related to several other big African languages (putting sentences into Google Translate, the language is mostly auto detected as Kiswahili or Shona, and the English translation is sometimes usable).

I know building a translator for a new language is by no means trivial, but the problem domain is very small and I think it should be doable. Are there any features of the big cloud providers that support this, ML frameworks, or vendors that build these models as a service?

If this is not the right stack exchange for this question, kindly guide me to a better place.


texts: Predicting labels which are not independent of one another

texts:
I want to make a multilabel classifier where the labels are dependent on one another. Concretely, I have a situation where:

If label A is true, then label B is also true.

If label B is true, label A could be true or false.

If label B is false, then label A is false.

How can I model this in a multilabel classifier?


texts:
KNN doesn't work well with high-variance data, so how should I fit my data?
Here is an example of what the data looks like:


texts: Postitive event but unsure when it occurred in time

texts:
Morning,

I have a lot of data of which I am positive an event (target data / the event I want to predict going forward) occurred within a two week time frame, however I am unsure when it happened within this time frame.

I can get daily or more frequent feature data, but the target data only appears in 2 week gaps or in some cases 4 week gaps.

Currently I use an average of any feature during the timeframe or gather the feature data on the same day as when the target data is available but is there a more appropriate / better way?

I am using the data for machine learning purposes.


texts: About building database for persian speech to text project

texts:
In connection with the persian speech to text project on github, I am trying to collect the database from the audio and text available on the ganjoor site and Youtube Persian videos with Persian subtitles (also, Maybe compare to others database via internet like this)
But since I am new to this fields, your help can be very It worked.
If it is possible, please help in how to properly save the audio and text by suggesting proper format for saving data and pre filtering for better sorting data like said here. (Any python sample codes, would be very helpful.)
Project github site (Forked):
https://bit.ly/3hRIb1V
Ganjoor scraper page (text and audio collection):
https://bit.ly/3kGlijQ
Question related ( Ganjoor scraper codes):
https://bit.ly/3cnR8zd
Thanks.


texts: Using class weights in Keras with multiple binary outputs which are not simply one-hot-encoded

texts:
My labels are binary vectors of length 5, e.g., [0, 0, 1, 1, 1].

My label set is very biased, 1-to-50, where the case [0, 0, 0, 0, 0] is very common while all other combinations are not. I'd like to weight the uncommon versions using the class_weight argument in the model.fit() function in Keras. I get an error message which says class_weight must contain all classes in the data.

  The classes {0, 1, 2, 3, 4} exist in the data but not in class_weight.

Based on this I suspect Keras expects that I only have classes of the form [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0], etc., i.e., only 1 in each entry in the vector. And that I'm supposed to supply the weights as follows:

weight_classes = {
    0: 50.0,        # for [1, 0, 0, 0, 0]
    1: 1.0,         # for [0, 1, 0, 0, 0]
    # etc.
}

Is there a way to tell Keras to give weights in this fashion:

weight_classes = {
    (0, 0, 0, 0, 0): 1.0,
    (0, 0, 1, 1, 1): 50.0,
    (0, 0, 0, 0, 1): 50.0,
    # etc.
}

Or is there some other way to just say that the all zero case is less important?

I've consider just transform the label set to conform to the one-hot-encoding type of representation. But I'd rather not, because the labels will become very large, and there are interdependence between the labels I think might be better represented by having them encoded as I've done. They are interdependent similar to how words in a sentence are dependent, but where individual predictions are valuable independently.

I have also consider just sticking with sub/supersampling, and skip this whole class weighting.


texts:
I have a data frame in which one of my columns is the target value and there are lots of ordinal categories in columns of data frame.

I want to encode these ordinal categories in columns with this method which is proposed here

I didn't understand what I should do. Should I find the probability of each category in each column and substitute the amount of probability with the string of each category?


texts: How to get the inertia at the begining when using sklearn.cluster.KMeans and MiniBatchKMeans

texts:
When I cluster a lot of data, it is hard to run KMeans and wait it stop until centers has not change, so I have to stop KMeans when it reach maximum number of iterations.
Here come problem: how can I evaluate the impact of this KMeans did for my data. I know I can get the inertia_ after KMeans fitting my data to see the sum of distances of samples to their closest cluster center. But How can I get the the inertia_ before KMeans fitting with which I can compare it with the the inertia_ after KMeans fitting, so that I can see the improvement of KMeans did for my data.


texts: What can I do with a Decision Tree with poor ROC

texts:
Let's say I do a Decision Tree analysis.

But the performance characteristics are nothing great (e.g. ROC is nothing great).

Is there anything I can do with this "not so great" tree. Or do I typically need to trash it and try something else (either a new data set or a new analysis on the same data)?


texts: Different learning rate for each of the layers?

texts:
I noticed that some popular deep learning frameworks like Keras or Pytorch allow you to set different learning rate for each layer.

What are the benefits of that approach?


texts:
I want to check an algorithm that learns the connections between different tables in a database.

Where can I find datasets that are compound of multiple tables (i.e. more than 5) that have connections between them?

All I can find on the web (mainly on Kaggle.com) are datasets with 1 or 2 tables.

I'm aware that my question is kind of off topic but still I'm sure it can help people that have the same problem as I do


texts: How can I output tokens from MWE Tokenizer?

texts:
How to output the tokens produced using MWE Tokenizer?

NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the user to enter multiple word expressions prior to using the tokenizer on the text.

Currently, I have a file consisting of phrases / multi-word expression I want to use with the tokenizer.  My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text.

So this leads me to ask if anyone knows how to output the token generated by add_mwe() so that I can verify that I am correctly passing the phrase to the function?


texts: Choosing a model for input: categorised, weighted sequence, output: binary variable

texts:
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements.
Illustrated example
Say I have the following categories: A, B, C, D.
Each category can have a weight between zero and one.
Example sequences:

A (0.33), B (0.71), C (0.0), D (0.95)   => 1
C (0.21), A (0.67) => 0
B (0), D (1) => 1


texts:
Suppose I want to develop and train a big end-to-end deep learning model using Tensorflow (1.15, for legacy reasons). The objects are complex, with many types of features that can be extracted: vector of numeric features of fixed length, sequences, unordered sets, etc. Thus, the model will include many submodules to deal with various types of features.

I have access to a server with several GPUs, so I want to distribute the model across them. What is the best way to do so? So far I'm thinking about placing subsystems on separate GPUs, but this presents some questions:

How costly would be the transfer of computation results between GPUs? Tensorflow does it automatically, right?
How costly would gradient computation and descent be, considering variables are placed on different GPUs? Would gradients also be computed on the same GPUs as their corresponding variables?


texts: Multi-inputs Convolutional Neural Network takes different number of images

texts:
I am using Keras to build a CNN model that takes two types of images as inputs (input1, input2) and produce one output. The model classifies the two inputs into a class, and the number of images in each type are different. (The number of images in the input1 is 1500 and in input2 is 4000).

This is my code:

in1 = Input(...)
 x = Conv2D(...)(in1)
# rest of the model
 out1 = Dense(...)(x)

 in2 = Input(...)
x = Conv2D(...)(in2)
# rest of the model
out2 = Dense(...)(x)

concatenated_layer = concatenate([out1, out2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(concatenated_layer)
modal= Model(inputs=[in1, in2], outputs=[output_layer])

Image generator code:

input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator()

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=1,
                                          img_height=224,
                                          img_width=224)

My questions are:

How can I determine the value of steps_per_epoch and validation_steps if the number of the images in each type are different?

In general, How to deal with the multi input Convolutional Neural Network that takes different number of images?


texts:
If there's a website/app that sells products and my job is to determine the order/ranking in which the products should be displayed.

For example : I click on restaurants and a list of restaurants pops up, I have to determine in what order the restaurants should be displayed.

All the data like ratings, distance to the customer, profit to us, prices, CTR, total number of views etc is available.

But how should I approaching this problem of rankings them in an efficient order ?
I have read about Learning to Rank and Collaborative Filtering but I am not sure if those are right for my problem, please help me out.

Thanks


texts:
I'm looking for real datasets on which I could test my DBSCAN algorithm implementation, that is, a dataset of points in (ideally 2 dimmensional) space, or a set of nodes and info about the distances between them.

I have looked on SNAP and CRAWDAD for such datasets, like datasets of road networks with distances, or cities with GPS coordinates, etc, but I haven't found any!

I know that the DBSCAN is said to be one of the best algorithims of it's kind on real data, but can't seem to find the real data sets people use...

Suggestions?


texts: Correlation with missing values. Is least squares an acceptable option?

texts:
I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation, but if I use listwise deletion I am afraid I'll end up with too little data.

I am wondering if I could estimate the missing values using least squares, and calculate the correlation using the now complete data set.


texts: Does anyone have any thought how come the fuzzy rules can be verified and validated?

texts: Fuzzy logic was utilized to derive a performance indicator of some manufacturing facilities in an uncertain environment. There are some historical data obtained from some manufacturing facilities’ rigorous assessments. How come can I use this data to verify and validate the fuzzy rules? Any insights and feedback would be appreciated.


texts: Why do I get such a low on image classification task with neural networks?

texts:
Below is my r code which I used on this kaggle dataset.

I'm not sure why the accuracy is so low, but I suspect that I encoded the pixel matrix incorrectly.

I transformed each pixel matrix picture into itself as a row of 200*200 columns and then did that for each pixel matrix making the individual rows for the 14970 pictures in the dataset. I'm not sure if the neural network understands the data and since it is taking data batch after batch I wonder how it can make sense of it.

A quick explanation and some suggested improvements to my code would be appreciated.

library(png)
library(colorspace)
library(abind)
library(reticulate)
library(keras)

setwd("~/E-books/Coding/Machine Learning with R/Neural Networks/four-shapes")

# Here I create a list of links to retrieve each image
files <- list.files(pattern = ".png$", recursive = TRUE)
filePathFunction <- function(x){
a <- matrix(nrow=length(x))
for(i in 1:length(x)){
conc <- c("~/E-books/Coding/Machine Learning with R/Neural Networks/four-shapes",x[i])
a[i] <- paste(conc, collapse="/")
}
return(a)
}
filePaths <- filePathFunction(files)
image(readPNG(filePaths[1]), useRaster=TRUE, axes=FALSE)

#circle is 0
#square is 1
#star is 2
#triangle is 3

circle <- rep(0,3720)
circle_array <- array(circle, dim=c(3720,1))
square <- rep(1,3765)
square_array <- array(square, dim=c(3765,1))
star <- rep(2,3765)
star_array <- array(star, dim=c(3765,1))
triangle <- rep(3,3720)
triangle_array <- array(triangle, dim=c(3720,1))
labels <- rbind(circle_array,square_array,star_array, triangle_array)

set.seed(101)
ind <- sample(nrow(filePaths),round(0.75*nrow(filePaths)),replace=F)
train_data <- filePaths[ind,]
test_data <- filePaths[-ind,]

train_labels <- labels[ind,]
test_labels <- labels[-ind,]

train_labels <- to_categorical(train_labels)
test_labels <- to_categorical(test_labels)

#trainPathArray and testPathArray reshape each pixel matrix into a 11228x200*200 array and
#3742x200*200 array matrix

trainPathArray <- matrix(ncol=200*200, nrow= length(train_data))
for(i in 1:length(train_data)){
trainPathArray[i,] <- array_reshape(readPNG(train_data[i]), dim= c(1,200*200))
}
testPathArray <- matrix(ncol=200*200, nrow= length(test_data))
for(i in 1:length(test_data)){
testPathArray[i,] <- array_reshape(readPNG(test_data[i]), dim= c(1,200*200))
}

network <- keras_model_sequential() %>%
layer_dense(units = 1000, activation = "relu", input_shape = c(200*200)) %>%
layer_dense(units = 1000, activation = "relu") %>%
layer_dense(units = 4, activation = "softmax")

network %>% compile(
optimizer = "rmsprop",
loss = "categorical_crossentropy",
metrics = c("accuracy")
)
network %>% fit(trainPathArray, train_labels, epochs = 5, batch_size = 12smilie
metrics <- network %>% evaluate(testPathArray, test_labels)


texts:
I have a question about the decoder transformer feed forward during training.
Let's pick an example: input data "i love the sun" traduction i want to predict (italian traduction) "io amo il sole".
Now i feed the encoder with the input "i love the sun" and i get the hidden states.
Now i have to do multiple feed forwards on the decoder with the input "BOS io amo il"
where BOS is a token that stands for beginning of sentence.
So i have this feedforward i assume

[BOS, IO, AMO, IL] -> decoder -> IO
[BOS, IO, AMO, IL] -> decoder -> AMO
[BOS, IO, AMO, IL] -> decoder -> IL
[BOS, IO, AMO, IL] -> decoder -> SOLE

I think this is the correct way. And what should be applied to differentiate the training i think is the masked attention mechanism maybe(?)
is it right to assume that the masking will be
[1 0 0 0,
 0 0 0 0 ,
 0 0 0 0,
 0 0 0 0]   for the first feed forward

[1 0 0 0,
 1 1 0 0 ,
 0 0 0 0,
 0 0 0 0]   for the second feed forward

[1 0 0 0,
 1 1 0 0 ,
 1 1 1 0,
 0 0 0 0]   for the third feed forward

[1 0 0 0,
 1 1 0 0 ,
 1 1 1 0,
 1 1 1 1]   for the fourth feed forward

is it the correct way? or what should be different?
If you can provide me also a python implementation could be useful, thanks in advance.


texts: How to Process Large JSON Files with Python

texts:
I've been using Python for quite some time now, scripting command line tools, etc.  I'm just now moving to Data Analysis with the language; are there any tips for handling larger json files.  Particularly, determining appropriate specs for a system, threading/multiprocessing, etc.

I'm finding that it's taking an excessive amount of time to handle basic tasks; I've worked with python reading and processing large files (i.e. Log files), and it seems to run a lot faster.

Example, I'm downloaded a json file from catalog.data.gov for traffic violations.  The file is 758Mb in size and it takes a long time to do something very simple.

>>> import ijson
>>>
>>> filename = 'rows.json'
>>> with open(filename, 'r') as f:
...     objects = ijson.items(f, 'meta.view.columns.item')
...     columns = list(objects)
...     print columns[0]
...     f.close()
...

{u'name': u'sid', u'format': {}, u'dataTypeName': u'meta_data', u'fieldName': u':sid', u'renderTypeName': u'meta_data', u'position': 0, u'id': -1, u'flags': [u'hidden']}

I've worked with json before, but this seems to be taking a long time for something so simple.  I'm did this on a laptop with 4GB of RAM (~2G free).


texts: Analysis of Split (A/B) tests using Poisson and/or Binomial Distribution

texts:
Cross posting this from Cross Validated:

I've seen this question asked before, but I have yet to come across a definitive source answering the specific questions:

What's the most appropriate statistical test to apply to a small A/B test?
What's the R code and interpretation to analyze a small A/B test?

I'm running a small test to figure out which ads perform better. I have the following results:

Position 1:

variation,impressions,clicks
row-1,753,26
row-3,767 7

Position 2:

variation,impressions,clicks
row-1,753,16
row-3,767 13

Position 3:

variation,impressions,clicks
row-1,753,2
row-3,767 7

I think it's safe to say these numbers are small and likely to be not normally distributed. Also, it's click data so there's a binary outcome of clicked or not and the trials are independent.

Appropriate test

In analyzing each position for significance, I think comparison with a binomial or Poisson distribution makes the most sense.

According to the OpenIntro Stats (and other sources) book, a variable follows a Poisson distribution "... if the event being considered is rare, the population is large, and the events occur independently of each other."

The same source classifies a binomial variable approximately the same way adding that the probability of success is the same and the number of trials is fixed.

I appreciate this is not an either/or decision and analysis can be done using both distributions.

Given A/B (split) testing is a science that has been practiced for several years, I imagine that there is a canonical test. However, looking around the internet, I mostly come across analysis that uses the standard normal distribution. That just seems wrong :)

Is there a canonical test to use for A/B tests with small #'s of clicks?

Interpretation and R code

I've used the following R code to test significance for each position:

Position 1:

binom.test(7, 767, p=(26/753))

Exact binomial test

data:  7 and 767
number of successes = 7, number of trials = 767, p-value = 1.077e-05
alternative hypothesis: true probability of success is not equal to 0.03452855
95 percent confidence interval:
 0.003676962 0.018713125
sample estimates:
probability of success
           0.009126467

I interpret this result to mean: The probability of success in the test group is indeed different than the control group with a 95% confidence interval that the success probability is between .368% and 1.87%

ppois(((26-1)/753), lambda=(7/767), lower.tail = F)
[1] 0.009084947

I interpret this result to mean: Given a Poisson distribution with a click rate of 7 per 767 trials, there is a 0.9% chance of having a click rate of 26 or more per 753 trials in the same distribution. Contextualized in the ad example,
there is a .1% chance that the control ad actually performs the same as the test ad.

Is the above interpretation correct? Does the test and interpretation change with the different positions (i.e. are the results of the Poisson test more appropriate for Position 3 given the small numbers)?


texts: Directly proportional Trend in training and cross validation curves

texts:
In continuation with a question already asked, I tried the same curve on a different dataset I found. My model is a simple Logistic regression curve with OnevsRest Classifier. But the graph I obtained this time was different.

What could be a possible reason for this? As explained in the other answer, if it takes less data to fit better, shouldn't the accuracy decrease here as well?


texts: Input representation in a neural network

texts:
I have been going through this book (https://www.cs.toronto.edu/~graves/preprint.pdf) on sequence labelling by Alex Graves. On page 29, under the section of input representation he states that the performance of neural networks are almost similar over a wide range of input representations. If some of you can put some light on this paragraph from the book and explain why is it that neural networks are relatively robust to input data representations?

Thanks!


texts:
I was studying the algorithm of CART (classification and regression trees), but the formula of the prediction is irritating me.
First we have the following definition:
Let $X:={x_1,...,x_N} \subset \mathbb{R}^d $ of datapoints and $B$ the smallest box:
$$ B(X):=\{z\in \mathbb{R}^d : \min_{x\in X} x_j \leq z_j \leq \max_{x\in X} \forall j\in [d]\}$$
and let $I$ be the indicator function:
$$I[p]=1 \text{ if }p \text{ holds } 0 \text{ otherwise}$$
So let's imagine that the CART algorithm has split the datapoints resulting in a set of boxes $B_1,B_2,..., B_k$.
Now our script says:

For prediction, one simply follows the decisions of the tree until one reaches a leaf box $B$. Then the predicted class is the majority of the classes in $B$. If $B_1,...,B_k$ are the boxes output at the end of the algorithm, then one computes
$$ f(x) = \Sigma_{j=1}^k w_j I[x\in B_j],$$
where
$$ w_j=\frac{\|\{x_i \in X \cap B_j : y_i=1\}\|}{\|X\cap B_j\|}$$
is the prediction (majority) of the class 1 in box $B_j$. Thus, $f(x)\in[0,1]$.

I cannot understand how with this formula we get $f(x)\in[0,1]$.
Here an example: Let's imagine we have a binary problem ($y\in\{0,1\}$) and we have an $x$ and as stated we follow the decision tree until we reach a leaf box $B$. In $B$ we have $|B_0|=2,|B_1|=1$ . Now we want to compute the majority vote for the prediction:
$$ f(x) = \Sigma_{j=0}^k w_j I[x\in B_j]=w_0 I[x\in B_0]+w_1 I[x\in B_1],$$
If I understand right, we have:
$$w_0=\frac{2}{3}, w_1=\frac{1}{3}$$
and
$$I[x\in B_0]=1, \text{  } I[x\in B_1]=1$$
If this is correct, then
$$f(x)=1$$ and it will always be in any case $f(x)=1$.
Am I misunderstanding something or is something wrong in the notation of the script?


texts: How do AI's learn to act when the problem space is too big

texts:
I learn best through experimentation and example. I'm learning about neural networks and have (what I think) is a pretty good understanding of classification and regression and also supervised and unsupervised learning, but I've stumbled upon something I can't quiet figure out;

If I wanted to train an AI to play a complicated game; I'm thinking something like a RTS (eg. Age of Empires, Empire Earth etc.). In these types of games there is typically a number of entities controlled by the player (units, buildings) each with different capabilities. It seem like the problem of that the AI does would be classification (eg. choose that unit, and that action), however since the number of units is a variable how does one handle a classification problem in this way?

The only thing I can think of is multiple networks that do different stages (one for overall strategy, one for controlling this type of unit, one for that type of building etc.); but this seems like I'm making the problem to complicated.

Are there any good example of machine learning/neural networks learning complex games (not specifically RTS, but more complicated the Mario)?


texts:
I am working on a problem on Named Entity Recognition. Given a text, my model is detecting the Named Entities and extracting that info for the end user. Now the ask is end user needs a confidence score along with the extracted entity. For example, given text is: XYZ Bank India Limited is a good place to invest your money - Our model is detecting XYZ Bank as an Org, but India as a Location (which is wrong - the whole XYZ Bank India Limited is the name of the organization). Our model also gives a probability score for each token it classifies. But the end user wants to know the confidence of the model that it did not mistake to detect the subsequent tokens as the parts of the organization name.
Question is - how can we efficiently measure that in a given sequence our model is detecting a certain sub-sequence as an Organization name (or a Location or something else) correctly or not? How can we say that it did not miss out any subsequent or preceding token which actually a part of the named entity (like it missed India Limited in the above example)?


texts:
Suppose a convolution neural network is trained on small images of an object, say flower, as in following 3 training images:

Will this CNN correctly classify if the same object is present in zoomed form in a test image? As in following example:

What if the situation is reverse, i.e. training on large sized objects and testing with small sized object?

What is the best way to take care of different size of object that may be present for correct classification of images?

(The images are from: https://www.shutterstock.com/video/search/flowers )


texts:
I had this basic query on ML and would like to get basic ideas on modelling prediction models using ML and Python.

Say I have a training data of 1000 items as

Item_name, Attrib_1, Attrib_2, Attrib_3,....Attrib_N, Cost

And my aim is to create a model to predict cost for a new item given the attributes.

So where should I start and what are the different ways to prediction and solve this problem ? Also how to evaluate different methods ?


texts: Regression and Classification in one Neural network

texts:
For example consider object localization problem. Here NN will have 5 ouputs. output[0] will tell probability of object present in image, other 4 will tell bounding box coordinates.
As we see that output[0] has to use classification loss like cross entropy and output[1] to output[4] will have to use regression loss like Mean-squared-error.
So Total loss is something like this:
loss=Cross_entropy(output[0],Y[0])+MSE(output[1:5],Y[1:5]) #Y is true value

Are loss like that backprogationable in vectorised form?
Can I implement that kind of loss in tensorflow?
If yes, how does tensorflow do that? Does it perform differentiation on each element of vector or  matrix instead of whole thing at once?


texts: Can't reproduce results from GridSearchCV?

texts:
I am trying to find optimized n_neighbors value for KnearestClassifier using GridSearchCV. I am able to get optimized parameters but when I enter those in my classifier results don't match with GridSearchCVs best results.
clf = KNeighborsClassifier(n_neighbors=15, weights='uniform')

clf.fit(features_train, labels_train)

print('Score using optimized parameters: {}'.format(clf.score(features_test,       labels_test)))

params = {'n_neighbors':[1,10,15,20,25,30,35,40,45,50,60,70,80,90,100],    'weights':['uniform', 'distance']}
grid = GridSearchCV(clf, params, cv=10, )
grid.fit(features_train, labels_train)

print('Optimized Parameters:{}'.format(grid.best_params_))
print('Best Score from GridsearchCV parameters{}'.format(grid.best_score_))

Output:

Score using optimized parameters: 0.928
Optimized Parameters:{'n_neighbors': 15, 'weights': 'uniform'}
Best Score from GridsearchCV parameters: 0.962666666667


texts: Is it okay to use training data for validifying the trained model?

texts:
Currently, I have trained my model through 5-fold cross validation with very small amount of the sample (n=100).

I used whole data set to train and got quite low performance in terms of accuracy, which is bit higher than 70%.

However, if I put my data which was used for training back to trained model to validate, it gives me higher accuracy (80%).

So, my question is it okay to say that I have verified my trained model using training set and got 80 of accuracy? or should I have to stick with 70 % of accuracy that I received from 5-fold cross validation?


texts: Tool/Technology/Framework to process file with 1GB

texts:
I'm new to datascience, but I have the following problem:

1) Read a file of 1GB, which each line is a json object
2) There are two more files, much smaller, which I need to JOIN some data

In this case, which kind of tools is the best?

I saw some examples in pyspark, but I have no idea what happen under the hood.


texts: Which machine learning algorithms support a feedback ingestion loop?

texts: I understand stochastic gradient descent does. any other alorithm which supports incremental training preferably updating on a real time basis?


texts: What is the difference between a hashing vectorizer and a tfidf vectorizer

texts:
I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer

I understand that a HashingVectorizer does not take into consideration the IDF scores like a TfidfVectorizer does. The reason I'm still working with a HashingVectorizer is the flexibility it gives while dealing with huge datasets, as explained here and here. (My original dataset has 30 million documents)

Currently, I am working with a sample of 45339 documents, so, I have the ability to work with a TfidfVectorizer also. When I use these two vectorizers on the same 45339 documents, the matrices that I get are different.

hashing = HashingVectorizer()
with LSM('corpus.db')) as corpus:
    hashing_matrix = hashing.fit_transform(corpus)
print(hashing_matrix.shape)

  hashing matrix shape (45339, 1048576)

tfidf = TfidfVectorizer()
with LSM('corpus.db')) as corpus:
    tfidf_matrix = tfidf.fit_transform(corpus)
print(tfidf_matrix.shape)

  tfidf matrix shape (45339, 663307)

I want to understand better the differences between a HashingVectorizer and a TfidfVectorizer, and the reason why these matrices are in different sizes - particularly in the number of words/terms.


texts:
Suppose that i have multidimensional dataset and performed some partitioning clustering on it. Is there any way to find out what objects in a particular cluster have in common (except the fact that clustering algorithm decided to put them together)?

I've read many times that clustering is not a well-posed problem in general and that one should not overinterpret its results but still people are trying to cluster multidimensional data and make some practical sense of the results. I just can't find any good source on how the interpretation is done in practice.

Any tips and resources will be highly appreciated.


texts: How to bootstrap the AUC on a data-set with 50,000 entries?

texts: I am learning classification. I do have AUC-ROC curve of the data-set. How should I proceed to bootstrap the AUC? If I do the sampling of the same size as data-set then my AUC value would not be the same all the time?


texts: What kind of data analysis problem is this?

texts:
This might be a dumb question, but I am an absolute beginner.
What kind of data analysis problem is the following?
Is it an Exploratory Data Analysis problem/Data Mining problem/data analytics problem?
Secondly, what steps can I follow to solve the following problem? Is there any algorithm?

Records in this data set describe decks of cards used in a popular
collectible card video game called "Clash Royale". These decks were
obtained using the RoyaleAPI.com service, from games that took place
in January 2019. Each record consists of five values:

timestamp of the game (column timestamp),
arena ID (column arena_id – higher the arena, more skilled/experienced a player is)
outcome of a game (column has_won, 1 – the player won, 0 the player lost)
a player ID (column tag)
list of exactly eight cards in the player’s deck separated by “_” signs (column player_deck)

Your task is to analyze this data and search for interesting card
usage patterns, and interactions/dependencies between cards.
For example:

find card combos that were particularly popular in January 2019 (e.g., top 100 card sets with regard to their support, top 100 card
sets of size 2, size 3, etc.),
identify those card combos which have high win-rates (e.g., top 100 card sets with regard to win-rate and with support > 1%),
does the card usage/popularity/effectiveness changes in time?
does the arena level have any influence on card usage/popularity/effectiveness?
find interesting associations between cards,
can you cluster players according to their play patterns and card preferences?     Additionally, design and construct a card recommender
system that allows players to indicate four cards which they want to
have in a deck, and recommends the remaining four to create a
reasonable deck. How can you evaluate the effectiveness of the
recommendations?    You have to report the discoveries in the form of
an R notebook (with code and all computation outcomes). You have to
remember about visualizations. I.e. you have to make this report as
visually interesting for a reader as you can.


texts: Creating a convolutional layer with weight normalization?

texts:
This paper proposes parameterizing convolutional weights by having the primary weights normalized to unit norm plus an extra scalar weight: https://arxiv.org/pdf/1602.07868.pdf Implementations do exist, but they seem kind of limited and finnicky with Keras versions and optimizers.

Would it be fundamentally different mathematically when doing gradient descent during training with backpropagation if I implemented this with 3 layers instead?

The 1st layer(s) would be branching convolutional layers with a unit norm kernel constraint (which is officially supported in Keras) and no bias weights. This way the input channels wouldn't be summed over.
The 2nd layer would be a merging layer that would concatenate along the input channel dimension we conveniently left untouched.
The 3rd layer would be another convolutional layer this time with a 1x1 convolution plus bias weights to perform the weighted summation with bias that would have followed the filtering done in a normal convolutional layer.


texts: Anomaly detection for high dimensional categorical data

texts:
I have a dataset with around 200+ categorical variables  $X_i$  and the sizes of their domain $|X_i|$ range from 2 to 8k. So, if I one-hot encode the combination of these variables, the vector space (around 50k) would become extremely sparse.
I am wondering if there is an effective way to encode such data. To be specific, the data contain choices users made when requiring SMS service through websites, something like "source_region", "browser_type", and users' own properties like "Is_Vip". I hope these details help.
The number of labels is limited, so the model should be trained in an unsupervised fashion.
What I have tried
I replaced region-related columns with their corresponding GDPs, as well as lamptitudes and attitudes to add geographical and economic information. And I tried a naive auto-encoder to encode the features and chose those samples with high reconstruction loss as anomalies. But the recall on the test set is only 10%.
Though I know the anomalies here are based on the context like users, building a model for every user is impractical. And I've thought about embedding these requests and so we can get something like user vector inspired by recommendation system. But due to the limit of unsupervised models,I've lost my way.


texts: Is there a scientific method for grouping continuous numbers for this problem

texts:
I use python, and I have a list of numbers that contains 19 elements and I would like to divide this list into 6 groups or fewer.

The list could have numbers between 0 and 1. and it mustn't be ordered, I need to keep it on its form and getting cut-off

list:

Numbers :  [[ 0.867   - 0.808   - 0.740    - 0.746    - 0.674   - 0.669   -
              0.648   - 0.722   - 0.781    - 0.612    - 0.575   - 0.566   -
              0.500   - 0.555   - 0.818    - 0.800    - 0.500   - 0.500   - 0.666 ]]

I would like to get clusters like :

A: [[ 0.867   - 0.808   - 0.740    - 0.746 ]]

B: [[ 0.674   - 0.669   - 0.648 ]]

C: [[ 0.722   - 0.781 ]]

D: [[ 0.612   - 0.575   - 0.566    - 0.500    - 0.555  ]]

E: [[ 0.818   - 0.800 ]]

F: [[ 0.500   - 0.500    - 0.666 ]]

I make splitting by eyes and for that, I ask for a scientific method for getting my objective.

Concerning how I defined these clusters :

I have a value that is getting from an extra function, it is equal to 0.80. I need to compare each value on the list with 0.80 for knowing the difference.

After making a comparison I get the following table

Numbers      Difference_0.80
0.867            +0.06
0.808             0.0
0.740            -0.06
0.746            -0.06

0.674            -0.13
0.669            -0.14
0.648            -0.16

0.722            -0.08
0.781            -0.02

0.612            -0.19
0.575            -0.23
0.566            -0.24
0.500            -0.3
0.555            -0.25

0.818            +0.01
0.800             0.0

0.500            -0.3
0.500            -0.3
0.666            -0.14

when I tried clustering method (with n_clusters=2) I ve got :

0   category_Kk-mean
0.867   0
0.808   0
0.740   0
0.746   0
0.674   1
0.669   1
0.648   1
0.722   0
0.781   0
0.612   1
0.575   1
0.566   1
0.500   1
0.555   1
0.818   0
0.800   0
0.500   1
0.500   1
0.666   1

But I want to know also that  this category (D) have a big decrease than a category (B):

category D
0.612   1
0.575   1
0.566   1
0.500   1
0.555   1

Category B

0.674   1
0.669   1
0.648   1

I tried with n_clusters=3 but I got a totaly bad result

Is there any method in the statistic or mathematic that can help me for getting that


texts: How to find correlations between data over time?

texts:
I have daily data of my sales and a daily update of how the COVID-19 cases increase. My daily sales contains information on my customer and on my product. My end goal would be to see if certain products are correlated (sold more or less) after covid than before. Or if certain regions had more sales before or after COVID.

My initial thought would be to do some kind of event analysis and compare if I have more sales before and after that certain event. Does anyone has some experience with this or can link some papers/references on this topic?


texts: spatiotemporal prediction using support vector regression

texts:
i am using a geographic dataset and i intend to use SVR as machine learning method for predicting spatiotemporal patterns from this dataset.

My question is does SVR canensure spatiotemporal prediction from geographic datasets?


texts:
I don't have a maths / stats / data science background and need to evaluate which of the two evaluations below (numerical regression on Amazon Machine Learning) predict more accuracy. Both models use the same data set but it's looking at different time frames both on the independent and dependent variables.

How can I evaluate which one of the two models is more accurate? And is there a way to tell how accurate these two models are in general (e.g. 75%)?

Model 1

Model 2


texts: Why do we convert skewed data into a normal distribution

texts:
I was going through a solution of the Housing prices competition on Kaggle (Human Analog's Kernel on House Prices: Advance Regression Techniques) and came across this part:

# Transform the skewed numeric features by taking log(feature + 1).
# This will make the features more normal.
from scipy.stats import skew

skewed = train_df_munged[numeric_features].apply(lambda x: skew(x.dropna().astype(float)))
skewed = skewed[skewed > 0.75]
skewed = skewed.index

train_df_munged[skewed] = np.log1p(train_df_munged[skewed])
test_df_munged[skewed] = np.log1p(test_df_munged[skewed])

I am not sure of what is the need for converting a skewed distribution into a normal distribution. Please, can someone explain in detail:

Why is this being done here? or How is this helpful?
How is this different from feature-scaling?
Is this a necessary step for feature-engineering? What is likely to happen if I skip this step?


texts: Neural Network backprop formula - Matrix dimensions won't match?

texts:
I want to start by taking an example for a normal neural network with 2 input nodes, 3 hidden nodes and 2 output nodes. Let the weights between input and hidden nodes are $W_i{_j}$ (2x3) and weights between hidden nodes and output (3x2) are $W_j{_k}$. Last layer has linear activation. The gradient of error wrt:

$$W_i{_j}= W_j{_k} * g'(W_i{_j} * x) * x * err$$

But this does not satisfy the weight update equation as the product gives the size 2x1. But it should be 2x3.


texts:
I know there is f1_score metric to get all types of F1 scores(micro, macro and weighted). But I want to be able to print micro averaged F1 score using classification_report of sklearn. By default, it seems to be returning weighted micro averaged F1. But I want the micro averaged F1 in the classification_report. How do I do that?

Also, I know the difference in formula between weighted and microaverages, but what are the instances where one would be preferred over other? And, what information do they convey?


texts:
I have a loss function that minimizes the error according to what I want the neural network to do. The problem is, that it is a nondifferentiable function. How can I handle this?

the loss function: $(1-y) \cdot log(1-p) + min((1-y)-(y \cdot log(p)))$

$y$: target
$p$: prediction
len((1-y)-(y*log(p))) = len(y) = len(p)

I have tried to smooth the minimum, but I am not sure this is good enough. As you can see, the min operator is nondifferentiable

How to handle a nondifferentiable loss function with Neural Networks?


texts: Why do I get two different values in heatmap and feature_importances?

texts:
I'm running a feature selection using sns.heatmap and one using sklearn feature_importances.
When using the same data I get two difference values.
Here is the heatmap
and heatmap code
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np
import seaborn as sns

training_data = pd.read_csv(
    "/Users/aus10/NFL/Data/Betting_Data/CBB/Training_Data_Betting_CBB.csv")

df_model = training_data.copy()
df_model = df_model.dropna()
df_model = df_model.drop(['Money_Line', 'Money_Line_Percentage', 'Money_Line_Money', 'Money_Line_Move', 'Money_Line_Direction', "Spread", 'Spread_Percentage', 'Spread_Money', 'Spread_Move', 'Spread_Direction',
                          "Win", "Money_Line_Percentage", 'Cover'], axis=1)

X = df_model.loc[:, ['Total', 'Total_Move', 'Over_Percentage', 'Over_Money',
                     'Under_Percentage', 'Under_Money']]  # independent columns
y = df_model['Over_Under']  # target column

# get correlations of each features in dataset
corrmat = df_model.corr()
top_corr_features = corrmat.index
plt.figure(figsize=(20, 20))
# plot heat map
g = sns.heatmap(
    df_model[top_corr_features].corr(), annot=True, cmap='hot')

plt.xticks(rotation=90)
plt.yticks(rotation=45)

plt.show()

Here is the feature_importances bar graph

and the code
import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt
from sklearn.model_selection import StratifiedKFold
from sklearn.inspection import permutation_importance

training_data = pd.read_csv(
    "/Users/aus10/NFL/Data/Betting_Data/CBB/Training_Data_Betting_CBB.csv", index_col=False)

df_model = training_data.copy()
df_model = df_model.dropna()

X = df_model.loc[:, ['Total', 'Total_Move', 'Over_Percentage', 'Over_Money',
                     'Under_Percentage', 'Under_Money']]  # independent columns
y = df_model['Over_Under']  # target column

model = RandomForestClassifier(
    random_state=1, n_estimators=100, min_samples_split=100, max_depth=5, min_samples_leaf=2)

skf = StratifiedKFold(n_splits=2)

skf.get_n_splits(X, y)

StratifiedKFold(n_splits=2, random_state=None, shuffle=False)

for train_index, test_index in skf.split(X, y):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

model.fit(X_train, y_train)
# use inbuilt class feature_importances of tree based classifiers
print(model.feature_importances_)
# plot graph of feature importances for better visualization
feat_importances = pd.Series(model.feature_importances_, index=X.columns)
perm_importance = permutation_importance(model, X_test, y_test)
feat_importances.nlargest(5).plot(kind='barh')

print(perm_importance)

plt.show()

I'm not sure which one is more accurate or if I'm using them in the correct way? Should I being using the heatmap to eliminate collinearity and the feature importances to actually selection my group of features?


texts: Regressor to estimate probability from samples

texts:
What is a good way to estimate probabilities in a dataset of samples? Each sample is a measurement, that is usually 1 or 0. The goal is to calculate probabilies based on all feature rows.

Simple example dataset:

    feature   label
    dog       1
    dog       0
    dog       0
    dog       0
    cat       1
    cat       0

Expected Output:

    feature  result
    dog      0.25
    cat      0.5

The real dataset has around 50 features.