texts: Combining heterogeneous data sets for more powerful machine learning

texts:
Suppose we have two data sets of movie reviews; one from IMDB and one from Rotten Tomatoes (RT). Each entry has a written-review and a score attached to it. The concatenated datasets might look like

Movie|Score|Review|Site

The Lion King|9.8|"This is one of the best movies ever!"|IMDB

The Lion King|4.5|"This move absolutely made my childhood"|RT

etc.

Our task is to predict scores from reviews. This is similar to the Kaggle IMDB sentiment analysis challenge, except we're trying to predict a continuous variable from two data sources. For a single data set, we might train a model in Keras with an embedding and lstm layer. In R:

model <- keras_model_sequential() %>%
   layer_embedding(input_dim = features, output_dim = 32) %>%
   layer_lstm(units = 32) %>%
   layer_dense(units = 1, activation = "linear")

where features is the number of words in our corpus.

How do we combine the IMDB and Rotten Tomatoes data sets to maximize our predictions?

My first thought is to include the site column as a categorical variable and then let the model learn how to best combine them. Would this cause a skew in the predictions?

We could also try multi-task learning, but most of these methods seem to be designed to predict repeated observations. Such as 5 different people reviewing the same 100 movies.

Some things to consider:

IMDB and RT don't have all the same movies (i.e. not a repeated observation problem)
There are words local to each dataset. i.e. the word "rotten" might be a frequent word on RT, but never appears in the IMDB dataset.
The outcome measures are different. RT uses a 5 star scale, whereas IMDB uses a 10 star one.

How do we deal with this heterogeneity?


texts:
I'm using fitctree function from Matlab to fit a CART tree to my dataset (10'000 data points, 20 features).

First, how can I exactly prune it and to what level should the tree be pruned? I think I have to use the prune method but I'm confused by the 'MergeLeaves' and 'Prune' flag.

Second, what values of 'MinLeafSize', 'MinParentSize' and 'MaxNumSplits' is reasonable to try?


texts: I recently came across this term recurrent heavy subgraph in a talk. I don't seem to understand what it means and Google doesn't seem to show any good results. Can someone explain what this means in detail.


texts: Interpreting high precision and very low recall score

texts:
I was training model on a very imbalanced dataset with 80:20 ratio of two classes. The dataset has thousands of rows and I trained the model using

DeccisionTreeClassifier(class_weight='balanced')

The precision and recall I get on the test set were very strange

Test set precision : 0.987767
Test set recall : 0.01432

I'm unable to interpret the results. What does this tell me about my classifier?


texts: I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec on it, or is there a better way?


texts:
I'm trying to understand the role of data augmentation and how it can affect the performance/accuracy of a deep model. My target application is a fire classification (fire or not, on video frames), with almost 15K positive and negative samples, and I was using the following data augmentation techniques. Does using ALL the followings always increase the performance? Or we have to choose them somehow smartly given our target application?

rotation_range=20, width_shift_range=0.2, height_shift_range=0.2,zoom_range=0.2, horizontal_flip=True

When I think a bit more, fire is always straight up, so I think rotation or shift might in fact worsen the results, given that it makes the image sides stretch like this, which is irrelevant to fires in video frames. Same with rotation. So I think maybe I should only keep zoom_range=0.2, horizontal_flip=True and remove the first three. Because I see some false positives when we have a scene transition effect in videos.

Is my argument correct? Should I keep them or remove them?


texts: Bayesian regularization vs dropout for basic ann

texts:
Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization?

On one hand I would think that technically this should work just fine, but on the other hand if bayesian regularization is like finding the MAP estimate given some prior shouldn't the effect just be that it creates random noise in your outcomes?


texts: How to plot cost versus number of iterations in scikit learn?

texts:
One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is:

  Debugging gradient descent. Make a plot with number of iterations on the x-axis. Now plot the cost function, J(θ) over the number of iterations of gradient descent. If J(θ) ever increases, then you probably need to decrease α.

Do gradient descent based models in scikit-learn provide a mechanism for retrieving the cost vs the number of iterations?


texts: Featurization for Relation Extraction using Support Vector Machine(SVM)

texts:
Regarding Relation Extraction using SVM paper(https://www.researchgate.net/publication/225671271_Relation_Extraction_Using_Support_Vector_Machine)I am looking for any code references on how to make features(word, pos, syntactic features, semantic feature types, chunk tags,distance) for relation extraction between two entities.
Entity1 - Drugs,
Entity2 - Disease,
Relation types:1.taking a drug caused disease
               2. Drug was taken for a disease,
first model is a binary classification model to classify if the entities are related to each other or not.
Second model takes only positive outputs from first model and classifies the relation which is a multi class classification model


texts: Error after merging two Deep Learning models VGG16 and ResNet50

texts:
I have merged two different models namely VGG16 and ResNet50 and given the outputs of the two models as input to another model. I have checked the Layers graph is correct. Before merging the code was running perfectly fine giving correct outputs. I am getting an error:
"ValueError: Shapes (None, None) and (None, 7, 7, 3) are incompatible" on the line 6

ValueError
Traceback (most recent call last)
<ipython-input-36-620554d0106f> in <module>()

4     epochs = 200,

5     validation_data = validation_generator,

----> 6     validation_steps=2

my code is:
inputs_2 = keras.Input(shape=(224, 224, 3), name="img")
vgg = VGG16(input_tensor=inputs_2, weights='imagenet', include_top=False)
for layer in vgg.layers:
    layer.trainable = False

resnet = ResNet50(input_tensor=inputs_2, weights='imagenet', include_top=False)
for layer in resnet.layers:
    layer.trainable = False

mergedOutput = Concatenate()([vgg.output, resnet.output])
x = layers.Dense(256, activation="relu")(mergedOutput)
prediction = Dense(3, activation='softmax')(x)
model = Model(inputs=vgg.input, outputs=prediction)

model.compile(loss="categorical_crossentropy",optimizer='adam',metrics=['accuracy'])

keras.utils.plot_model(model, "mini_resnet.png", show_shapes=True)
train_datagen = image.ImageDataGenerator(
rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True,
)

test_dataset = image.ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
TRAIN_PATH,
color_mode = "rgb",
target_size = (224,224),
batch_size = 32,
class_mode = 'categorical'
)
print(train_generator.class_indices)

validation_generator = test_dataset.flow_from_directory(
VAL_PATH,
color_mode = "rgb",
target_size = (224,224),
batch_size = 32,
class_mode = 'categorical')

history = model.fit_generator(
train_generator,
steps_per_epoch=8,
epochs = 200,
validation_data = validation_generator,
validation_steps=2
)


texts: What is exactly meant by neural network that can take different types of input?

texts:
There is a scientific document that implements a convolutional neural network to classify 3 different types of data, although how exactly, is unknown to me.

Here's the explanation of network architecture:

  This section describes architecture of our neural net which is
  depicted in Fig. 3.
  Our network has three types of inputs: Screenshot (we use upper crop of the page with dimensions 1280 × 1280, however this net can
  work with arbitrarily sized pages), TextMaps (tensor with dimensions
  $128 \times 160 \times 160$) and Candidate boxes (list of box coordinates of
  arbitrary length).

  A screenshot is processed by three convolutional layers (the first two layers are initialized with pretrained weights from BVLC
  AlexNet). TextMaps are processed with one convolutional layer with
  kernel size $1 \times 1$ and thus its features capture various
  combinations of words. These two layers are then concatenated and
  processed by final convolutional layer.

What exactly is implied by Our network has three types of inputs above? Is it possible for convolutional neural network to pass different types of inputs differently?

From my understanding, neural network for Screenshot input would be created like this:

def CNN(features, labels, mode):
  input_layer = tf.reshape(image, [-1, 1280, 1280, 1])

  # Conv+ReLU
  conv_relu_1 = tf.layers.conv2d(
      inputs=input_layer,
      filters=96,
      kernel_size=[11, 11],
      padding="same",
      activation=tf.nn.relu)

  # MaxPool
  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[3, 3], strides=2)

  # Conv + ReLU
  ...

So let's say this is first neural network, then should I create another neural network for TextMaps and concatenate results? Or does every magic just happen in a single neural network?

In short, can I create neural network that takes different types of input individually or do I use different neural networks for each of them and then group their outputs?

Thank you!


texts: Clustering combining numeric features and weekday & hour cyclic features

texts:
The question is strictly related to
What is a good way to transform Cyclic Ordinal attributes?
and
Ways to deal with longitude/latitude feature

They presented a very clear answer about the approach to normalise hour variable and latlong variables.

The addition of this question is about how to approach normalization having both weekday and hour features. I wonder this because they are different granular level of the same information, hence, strictly related.

The objective is to run a k-means algorithm combining the normalised weekday and hour features with other traditional numeric features.


texts: Autoencoder: using cosine distance as loss function

texts:
I'm trying to train an autoencoder (in PyTorch) to reconstruct gene profiles. At the moment I'm using the Mean Squared Error (MSE) loss for training: the model is not overfitting and both the training and validation loss are decreasing. The problem is that the cosine similarity on the validation set between original and reconstructed vectors has a mean of 0.4. I was thinking of using the cosine similarity as loss function instead of MSE.

At the following link (slide 18), the author proposes the following loss:

$$
l(x_1, x_2, y) =
\begin{cases}
max(0, cos(x_1, x_2) - m) & \text{if $y$ == -1} \\
1 - cos(x_1, x_2) & \text{if $y$ == 1}.
\end{cases}
$$

I'm not entirely sure whether this is the right approach, but I'm having some difficulties even understanding the formula. What is $y$ (the cosine similarity between $x_1$ and $x_2$?) and why it's an input to the loss?


texts:
What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at random.
An example of missing not at random categorical feature

Class 1: User has a red car
Class 2: User has a blue car
Class 3: User has no car (missing value)

In this case is it better to treat this feature as a binary 0/1 with NaN missing value, or as multi-label feature: 0,1 and -999 for missing?
The same question applies for a numerical feature that indicates age of user's car. Here, missing values indicate the user has no car. Is it better to keep missing values as NaN, or to impute these values? If imputing is better, should I impute using the median value and add an interaction feature for when the values are missing?


texts:
I am exploring Amazon SageMaker as a scalable machine learning solution.

My question is; is it required that the training data first be uploaded in Amazon S3?


texts: I have a small data set containing around 80 images for people, and the corresponding ground truth for skin regions. I want to train a classifier to be able to detect a skin, and use it later on my own generated data, but I'm not sure what is the right way to go. I believe it would be hard to train a CNN due to the small number of images. So can someone suggest a direction?


texts:
The code below (I found from googling) does the trick in terms of outcome. What I need to work out is the formula for this, so for any Y point we can get corresponding X.

This is part of a larger calculation using binned data Y to attribute time savings X.

Any help much appreciated.


texts:
Is there any NLP framework/ algorithm which is able to derive relationships between different entities?

Dogs bark
For example, if the system can identify and understand a phrase, 'Dogs were barking loudly' and can relate to another phrase, 'Usually dogs bark loud, when they sense danger'. And from this piece of information, the system can imply/ derive: 'Dogs had sensed danger' - derived sentence.

Can BERT help in identifying and implying?


texts: How can I avoid requiring global information for performing regression on meter variables?

texts:
Note: With a meter variable a timestamped value is the sum of all previous differences plus a difference to the most recent value. Think of a electricity meter counting the use of energy.

The goal here is to perform some form of regression (e.g. a Random Forest method) on a data series of a meter variable and then use the resulting model to fix gaps in the data series and possibly do further analysis on the data, for example removing noise from a faulty sensor.

However the possible patterns in the data are likely periodic on timescales smaller than the entire series. Thus we're not interested in modeling the sum of the values. We transform the data by calculating $$\delta_i=\frac{v_i-v_{i-1}}{t_i - t_{i-1}}$$ and then perform the regression on $\delta$.

With the generated model $m(t)$ we can calculate a missing value $v_i$ as $$v_i = v_{i-1} + (t_i - t_{i-1})\cdot m(t).$$

Easy, right?

But now let's throw in some serious noise. Depending on the chosen regression method this doesn't matter much and only makes the predictions worse, but still useful. But what if we want to calculate a missing $v_i$ and $v_{i-1}$ happens to be some bogus value? The model doesn't mind, but for the calculation of $v_i$ we have to assume that $v_{i-1}$ is correct.

Is there a way around that? Is it possible to calculate missing values using only (time-)local information?


texts: Combining data sets without using row.name

texts:
I start with a data.frame (or a data_frame) containing my dependent Y variable for analysis, my independent X variables, and some "Z" variables -- extra columns that I don't need for my modeling exercise.

What I would like to do is:

Create an analysis data set without the Z variables;
Break this data set into random training and test sets;
Find my best model;
Predict on both the training and test sets using this model;
Recombine the training and test sets by rows; and finally
Recombine these data with the Z variables, by column.

It's the last step, of course, that presents the problem -- how do I make sure that the rows in the recombined training and test sets match the rows in the original data set? We might try to use the row.names variable from the original set, but I agree with Hadley that this is an error-prone kludge (my words, not his) -- why have a special column that's treated differently from all other data columns?

One alternative is to create an ID column that uniquely identifies each row, and then keep this column around when dividing into the train and test sets (but excluding it from all modeling formulas, of course). This seems clumsy as well, and would make all my formulas harder to read.

This must be a solved problem -- could people tell me how they deal with this? Especially using the plyr/dplyr/tidyr package framework?


texts: Aggregate SHAP importances from different models

texts:
A couple of questions on the SHAP approach to the estimation of feature importance.
I would like to use the random forest, logistic regression, SVM, and kNN to train four classification models on a dataset. Parameters in each training are chosen to give the best accuracy and precision for every model.
A feature has a different magnitude of SHAP values in every model.

Are these differences meaningful, so as the feature indeed has different importance depending on an algorithm (RF vs. SVM vs. kNN...)?
Can I aggregate feature importances from these four models? For example, can I sum up SHAP values of every feature in all four models combined and have summary importance among all models?
If 2 is valid, do I need the same feature set for every model?

Please, also provide related references, if you can.
A related previous post, Is it valid to compare SHAP values across models?, unfortunately, got no answers.


texts: Why is logistic regression not sigmoidal?

texts:
The blue dots are the raw data and the line is my logistic regression. The line is quite straight and not sigmoidal as I would have expected. I suspect there is something wrong in my gradient descent equation but I don't understand the maths well enough to find the mistake.

This is the code I used to generate the logistic regression:

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from scipy import stats

def logit(z):
    return (1/(1+np.exp(-z)))

def cost(data, weights):
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    cost = np.sum(y_val*scores - np.log(1 + np.exp(scores)))
    return cost

def gradient_descent(data, weights, learning_rate):
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    weights -= (learning_rate/m)*np.dot(np.transpose(x_val),(logit(scores)-y_val))
    return weights

dataset = pd.read_csv("/Users/An/Desktop/data/glass.csv")
dataset.sort_values('Al', inplace=True) # sort by ascending "Al" values
dataset.head()
dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})
ones = np.ones(len(dataset))
data = np.stack((ones,dataset["Al"], dataset["binary"]), axis=-1)

weights = np.random.rand(2)

def log_reg(data, weights):
    for i in range(9000):
        weights = gradient_descent(data, weights, 0.000001)
        loss = cost(data, weights)
    return weights

w = log_reg(data, weights)


texts: Suggestions on what patterns/analysis to derive from Airlines Big Data

texts:
I recently started learning Hadoop,
I found this data set http://stat-computing.org/dataexpo/2009/the-data.html - (2009 data),

I want some suggestions as what type of patterns or analysis can I do in Hadoop MapReduce, i just need something to get started with, If anyone has a better data set link which I can use for learning, help me here.

The attributes are as:

1   Year    1987-2008
2   Month   1-12
3   DayofMonth  1-31
4   DayOfWeek   1 (Monday) - 7 (Sunday)
5   DepTime actual departure time (local, hhmm)
6   CRSDepTime  scheduled departure time (local, hhmm)
7   ArrTime actual arrival time (local, hhmm)
8   CRSArrTime  scheduled arrival time (local, hhmm)
9   UniqueCarrier   unique carrier code
10  FlightNum   flight number
11  TailNum plane tail number
12  ActualElapsedTime   in minutes
13  CRSElapsedTime  in minutes
14  AirTime in minutes
15  ArrDelay    arrival delay, in minutes
16  DepDelay    departure delay, in minutes
17  Origin  origin IATA airport code
18  Dest    destination IATA airport code
19  Distance    in miles
20  TaxiIn  taxi in time, in minutes
21  TaxiOut taxi out time in minutes
22  Cancelled   was the flight cancelled?
23  CancellationCode    reason for cancellation (A = carrier, B = weather, C     = NAS, D = security)
24  Diverted    1 = yes, 0 = no
25  CarrierDelay    in minutes
26  WeatherDelay    in minutes
27  NASDelay    in minutes
28  SecurityDelay   in minutes
29  LateAircraftDelay   in minutes

Thanks


texts: How to encode data for a semi-supervised Adversial Autoencoder (AAE)?

texts:
I'm trying to recreate the model described in section 5 of the AAE paper.

I'm having trouble understanding the architecture of the encoder so I could incorporate both the input image and the class label and get the correct latent output for both.

Currently I have an encoder that simply separates the two as follows:

    def build_encoder(self):

    input_img = Input(shape=self.img_shape)
    input_class = Input(shape=self.num_classes,)
    """ Features first"""
    encoder_f = Flatten()(input_img)
    encoder_f = LSTM(512, activation=LeakyReLU(alpha=0.2))(encoder_f)
    encoder_f = LSTM(512, activation=LeakyReLU(alpha=0.2))(encoder_f)
    # For feature extraction
    features = Dense(self.latent_dim, activation='linear')(encoder_f)
    """ Then labels"""
    encoder_c = Flatten()(input_class)
    encoder_c = LSTM(8, activation=LeakyReLU(alpha=0.2))(encoder_c)
    encoder_c = LSTM(8, activation=LeakyReLU(alpha=0.2))(encoder_c)
    # For signal classification
    classes = Dense(self.num_classes, activation='softmax')(encoder_c)

    return Model([input_img, input_class], [classes, features])

But it doesn't sit quite right with me, and I don't think that's what the authors intended. Can I simply run both the flattened input image and label vector through the same encoder like

    imgs= Flatten()(input_img)
    labels = Flatten()(input_class)
    encoder = LSTM(512, activation=LeakyReLU(alpha=0.2))(imgs)(labels)
    encoder = LSTM(512, activation=LeakyReLU(alpha=0.2))(encoder)

    features = Dense(self.latent_dim, activation='linear')(encoder)
    classes = Dense(self.num_classes, activation='softmax')(encoder)

    return Model([input_img, input_class], [classes, features])

And get the correct latent output?

EDIT:
On a closer reading, section A.2.1 of the original paper describes the encoder architecture:

  The q(y|x) and q(z|x)
  share the first two 1000-unit layers of the encoder. The dimensionality of both the style and label
  representation is 10. On the style representation, we impose a Gaussian distribution with the standard
  deviation of 1. On the label representation, we impose a Categorical distribution

This implies that my original implementation was not correct, but the alternative of running both inputs through the same encoding layers is.


texts:
The paper Going deeper with convolutions describes GoogleNet which contains the original inception modules:

The change to inception v2 was that they replaced the 5x5 convolutions by two successive 3x3 convolutions and applied pooling:

What is the difference between Inception v2 and Inception v3?


texts:
I know that cross validation is used to find the best hyperparameters that minimize the average error. For example, the number of neurons that minimize the average error of cross-validation is estimated in ELM.

But I would like to know how I can apply cross-validation (eg K-Fold) in EM-ELM networks:

Feng, G., Huang, G. B., Lin, Q., & Gay, R. (2009). Error minimized extreme learning machine with growth of hidden nodes and incremental learning. IEEE Transactions on Neural Networks, 20 (8), 1352-1357.

Where the architecture is estimated automatically using a training set. What is the best and most standard way to apply cross-validation in EM-ELM? and is it possible to incorporate the cross-validation process into the growth of the network?


texts: Maybe wrong values for precision and recall

texts:
I'm trying to do some data mining with RapidMiner studio.
I've applied the K-nearest neighbor algorithm with different values of K.
As I expected, accuracy increase and after K=5, it decrease. But I cannot understand why value of recall for Basic increase (as I expected) but recall for Premium decrease. The same for values of precision.

Below my results:
Basic and Premium are the values of my class label

With K=5

True Basic
Class recall: 91.83%
Class precision: 81.18%

True Premium
Class recall: 32.87%
Class precision: 56.07%

With K=2

True Basic
Class recall: 81.99%
Class precision: 82.94%

True Premium
Class recall: 32.87%
Class precision: 45.20%


texts:
I'm learning Random Forest Classifier from a video, where the instructor got a score of 0.44, while I'm getting 0.9985 ( But actually it's perfect). Did I overfit it? If so what is the next step? Shouldn't it 'forget' and 'relearn' every time I compile the code again?
Please check the code below:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

bird = pd.read_csv('image_attribute_labels.txt', sep= '\s+', header=None,
        error_bad_lines=False , warn_bad_lines = False,
        usecols= [0,1,2], names =['imgid', 'attid', 'present'])

bird2 = bird.pivot(index = 'imgid', columns= 'attid', values ='present')

imglbl = pd.read_csv('image_class_labels.txt', sep=' ', header=None, names= ['imgid', 'label'])

imglbl =imglbl.set_index('imgid')

#merging the two arrays
df= bird2.join(imglbl)
#Shuffling the array
df= df.sample(frac=1)

df_att = df.iloc[:,:312]
df_lbl = df.iloc[:,312:]

#choosing the train and test data
df_train_att = df_att[:8000]
df_train_lbl = df_lbl [:8000]
df_test_att = df_att[8000:]
df_test_lbl = df_lbl[8000:]

df_train_lbl = df_train_lbl['label']
df_test_lbl = df_test_lbl ['label']

clf= RandomForestClassifier(max_features=50 , random_state= 0, n_estimators= 100)
clf = clf.fit( df_train_att, df_train_lbl)

print(df_train_lbl.head())
print(clf.predict(df_train_att.head()))
score = clf.score (df_train_att, df_train_lbl)
print (score)

The output for three times was as follow:

1:

imgid              #]
4499     78        #|
7442    127        #|
3200     56        #| The actual data
7271    125        #|
2601     46        #]
Name: label, dtype: int64
[ 78 127  56 125  46] # The predicted ones
0.99875 # The score

2:

imgid
10982    187
11632    198
1536      28
5449      94
8503     145
Name: label, dtype: int64
[187 198  28  94 145]
0.9985

3:

imgid
6782     116
10906    186
2465      43
6660     114
9257     158
Name: label, dtype: int64
[116 186  43 114 158]
0.9985


texts: NER with LSTM - How to recognize person names that are not part of the vocabulary?

texts:
I am learning Named Entity Recognition and going through posts similar to this one:
Named-Entity Recognition (NER) using Keras Bidirectional LSTM
So the sentences are fed into the model as a sequence of integers - every int corresponding to the index in the vocabulary - from what I understand this is how the embedding layer works.
My question is - does that mean the model would not be able to recognize a person's name if it doesn't exist in the vocabulary?
For example, from the sentence:
"John Doe went for a walk"
given John Doe is in the vocabulary, it will be recognized as a person name but the sentence:
"Unknown Name went for a walk"
will not be properly tagged if Unknown Name is not in the vocab?
To me, this would be a little strange as Unknown Name is in the same context as John Doe so I was hoping to be able to somehow tag it properly.
I'm obviously lacking knowledge here so I'd be very grateful for any suggestions and reference materials.


texts:
I'm working on a project and need resources to get me up to speed.

The dataset is around 35000 observations on 30 or so variables.  About half the variables are categorical with some having many different possible values, i.e. if you split the categorical variables into dummy variables you would have a lot more than 30 variables.  But still probably on the order of a couple of hundred max.  (n>p).

The response we want to predict is ordinal with 5 levels (1,2,3,4,5).  Predictors are a mix of continuous and categorical, about half of each.  These are my thoughts/plans so far:
1.  Treat the response as continuous and run vanilla linear regression.
2.  Run nominal and ordinal logistic and probit regression
3.  Use MARS and/or another flavor of non-linear regression

I'm familiar with linear regression.  MARS is well enough described by Hastie and Tibshirani.  But I'm at a loss when it comes to ordinal logit/probit, especially with so many variables and a big data set.

The r package glmnetcr seems to be my best bet so far, but the documentation hardly suffices to get me where I need to be.

Where can I go to learn more?


texts: What should I use as training data for base (level 1) classifiers in ensembling?

texts: Can I just take all training data that I have, train the base models on them and then take their results and use them for training level 2 model? Is this a good practice, or should it be done differently?


texts: specific feature engineering for specific algorithm

texts:
Days ago,One AI financial service provider offered us a lesson and mentioned that you are supposed to perform specific feature engineering according to the specific algorithm you are using.For example,when using logistics regression,fitting more features（uncorrelated）like binning the continuous variable into discrete ones are often suggested.Because logistics regression is a simple algorithm and we try to raise dimension in the way that samples will be separated better.

I searched a lot(maybe not yet),most of materials are "why/what feature engineering important","scaling/standardization/binning continuous variable","dealing with null value" or some theoretical comments with no discrete manipulation.

why and how the specific feature engineering should work on specific algorithm.Or any advice on this saying,is it right or wrong?what do you think.any comments are appreciated.

（I am not good at English,sorry about that if I am not clear enough）

I am not looking forward a detailed answer,some deep thinking about this part is good.


texts:
I have a big list of mail addresses (around one million) and I want to use/adapt/create an algorithm to find similarities between them (basically to cluster them).

All the algorithms I've checked so far e.g. n-grams variants, bag of words etc. are only used for clustering strings that are part of a context but here is not the case.

So far I've tried Levensthein, LCS. Both of them are pretty accurate but I'm interested in an algorithm that uses machine learning techniques, if such an algorithm exists for this task. Also they are pretty slow for my dataset.


texts: Recommendation - item based vs user based

texts:
I have one clarification -

First the definitions-

User-based: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.

Item-based: Calculate similarity between items and make recommendations. Items usually don't change much, so this often can be computed offline.

Now the question - in item based CF the similarity between items are tracked via user behavior - since user behavior is changing will it not impact the similarity between items?


texts:
I want to analyse the Price situation for flat/appartements in my City.

How can I process the Text from flat ads (like in newspapers or online with its typical structure!) to get Features like size or number of rooms?

Is it simple to take a Package in R or Python (what is suitable?) and to Train the typical structure?


texts: After the training phase, is it better to run neural networks on a GPU or CPU?

texts:
My understanding is that GPUs are more efficient for running neural nets, but someone recently suggested to me that GPUs are only needed for the training phase. Once trained, it's actually more efficient to run them on CPUs.

Is this true?


texts: why the RFE to select features is changing when the os or computer is changed

texts:
I used the the feature selection method: RFE to select feature for features. Now I want to select 20 features(there are about 50 features or more) with LGBM model as following code: But I found that the features(sel.ranking_) are different when the os or os version, or computer is changed. I don't know what caused the changing. More, how to solve it that features selection is fixed. Thanks!

gbm = lgb.LGBMClassifier(
    boosting_type='gbdt',
    objective='binary',
    learning_rate=0.01,
    colsample_bytree=0.9,
    subsample=0.8,
    random_state=21,
    n_estimators=200,
    num_leaves=18)
sel = RFE(gbm, step=1, n_features_to_select=20, verbose=1)
sel.fit(X_train, y_train)
print (sel.ranking_)


texts: How to correctly infer vectors in Gensim doc2vec?

texts:
I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec.

I have a dataframe df with a feature, called name, and composed of two subsets train and test.

df = train + test

My aim is to find the most similar name in train given a name in test.
For doing this I have to train the doc2vec model, and I have two possible choices:

train the model on the entire df and then infer the most similar name by model.infer_vector() on test.
train the model on train, letting out test, and then use model.infer_vector() on test.

I suppose that the correct procedure is first one, but I am not sure.

Also, so doing, there is the possibility that the most similar name  given test will be again in test and not in train.


texts: What version of spark in latest Cloudera QuickStart VirtualBox?

texts: I am thinking to install the quick start virtual box of Cloudera, search on Cloudera download page, but unable to find the spark version in the latest Cloudera QuickStart virtual box, can any have knowledge of this, please let me know.


texts: R, xgboost: eval_metric for count:poisson

texts: I wonder what are the recommended eval_metrics for count:poisson as objective in xgboost in R?


texts: Softmax: Different output scikit-learn and TensorFlow

texts:
I'm trying to learn a simple linear softmax model on some data. The LogisticRegression in scikit-learn seems to work fine, and now I am trying to port the code to TensorFlow, but I'm not getting the same performance, but quite a bit worse. I understand that the results will not be exactly equal (scikit learn has regularization params etc), but it's too far off.

total = pd.read_feather('testfile.feather')

labels = total['labels']
features = total[['f1', 'f2']]

print(labels.shape)
print(features.shape)

classifier = linear_model.LogisticRegression(C=1e5, solver='newton-cg', multi_class='multinomial')
classifier.fit(features, labels)
pred_labels = classifier.predict(features)

print("SCI-KITLEARN RESULTS: ")
print('\tAccuracy:', classifier.score(features, labels))
print('\tPrecision:', precision_score(labels, pred_labels, average='macro'))
print('\tRecall:', recall_score(labels, pred_labels, average='macro'))
print('\tF1:', f1_score(labels, pred_labels, average='macro'))

# now try softmax regression with tensorflow
print("\n\nTENSORFLOW RESULTS: ")

## By default, the OneHotEncoder class will return a more efficient sparse encoding.
## This may not be suitable for some applications, such as use with the Keras deep learning library.
## In this case, we disabled the sparse return type by setting the sparse=False argument.
enc = OneHotEncoder(sparse=False)
enc.fit(labels.values.reshape(len(labels), 1)) # Reshape is required as
Encoder expect 2D data as input
labels_one_hot = enc.transform(labels.values.reshape(len(labels), 1))

# tf Graph Input
x = tf.placeholder(tf.float32, [None, 2]) # 2 input features
y = tf.placeholder(tf.float32, [None, 5]) # 5 output classes

W = tf.Variable(tf.zeros([2, 5]))
b = tf.Variable(tf.zeros([5]))

# Construct model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax

clas = tf.argmax(pred, axis=1)

# Minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()

# Start training
with tf.Session() as sess:

   # Run the initializer
   sess.run(init)

   # Training cycle
   for epoch in range(1000):
      # Run optimization op (backprop) and cost op (to get loss value)
    _, c = sess.run([optimizer, cost], feed_dict={x: features, y: labels_one_hot})

# Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
class_out = clas.eval({x: features})

# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("\tAccuracy:", accuracy.eval({x: features, y: labels_one_hot}))
print('\tPrecision:', precision_score(labels, class_out, average='macro'))
print('\tRecall:', recall_score(labels, class_out, average='macro'))
print('\tF1:', f1_score(labels, class_out, average='macro'))

The output of this code is

(1681,)
(1681, 2)
SCI-KITLEARN RESULTS:
    Accuracy: 0.822129684711
    Precision: 0.837883361162
    Recall: 0.784522522208
    F1: 0.806251963817

TENSORFLOW RESULTS:
    Accuracy: 0.694825
    Precision: 0.735883666192
    Recall: 0.649145125846
    F1: 0.678045562185

I inspected the result of the one-hot-encoding, and the data, but I have no idea why the result in TensorFlow is much worse.


texts:
I am interested to use multivariate regression with LSTM (Long Short Term Memory).
As an example, we can take the stock price prediction problem, where the price at time t is based on multiple factors (open price, closed price, etc.). Using this information we need to predict the price for t+1.
Another example can be the house price prediction (which depends on factors like, age of house, area, no. of beds, no. of baths, etc). Using this information, we can predict the price of a given house.

I have found many examples where they just used a single variable regression, but I am interested to use multiple features.

Has anyone tried this and can anyone point out the right direction for this?

I tried the following but have no idea if this is the efficient way.

# Importing the training set
training_set = pd.read_csv('Google_Stock_Price_Train.csv').iloc[:,1:3].values#has fields: Date,Open,High,Low,Close,Volume
# Feature Scaling
from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler()
training_set_scaled = sc.fit_transform(training_set)
X_train = training_set_scaled[0:1257]
y_train = training_set_scaled[1:1258]
# Reshaping
X_train = np.reshape(X_train, (1257, 1, 2))
X_train, y_train = np.array(X_train), np.array(y_train)
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 2))
from keras.models import Sequential
from keras.layers import Dense, LSTM
regressor = Sequential()
regressor.add(LSTM(units = 4, activation = 'sigmoid', input_shape = (None, 2)))
regressor.add(Dense(units = 2))
regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error')
regressor.fit(X_train, y_train, batch_size = 32, epochs = 100)

Thanks.


texts: How to calculate the evaluation metrics (i.e., F1 score) in leave one subject out cv when a subject belongs to single class only

texts: I have dataset of 10 subjects. the  dataset has 4 classess. 0,1,2 and 3. The distribution of  classes are not same. For example subject 1 does not have 1,2 and 3. It belongs to zeros class. currently I am evaluating the model using leave one subject out.In such scenario how to calculate the evaluation metrics like F1 score. I  tried the sklearn classification report but for  some subjects there are  only one class. shall I consider the macro F1 score?


texts: Residuals in a gradient boosted classification

texts:
I know that we iteratively model the residuals in case of a gradient boosted regression problem. The intuition is very well explained at kaggle.

Can someone explain what are the residuals that are modeled in case of a classification scenario?


texts: Binary classification of multiple Sequences using Keras

texts:
I am trying to classify multiple independent sequences using Keras. My data looks like this (example with different stocks and their values).
  _stock     2010   2011   2012   2013   2014
----------- ------ ------ ------ ------ ------
 foo          100    200    250    300    400
 bar           50    100    100     50     25
 pear         100    250    250    300    400
 raspberry    100    200    300    400    500
 banana        50     20     10     10      5

I would like to classify the data like shown in the following structure. The labels are already pre-defined for each stock (supervised learning).
  _stock          label
----------- -----------------
 foo         0 (not falling)
 bar         1 (falling)
 pear        0 (not falling)
 raspberry   0 (not falling)
 banana      1 (falling)

Finally, I would also like to predict the next timestep, if possible.
  _stock     2015
----------- ------
 foo          450
 bar           10
 pear         500
 raspberry    600
 banana         1

Currently I'm just using a bunch of Dense Layers which is working fine, but I don't think that I'm not utilizing the relationship between each column in the right way with this setup. Furthermore I don't think that a prediction is possible with this setup.
# current network
from keras.models import Sequential
n_timesteps = len(data.columns)

model = Sequential()
model.add(Dense(100, activation="relu", input_dim=n_timesteps))
model.add(Dense(100, activation="relu"))
model.add(Dense(1, activation="sigmoid"))
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

model.fit(x_train, y_train, epochs=100, validation_data=(x_test, y_test))


texts: I am trying to understand the abstract details that explain how h2o is faster than R and SAS for data science computations.


texts: De-Normalizing Predicted Values from Neural Network

texts:
When fitting the neural network model I normalized the data using the min/max formula:

normalized = (data - min(data)) / (max(data) - min(data))

I fitted the neural network model and got the predicted values (values ranging from 0 to 1, since the data was normalized).

Now I would like to get the Root-Mean-Square-Error (RMSE).
I could calculate it with predictions and the normalized data, but since I want to compare it to RMSE from some other models (where I didn't normalize the data), the result doesn't really help me.

So I tried to de-normalize the predicted data but am really not sure if this is the right way of doing it (it just seemed right, and the RMSE looks like it could be correct).

This is how I did it:

nn = neuralnet (function, data = train_normalized)

Predictions

prediction_train = predict (nn, train_normalized)
prediction_test = predict (nn, test_normalized)

In-sample RMSE

predicted_de_normalized_train = predicted_train * (max(data_train) - min(data_train)) + min(data_train)

#data_train is the original train data (un-normalized) on which the model is based

sqrt(mean((predicted_de_normalized_train - data_train)^2))

Out-of-Sample RMSE

predicted_de_normalized_test = predicted_test * (max(data_test) - min(data_test)) + min(data_test)

#data_train is the original test data (un-normalized)

sqrt(mean((predicted_de_normalized_test - data_test)^2))

Thanks for the help


texts:
I recently discovered a new R package for connecting to the LinkedIn API. Unfortunately the LinkedIn API seems pretty limited to begin with; for example, you can only get basic data on companies, and this is detached from data on individuals. I'd like to get data on all employees of a given company, which you can do manually on the site but is not possible through the API.

import.io would be perfect if it recognised the LinkedIn pagination (see end of page).

Does anyone know any web scraping tools or techniques applicable to the current format of the LinkedIn site, or ways of bending the API to carry out more flexible analysis? Preferably in R or web based, but certainly open to other approaches.


texts: Optimizing decision threshold on model with oversampled/imbalanced data

texts:
I'm working on developing a model with a highly imbalanced dataset (0.7% Minority class). To remedy the imbalance, I was going to oversample using algorithms from imbalanced-learn library. I had a workflow in mind which I wanted to share and get an opinion on if I'm heading in the right direction or maybe I missed something.

Split Train/Test/Val
Setup pipeline for GridSearch and optimize hyper-parameters (pipeline will only oversample training folds)
Scoring metric will be AUC as training set is balanced at that point
Since model was trained on balanced dataset, it will probably be very conservative and predict a lot of false positives
Taking above into consideration, model will be calibrated to have more accurate probabilities (CalibratedClassifierCV)
View precision/recall curve with calibrated probability thresholds on validation set and determine optimal point

Does this process sound reasonable? Would appreciate any feedback/suggestions


texts: What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?


texts:
The penalty term for Elastic Net Regression is written as:
$\lambda_1  \|\theta\|_1 + \lambda_2\|\theta\|_2^2$
How are the values of lambda's calculated if the slider is moved from the right to the left?
I have read most of the material available on the internet on Elastic Net Regression. Here, they define $\alpha=\lambda_2/(\lambda_1 + \lambda_2)$. If the value of $\lambda_2 $ and $\alpha$ is given, then $\lambda_1$ is calculated. In Orange the slider is provided.


texts: How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

texts:
First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library. I was thinking of averaging all of the Word Piece embeddings for each document so that each document has a unique vector. I would then use those vectors for clustering. Please feel free to comment if you think this is not a good idea, or if I am missing something or not understanding something.
The issue that I see with this is that you are only using the first N tokens which is specified by max_length in Hugging Face library. What if the first N tokens are not the best representation for that document? Wouldn't it be better to randomly choose N tokens, or better yet randomly choose N tokens 10 times?
Furthermore, I realize that using the WordPiece tokenizer is a replacement for lemmatization so the standard NLP pre-processing is supposed to be simpler. However, since we are already only using the first N tokens, and if we are not getting rid of stop words then useless stop words will be in the first N tokens. As far as I have seen, in the examples for Hugging Face, no one really does more preprocessing before the tokenization.
[See example below of the tokenized (from Hugging Face), first 64 tokens of a document]
Therefore, I am asking a few questions here  (feel free to answer only one or provide references to papers or resources that I can read):

Why are the first N tokens chosen, instead of at random? 1a) is there anything out there that randomly chooses N tokens perhaps multiple times?
Similar to question 1, is there any better way to choose tokens? Perhaps using TF-IDF on the tokens to at least rule out certain useless tokens?
Do people generally use more preprocessing before using the Word Piece tokenizer?
To what extent does the choice of max_length affect performance?
Why is there a limit of 512 max length in Hugging Face library? Why not just use the length of the longest document?
Is it a good idea to average the WordPiece embeddings to get a matrix (if you want to do clustering)?
Is it a good idea to use BERT embeddings to get features for documents that can be clustered in order to find similar groups of documents? Or is there some other way that is better?

original:
'Trump tries to smooth things over with GOP insiders. Hollywood, Florida (CNN) Donald Trump\'s new delegate guru told Republican Party insiders at a posh resort here on Thursday that the billionaire front-runner is recalibrating the part "that he\'s been playing" and is ready
tokenized:
['[CLS]',
 'trump',
 'tries',
 'to',
 'smooth',
 'things',
 'over',
 'with',
 'go',
 '##p',
 'insider',
 '##s',
 '.',
 'hollywood',
 ',',
 'florida',
 '(',
 'cnn',
 ')',
 'donald',
 'trump',
 "'",
 's',
 'new',
 'delegate',
 'guru',
 'told',
 'republican',
 'party',
 'insider',
 '##s',
 'at',
 'a',
 'po',
 '##sh',
 'resort',
 'here',
 'on',
 'thursday',
 'that',
 'the',
 'billionaire',
 'front',
 '-',
 'runner',
 'is',
 'rec',
 '##ali',
 '##bra',
 '##ting',
 'the',
 'part',
 '"',
 'that',
 'he',
 "'",
 's',
 'been',
 'playing',
 '"',
 'and',
 'is',
 'ready',
 '[SEP]']


texts: Different activation function in same layer of a Neural network

texts:
My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers.

Suppose I have 3 relu units in the starting and 3 tanh after that and other activation function I the same hidden layer and for the other hidden layers I am scaling all the nodes with the same scale (decreasing/increasing) and the arrangement and order of the activation function is not changing.


texts: Single layer multiple input neural network?

texts:
I am implementing single layer neural network using stochastic gradient descent.
When I train the model for single input it gives the answer correctly. Now when I use the second input to update my weights two things are happening:

Weights obtained from previous iteration are updated due to which my previous answer is lost.

Weights obtained from this cycle are not able to give correct output.
If this keep happening how I can fit the whole data set?

My weight updating routine is as follows:
$W = W - (alpha) * (input) * (delta)$
Edit:
I have trained my model upto 1000 iterations. Now when i predict the output using training data it only shows 80% accuracy which is same as accuracy obtained with test data.
Why my trained model not converging or may be overfitting?
input_nodes = 4
hidden_nodes = 10
output_nodes = 2
learning_rate = 0.1


texts: Why should the data be shuffled for machine learning tasks

texts:
In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find any valuable reason for shuffling data.

I have read this post here discussing when we need to shuffle data, but it is not obvious why we should shuffle the data. Furthermore, I have frequently seen in algorithms such as Adam or SGD where we need batch gradient descent (data should be separated to mini-batches and batch size has to be specified). It is vital according to this post to shuffle data for each epoch to have different data for each batch. So, perhaps the data is shuffled and more importantly changed.

Why do we do this?


texts: What is a good hardware setup for using Python across multiple users

texts:
This question is likely somewhat naive.  I know I (and my colleagues) can install and use Python on local machines. But is that really a best practice?  I have no idea.

Is there value in setting up a Python "server"?  A box on the network where we develop our data science related Python code.  If so, what are the hardware requirements for such a box?  Do I need to be concerned about any specific packages or conflicts between projects?


texts: Pandas time series optimization problem: add year

texts:
I have a pandas DataFrame containing a time series column. The years are shifted in the past, so that I have to add a constant number of years to every element of that column.

The best way I found is to iterate through all the records and use

x.replace(year=x.year + years)  # x = current element, years = years to add

It is cythonized as below, but still very slow (proofing)

cdef list _addYearsToTimestamps(list elts, int years):
    cdef cpdatetime x
    cdef int i
    for (i, x) in enumerate(elts):
        try:
            elts[i] = x.replace(year=x.year + years)
        except Exception as e:
            logError(None, "Cannot replace year of %s - leaving value as this: %s" % (str(x), repr(e)))
    return elts

def fixYear(data):
    data.loc[:, 'timestamp'] = _addYearsToTimestamps(list(data.loc[:, 'timestamp']), REAL_YEAR-(list(data[-1:]['timestamp'])[0].year))
    return data

I'm pretty sure that there is a way to change the year without iterating, by using Pandas's Timestamp features. Unfortunately, I don't find how. Could someone elaborate?


texts: Is it a good practice to always apply `ReduceLROnPlateau()`, given that models benefit from reducing learning rate once learning stagnates?

texts:
The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates.

Is it a good practice to always apply ReduceLROnPlateau()? What are some situations, if any, to not apply ReduceLROnPlateau()?


texts: Automated tools to generate synthetic training images out of synthetic 3D models and 2D backgrounds

texts:
I have 3D meshes and textures for a dozen of objects, which have to be detected in synthetic images. I have 2D textures of backgrounds these objects will be visible in front of. Object detection will be performed in 2D images taken inside a 3D simulation (all data is synthetic).
In order to get the training data for DNN, I would over impose varying viewpoint, scale and lighting 2D images of my objects rendered from 3D meshes, over all available background textures at different locations. I'm sure, this obvious method was used by others countless times in the past and presumably there are some tools to automate it. So far I found only a few cases of abandonware. Can someone point me in the right direction and post a link to an easy to use tool, which can do what I described above?


texts: Using keras tokenizer on Spark DataFrame. Column is not iterable

texts:
I am having trouble using the Tokenizer

The code is

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer()
tokenizer.fit_on_texts(df.descricao_despesa)

X = tokenizer.texts_to_sequences(df.descricao_despesa)

maxlen = 100

X= pad_sequences(df.descricao_despesa, padding='post', maxlen=maxlen)

I tried to cast column descricao_despesa to String but still doest work.

If i try to cast to StringType i receive

from pyspark.sql.functions import col
df = df.withColumn("descricao_despesa", col("descricao_despesa").cast('StringType'))

ParseException: u'\nDataType stringtype() is not supported.(line 1, pos 0)\n\n== SQL ==\nStringType\n^^^\n

I am using spark 2.0 and Python 2.7


texts:
The paper I read is Glorot et al (2010). And the math part is in Section 4.2.1.
Formula (5) and (10) make sense to me but I cannot derive formula (6) and (7) myself from (2) and (3).

I found many tutorials on the internet used the formula
$$Var[XY] = Var[X]Var[Y] + (E[X])^2 Var[Y] + Var[X](E[Y])^2$$
which requires the independence between X and Y.

But in formula (2) and (3) the gradients are not independent of W and Z, because all of them are related to each other through the output from the last layer.

I would appreciate it if anyone can give me a derivation of the formula (6) and (7).
Thanks in advance.


texts: Hyperparameter tuning results yield no improvement over spot-check

texts:
There is a balanced binary classified dataset as seen below.

Things I have tried using RandomForestClassifier as chosen model:

TimeSeriesSplit with n_splits=3 and 10
MinMaxScaler, MaxAbsScaler, StandardScaler, RobustScaler, QuantileTransformer, PowerTransformer

Snippet of RandomizedSearchCV set-up:

scorer = {...,
'precision': make_scorer(precision_score, average = 'weighted'),
...,
}
RandomizedSearchCV(model, grid, scoring=scorer, cv=TimeSeriesSplit(n_splits=10), refit='precision', verbose=10)

However despite all these efforts, precision (the main metric I have chosen to use) are all very similar. Inital spotcheck precision results were: ~.55-.56. Hyperparameter tuning results in the same range too, no improvement at all! Other "curious" metrics like F0.5 score and recall all have no improvements too with little variation.

The biggest noticeable difference so far are n_splits 3 and 10. With all else equal, the range of precision is bumped up to ~.57-.59 with n_splits=10. Hyperparameter tuning results also within ~.57-.59!

I am stumped. Does anyone have ideas on how to find out what may be the issue(s)? I can provide more information if needed.

Edit 1 (grid):

# Number of trees in random forest
n_estimators = [x for x in range(400, 2000, 200)]

# Number of features to consider at every split
max_features = ['log2']

# Maximum number of levels in tree
max_depth = [x for x in range(10, 110, 10)]
max_depth.append(None)

# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]

# Minimum number of samples required at each leaf node
min_samples_leaf = [4]

# Method of selecting samples for training each tree
bootstrap = [True, False]


texts: Keras Callback example for saving a model after every epoch?

texts: Can someone please post a straightforward example of Keras using a callback to save a model after every epoch? I can find examples of saving weights, but I want to be able to save a completely functioning model after every training epoch.


texts:
I have a dataset where I need to explore using unsupervised technics (clustering and association rules). What are the best strategies to discretize the numeric attributes? Also, does this (attribute discretization) makes sense for clustering (using K-Means)?

Since I'm using weka, I know that I can just use the Discretize filter, with findNumBins option or useEqualFrequency. But are there other strategies?

Also, does it makes sense to use discretization on the attributes, and then look for clusters (using K-Means) or not? (I suspect that it doesn't make that much sense, but I just want to confirm)

Thanks!


texts: Blocked in my project with reinforcement learning.. any help?

texts:
I would like advice on a problem I'm trying to solve. I'm stuck...
Here's an example: I have continuous data from a person such as what was eaten for a week (calories, carbohydrates, protein and fat) and also the goal such as the desired weight.
I want to do an action (without prior training) based on the values and the goal. The actions will consist of decreasing/increasing the number of carbohydrates, lipids, proteins or calories.
I have started to use reinforcement training but I really don't know if it is the best approach? The goal of reinforcement learning will be to reach the desired body weight with all of the actions.
One action can have a positive impact for a while and maybe a negative impact afterwards.
For the decision making algorithm, I created my own... but I would like to use something else that is better... I'm stuck with the choice.
Thanks!


texts: Identifying users based on smartphone data (Google)

texts: I've heard that Google has now a technique to identify users based on the smartphone touch input (how the user is using the phone). I have found nothing about that online. Is there a paper or some article available explaining how this is done (e.g. what features are used)?


texts: What is a good method for detection of rare occurencies of speech in noisy audio data?

texts:
I have some audio recordings (with relatively static but noisy background, e.g., wind in an open area) with small number of short occurrences of speech (~1% of the total audio duration).

What would be a good method to detect the speech occurences in an unsupervised manner?

I have tried simple thresholding on a spectrogram, but this is problematic since:

The intensity of the background can wary with time (i.e. noise is louder sometimes)
Different speech segments need not to be similar to each other
Often, speech is too quiet (compared to the average loudness of the background) and is overlaid by noise

This may seem like quite a hard task, however I can easily notice the speech segments by listening to the audio/looking at the spectrogram, since spectrogram of speech has some distinct structure (although it is non-trivial to rely on the structure for detection as it is still quite non-regular).

Note that I just want to detect intervals with something sounding like human speech (or, say, something distinct enough from the background, since data typically has no other sound sources besides background/speech).


texts: How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif

texts:
I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem.

As I understand from the calculation of ANOVA from basic statistics, we should have at least 2 samples for which we can calculate the ANOVA value. So does this mean in the Sklearn implementation that these samples are taken from within each feature? What exactly do these samples represent in the case of feature selection for this problem?

I tried to setup a very simple example which I have listed below, but I am still struggling to understand what the ANOVA value really means here? I'm also struggling to understand how to calculate this by hand, which usually helps me see what is happening on the inside.

In this example, repayment status 0 means the loan is repaid and 1 means it defaulted. I have only supplied 5 rows of data to keep it simple.

The code and results are as follows:


texts: If a script is launched from command line, I use "KERAS_BACKEND" env var to switch between Theano and  Tensoflow. What can be done to switch backend if script is running in a notebook ?


texts: Using tensorflow for any type of dataset

texts:
I have a problem that I have been working on with logistic regression which involves classifying clients based on their income and socio-demographic variables, all of which are categorical into those who have defaulted on their credit and those who haven't.

I have recently had tensorflow recommended as a package to use for neural networks, however on their website https://tensorflow.rstudio.com/tutorial_mnist_beginners.html they discuss tensorflow in terms of image classification.

My concrete question is thus:

2) Can tensorflow be used for any type of classification problem, for example, a binary classification of observations based upon categorical variables?

Cheers,
Steve


texts: How can i increase the memory of Jupyter?

texts:
What I have:
I have a data set (35989 rows × 16109 columns) and is unfortunately confidential.
But i receive this error massage:
Unable to allocate 4.32 GiB for an array with shape (16109, 35994) and data type float64

How can i solve this problem?


texts: Are there any graph embedding algorithms like this already?

texts:
I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples.

The question is:
Am I reinventing the wheel?
Does this approach have any practical advantages over existing solutions for embeddings generation?

Yes, I'm aware there are many algorithms for this based on random walks, but this one is pure deterministic linear algebra and it is quite simple, from my perspective.

In short, the algorithm is heavily inspired by PageRank.
Every node is described by its proximity vector which contains the node's closeness number to every other node or some selected subset of the nodes.
The "closeness" is not just the simple shortest distance.

Here's brief explanation from the repo's readme for directed/undirected unweighted graph (the idea generalizes to weighted graphs pretty intuitively):

Every node is assigned a vector. For node i the vector's j-th element is a number representing its closeness to node j (you can think of it as signal strength).
The closeness of the node i to the node j (the "central" node the signal flows from) is defined as a sum of the signal strengths of the neighboring nodes multiplied by the damping factor parameter. (The dumping factor works like a kind of distance penalty)
Signal a node "emits" to other connected nodes equal to the given signal's strength in the node itself divided by the number of edges it is able to emit to.
Initially, the node j is assigned a closeness (to itself) equal to 1.

The whole thing is computed by solving sparse linear systems.

"Signal" strengths from node 4 to other nodes. Note that the signal here travel in direction opposite to edge direction (like "follow" relation on Instagram)


texts: Can Boosting and Bagging be applied to heterogeneous algorithms?

texts: Stacking can be achieved with heterogeneous algorithms such as RF, SVM and KNN. However, can such heterogeneously be achieved in Bagging or Boosting? For example, in Boosting, instead of using RF in all the iterations, could we use different algorithms?


texts: How choosing a value for random_state argument in sklearn.model_selection works?

texts:
I have been using sklearn for quite some time and I understand using the same number say 100 or 200 as a value for the random_state argument will help me to produce the similar train and test data sets. However, how does it work?

Is it some kind of initialization? Do we need to pick a value based on data size? I want to know how random_state works.


texts: Transform large pd.Series into a DataFrame of n columns

texts:
I have a pd.Series with, for instance, n lines. I would like to transform this series in a pd.DataFrame as follows:
Ex:
Input: pd.Series([10,11,12,13,14,15]) and a variable chunk_size = 2 that will be the number of columns.
Target:
    0 | 1
    _   _
    10  11
    12  13
    14  15

The target DataFrame will have a shape of (n / chunk_size) rows by chunk_size columns.
Thanks in advance.


texts: What model should I use to find a common pattern for a specific user group based on the other dimensions?

texts:
I have a big .CSV database of 25k users with various attributes of the last user's activity and events during the past 6 weeks

This is an example of the data:

username        (B)      (C)             (D)        (E)
nicole          524      329             203        787
asteria         197      186             286        120

I want to create a common behavior pattern based on the values of the attributes of each user and run an algorithm to find a common pattern that defines this group's behavior and to find out if there is any correlation in the dimensions values and which dimension define this list of users.
I am fully aware that correlation does not necessarily equal causation.

Now I see several challenges in front of me and would greatly appreciate some input from others, or some good resources to find further information.

What is the model for this problem ?
What kind of Algorithm is the best to deal with this situation?
What is the tools that you recommend to use of the project ?

Any ideas would be great.


texts: How To Perform Time Series Data Patterns Classification

texts:
I'm fairly new to Python, and I want to create a Machine Learning algorithm that classifies Time Series Data Patterns.

The data I will be using comes from an accelerometer-arduino system fashioned into a glove worn in the hands, which writes data on a csv file. I need it to be time series data patterns, so that I can record hand gestures that last about 4 seconds. For example, I want to record a hand wave for five seconds, or I want to record a hand drawing a circle in the air for four seconds.

Then when all is final, what I will be doing is use the machine learning code to classify the live data based on the models created, as they are being performed.
From what I've gathered, what I should do is create a training data by performing the gestures repeatedly. However, I am clueless where to go from there.
Upon the final product, my goals should be these:

Create machine learning scripts that would classify Time Series Data Patterns. Another specific concern is setting the interval of patterns into every 4 seconds.
Use the models or the prediction script to classify matches from live data written to a csv file with the ones created in the model

I don't know how to move forward after creating csv files. I don't understand what algorithm do I need to use, and how do I go about it.

So far, I've understood what models are and what preprocessing is, but I don't know where to go from there. I understand that the training models may be stored into pickle files for reference in the actual comparison, but I'm inexperienced regarding the machine learning algorithm itself.

Any advices and help would really be appreciated. Thank you.


texts: CNN for checking existance of single label

texts:
i am just thinking about training a neural network which uses data of only one single label.

For example: Assuming i have many images which contain a dog. Now i want to teach the network how a dog looks like, without images containing no dogs.

More formal: I want to check for availability of one label, so in theory, i need 2 datasets: Dog and NoDog

The question i am raising to myself now is: How would i define the NoDog-class? I can throw in anything, some random noise or just any unlabelled data which does not contain a dog? But how do i make sure that i
sample the "no-dog"- space correctly? Which features does a "no-dog" image have? The "no-dog" space is huge, it is extremely difficult to describe it, somehow.

I had different ideas:

-Use the same dog-images, but cut out the dogs and fill it with noise

-Train a gan and use the discriminator for the binary classification

-Use tons of unlabelled data

Does anyone know how to tackle this problem in a systematic manner? Thank you


texts:
Why is the regularization parameter not applied to the intercept parameter?

From what I have read about the cost functions for Linear and Logistic regression, the regularization parameter (λ) is applied to all terms except the intercept. For example, here are the cost functions for linear and logistic regression respectively (Notice that j starts from 1):


texts:
I currently tried to figure out which paddings are directly supported by the frameworks:

Tensorflow (tf.nn.conv2d):

padding='VALID': No padding
padding='SAME': 0-padding such that the output has the same size as the input

By applying tf.pad and then padding='VALID' one can get reflect and symmetric padding.

Lasagne (lasagne.layers.Conv2DLayer) (and probably also Theano):

pad='full': A variant of 0-padding which results in a bigger image
pad='same': Like SAME for TF
pad='valid': Like SAME for TF

Caffe (docs): I have no idea which padding they support. As pad_h and pad_w seem to be in convolution_param, I guess padding is supported. Probably (only?) zero-padding.

Hence my question:

Are there publications where CNNs with padding, which was neither VALID nor SAME, or not 0-padding, was used?

For example, I know of the following other options:

reflect
nearest

both of them might have the advantage, that the filter does not detect a border, where no border is.


texts: Is it possible to (de)activate a specific set of cells in jupyter?

texts: I have a jupyter notebook and I would like to perform several runs, the code I want to run depending on the results of the previous runs. I divided my notebook into several cells, and for each run I would like to select which cell should be executed and which one should not. Is there a functionality which looks like "run all cells except those I explicitly deactivate"?


texts: How do i use the Gaussian function with a Naive Bayes Classifier?

texts:
I would like some help with the maths and to check I have understood the algorithm correctly. So I’ve been learning off of a video and I tried my own example. At the end of it I got dodgy results; I think they are “dodgy” because my outputs are not between 0 and 1. If I went with the higher output then yes it has correctly classified my new data but I’m just not sure since the numbers are completely different. I tried by doing the math by hand with a calculator instead of with python but I still got a weird result, so it makes me think that my method is wrong.Here’s what I did, please have a look and tell me what you think:

# data

x = [[65,50],

[62,49],

[130,78],

[124,79],

[54,47],            # the heights/weights of a child

[169,76],              (0), or an adult (1)

[59,58],

[169,79],

[52,52]]

y = [0,0,1,1,0,1,0,1,0]

new = [58, 54]

Okay so here’s what I think I should do:

Work out the prior probability (in this case 5/9 children, 4/9 adults)
Work out the means and variances for each feature and each class:

x̄(C,H) = 58.4        x̄(C,W) = 51.2

x̄(A,H) = 148          x̄(A,W) = 78

σ²(C,H) = 23.44    σ²(C,W) = 14.16

σ²(A,H) = 445.5    σ²(A,W) = 1.5

Compute the Gaussians f(x,x̄, σ²):

f(58, 58.4, 23.44) = 0.08212001659 (child height)

f(54, 51.2, 14.16) = 0.08038037412 (child weight)

f(58, 148, 445.5) = 2.129877168 x10(-6) (adult height)

f(54, 78, 1.5) = 1.343766812 x10(-84) (adult weight)
Multiply the Gaussians for the adult and child classes:

0.08212001659 x 0.08038037412 = 6.600837656 x10(-3) (child Gauss)
2.129877168 x10(-6) x 1.343766812 x10(-84) = 2.862058252 x10(-90) (adult Gauss)

Calculate the probability using bayes theorem:

(child gauss x child prob)/(child gauss x child prob x adult gauss x adult prob)
(adult gauss x adult prob)/(child gauss x child prob x adult gauss x adult prob)

(6.600837656 x10(-3) x 5/9)/(6.600837656 x10(-3) x 5/9 x 2.862058252 x10(-90) x 4/9) = 9.841633571 x10(89)
(2.862058252 x10(-90) x 4/9)/(6.600837656 x10(-3) x 5/9 x 2.862058252 x10(-90) x 4/9) = 341.3788376

...so those were the final probabilities. If i went with the higher one which you are supposed to then the algorithm would've correctly classed the data as a child, but the probabilities were not between 0 and 1 which makes me think this method may not work for other data. I also ran the data through Scikit learn which classified the new data as a child also. Please let me know what you think and if i have been unclear about something, let me know in the comments.


texts: Finding relationships between multiple readings from an IoT sensor network time-series data

texts:
I am trying to find possible relationships between a large sensor network (around 100 sensors) where data is recorded in daily intervals. Each sensor has multiple readings simultaneously (Value_X). Each row contains data in below format.
|Time|Sensor ID|Value_1|Value_2|Value_3|Value_4|
I have worked with Time series data (|Time|Sensor ID|Value_1|) where one sensor records a single data and analysed with "statsmodels" python library and but couldn't find a proper way to reduce 2D (multiple sensors,time vs multiple reading values) case for a such analysis. Any insight or direction would be highly appreciated.


texts: Axis date format for a surface plot in Plotly

texts:
My date format for my axis labels works fine if the chart type is heatmap or contour, but when I make a surface plot, my date format is not taken, and a default format is used instead. I want the date format to show the month and day, leaving out the year. This is because the year goes on the y-axis, while the calendar day goes on the x-axis. I am using the year 2000 as a template for the x-axis dates, but I don't want the labels to say 2000.  My workaround, for now, is to show the calendar date as a value from 0-365, but that makes the chart harder to understand.
How I formatted my labels:
2007   +      +      +      +
2008   +      +      +      +
2009   +      +      +      +
2010   +      +      +      +
     Jan-1  Apr-1  Jul-1  Oct-1

How Plotly formats my labels in spite of my instructions
2007     +      +         +         +
2008     +      +         +         +
2009     +      +         +         +
2010     +      +         +         +
     Jan 2000  Apr 2000  Jul 2000  Oct 2000

My ugly workaround:
2007  +  +  +   +
2008  +  +  +   +
2009  +  +  +   +
2010  +  +  +   +
      0  91 182 274

Some relevant links:

https://plotly.com/javascript/reference/surface/
https://plotly.com/javascript/reference/layout/xaxis/#layout-xaxis-tickformat

Is there any way to get the dates formatted on the x-axis for a surface plot?


texts:
What are the requirements to load the trained model by Keras in Java?

I checked that DeepLearning4J supports Keras models, network architecture and weights can be easily loaded. The only cons are probably that we need to use ND4J backend or it does not matter?

If there is a created model using keras and tensorflow, what is the best way to load it in Java ecosystem?

I tried to use frozen graph script to save tensorflow model, but it cannot be loaded with SavedModel.load. Loading model as a Tensor Graph object seems to be common way, but available examples in Java are chaos especially when input is an image.


texts: Training a Variational Autoencoder (VAE) for Random Number Generation

texts:
I have a complicated 20-dimensional multi-modal distribution and consider training a VAE to learn an approximation of it using 2000 samples. But particularly, with the aim to subsequently generate pseudo-random numbers behaving according to the structure of the distribution. However, my problems are the following:

Is my approach fundamentally or logically flawed? Specifically, because unlike image data, the random numbers are of geometric nature and thus take negative values and could also be considered noisy.
How do I find the right architecture aside from simple trial and error? Obviously, I do not necessarily need 2D-Convolutions. But instead, 1D-Convolutions could be considered a good choice to capture the correlations (i.e. modalities of the distribution). I'm also not sure about how I properly decide on the number of hidden layers and nodes.


texts: How can I infer psychological intentions from email corpus using text mining?

texts: I'm working with Enron email data set, and try to get some psychological intentions about email marketing. I know that we can derive Sentimental analysis like emotions and mode and opinion mining like positive or negative through text data. what else can I do regarding human psychological intentions?


texts: How to remove noise using morphological filtering

texts:
I have two groups of dots that both contain noise between them:

The line that separates the two groups in the picture is diagonal in shape.
I tried to use morphological filtering on this image to remove the noise between these two groups but failed.

This is the code that I tried to run on this image:

from skimage.morphology import opening, square

new_image = opening(image, square(3))

It did remove a little bit of noise, but not enough for them to become two distinct groups.

I am using DBSCAN on the new_image and DBSCAN still treats them as one group.

I would like to know whether I am giving the correct argument to the function opening. Or maybe there is some other way (better way) to remove the noise between the two groups?


texts: What tools do you use to clean corrupt data?

texts:
Customer often send currupt data for analysis. I spent a lot of time in cleaning the data or waiting for a correct dataset.
Can you recommend a tool that can handle the most common curruptions (eg. wrong set quotes)...


texts: What is the difference between NLP and text mining?

texts:
As discussed with Sean in this Meta post, I thought it would be nice to have a question which can help people who were confused like me, to know about the differences between text mining and NLP!

So, what are the differences between nlp and text-mining?

I have included my understanding as an answer. If possible, please explain your answer with a brief example!


texts: Is a dense layer required for implementing Bahdanau attention?

texts:
I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed.

This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then doing several operations on these vectors only. So what is the need of Dense( ) layer. Is the tutorial on 'how does attention work' wrong?


texts: Assessing significance / confidence of a crossvalidated performance measure

texts:
I have a prediction model $P$ and I use some performance measure $I$ to measure $P$'s accuracy.
The distribution of $I$ is unknown (it's a custom metric, which is somehow similar to the precision metric).

My validation prediction is as following:

Randomly split the data to $k$ stratified folds
Fit $k$ models
Estimate each model according to $I$ (which results $k$-crossvalidated values of $I$)
The final model prediction performance is calculated by the average of $k$-crossvalidated $I$ measures.

I would like to perform some significance testing - to be able to say the confidence for the model prediction performance - to what extent I am “sure” in this $I$?


texts:
I'm fiddling around with some data that represent grocery store transactions. The data are in the following form:

Each row represents a final transaction by a customer, with a column for user ID, timestamp of purchase, and basket contents. For example:

| ID | Timestamp  |              Basket           |

| 12 | 2016-04-02 | ['Celery', 'Beets', 'Cheese'] |

The question I'm trying to answer is "How do I suggest an ideal basket to an individual customer?"

Problems I'm dealing with:

I can't seem to think past just suggesting the most frequently ordered items to a customer. Is there some technique that I'm missing here?
Since we have a customer ID, we can have multiple baskets for a customer over time. Because of this, I can't use apriori or eclat as those assume each transaction is independent.
How can I take into account seasonality with whatever technique I choose?

I'd really appreciate some general direction here, I'm having some serious analysis paralysis.


texts: Build Deep Belief Autoencoder for Dimensionality Reduction

texts:
I'm working with a large dataset (about 50K observations x 11K features) and I'd like to reduce the dimensionality. This will eventually be used for multi-class classification, so I'd like to extract features that are useful for separating the data. Thus far, I've tried PCA (performed OK with an overall accuracy in Linear SVM of about 70%), LDA (performed with very high training accuracy of about 96% but testing accuracy was about 61%), and an autoencoder (3 layer dense encoder with 13000 - 1000 - 136 units, respectively, which performed about the same as PCA). I've been asked to try a Deep Belief Network (stack of Restricted Boltzmann Machines) in this problem.
Thus far, I foresee two challenges. First, I have access to a GPU that can be used, but I don't see many implementation of DBNs from the major players in the neural net community (e.g., TensorFlow/Keras, PyTorch), which means that this will need to be implemented on a CPU, bringing up challenge number two. Second, implementing this will require significant memory and will be pretty slow. This brings up my question: Are there any implementations of DBN autoencoder in Python (or R) that are trusted and, optimally, utilize GPU? If not, what is the preferred method of constructing a DBN in Python? Should I use sklearn?


texts: difference between scaling/normalizing data at a specific step

texts:
I am using the MinMaxScaler normalization method, however I have seen various ways that this can be done, I want to know if there is any actual difference between the following:
1. Standardizing/Normalizing the data before splitting the data into train and test
Code 1
scaler = MinMaxScaler() #Normalization
#Transform X and Y values with scaler
x = scaler.fit_transform(x)
y = y.reshape(-1,1)
y = scaler.fit_transform(y)

# Split Data in train and validation
x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.25)

2. Standardizing/Normalizing the data after splitting the data into train and test and then scaling on train and test
# Split Data in train and validation
x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.25)

# created scaler
scaler = MinMaxScaler() #Normalization

# transform training dataset
x_train = scaler.fit_transform(x_train)
# transform test dataset
x_valid = scaler.fit_transform(x_valid)

3. Standardizing/Normalizing the data after splitting the data into train and test. Then fitting on the training set and then scaling on both train and test
# Split Data in train and validation
x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.25)

# created scaler
scaler = MinMaxScaler() #Normalization
# fit scaler on training data
scaler = MinMaxScaler().fit(x_train)

# transform training dataset
x_train = scaler.fit_transform(x_train)
# transform test dataset
x_valid = scaler.fit_transform(x_valid)


texts: How do I read a dat file, for which I don't know its structure?


texts: Equitable selection of users through ranking

texts:
I am looking to take a dataset largely derived of user input in categorical form, this sign up sheet asks for many data points such as age group, race, sign up date, as well as a few others. My goal is to create a weighted system to choose users equitably based on their responses, I've tried a frequency approach but there are pit falls to that, if 65% of the sign ups are White/Caucasian there will be a disproportionate number of those users selected. I looked into various packages such as survey pewmethods R package to try and standardize the approach but that directly corresponds to survey data and this is not a survey problem. I have triedvarious other packages in python relating to OLS, Random Tree, however I do not feel this is a regression based problem( I could be wrong here). I've tried one hot encoding, and the category encoder library in python to little effect. Below is an example with some data to show what I am seeing. I am also not sure how to go about weighing the date as users who sign up earlier should be given a higher score. Any help or direction would be greatly appreciated, apologies for the poor formatting first time using stack exchange.
Edited for Clarity: Please review the below image, I have taken a percent of whole and conducted a count on Race and then  created a column called percent of whole and then taken (1-percentOfWhole) to create an indexcolumn, this  process has been replicated to other categorical variables such as over/under XX age, Ethnicity and Gender. I then took the index column in each of these variables and applied them to the original row level data so JaneDoe received .35 from race, .5 from age, .5 from ethnicity and .45 from sex for a total of 1.8, this was done for thousands of records  then sorted high to low to create a index or elo of sorts to contact these people from most under represented to over represented. To me this feels like too simple of a process and there is a more elegant approach, that removes all bias and can be purely math driven. Right now an obvious pitfall to this approach is there are penalties for certain group types which could lead to bias.


texts: What can I infer from similar accuracy using two different DNN architectures?

texts:
I am working on a 'two-class classification of multivariate time-series' problem. I used two different approaches:

1) Manual time-series feature engineering (such as slope, intercept, variance, etc. using tsfresh python library) followed by a multi-layer perceptron.

2) Conv1D followed by LSTM followed by Dense layers (keras/Python) without any feature engineering.

They both yield very similar results (~40% precision, ~95% recall).

Can I infer anything from the similarity of results from two different architectures?


texts:
I am working on text analysis problem.

Person X can log in his goals and his actions to achieve his goal. Also their  score is calculated based on some formula to measure progress of the goal.

For ex:

Goal1 : Loose weight  ... Score: 70%
Goal2: Finish reading this book.... Score: 10%
Action1: Go to the gym
Action2: Drink water
Action3: Read one page daily

Goals and actions don't have any relationship in my DB, so I joined them based on the person.

I have as an input

+--------------------------+---------------------+--------+-------+
|           Goal           |       Action        | Person | Score |
+--------------------------+---------------------+--------+-------+
| Lose weight              | Go to the gym       | X      |    70 |
| Lose weight              | Drink water         | X      |    70 |
| Lose weight              | Read one page daily | X      |    70 |
| Finish reading this book | Go to the gym       | X      |    10 |
| Finish reading this book | Drink water         | X      |    10 |
| Finish reading this book | Read one page daily | X      |    10 |
+--------------------------+---------------------+--------+-------+

I want to detect what are the good actions that can have high score and what are the bad actions that can have low score.

My Approach:

Apply clustering algorithm on Goals to find similar groups of goals then study their actions.
My Text data dont have the type of action in it, it usually starts with a noun, so I manually created a new column for Actions like below, to make it appear in action form, also to unify the meaning of other actions into one.

+----------------------------------------+--------------------------+
|              Action Title              | Actions (Manually filled)|
+----------------------------------------+--------------------------+
| Sending Video Via whatsapp             | send video               |
| WhatsApp the video                     | send video               |
| WhatsApp Video Broadcasting for Higher | send video               |
| Sending Video Via Whatsapp             | send video               |
| Send video                             | send video               |
| Video Broadcast                        | send video               |
| Send the video                         | send video               |
| Sending Video via Whatsapp             | send video               |
| Share video with 10 contact customers  | send video               |
| New Strategy Owners Contacts           | create strategy contacts |
| New strategy unit contact              | create strategy contacts |
+----------------------------------------+--------------------------+

I used R to build a term document matrix using bigram tokenizer (combination of 2 words).
I found words appearing only in high score (0-75).
I found words appearing only in low score (0-75).
I calculated probability of common words to know if it will occur more in high or low.

Conclusion : I concluded step 4 are result of good actions while step 5 are results of bad actions. The common words are either added to step 4 or 5 depending on probability.

Suggestion

Am I going in the right path?
Is there a ready API or a way to automate step 2.

I did step 2 because when I count the words appearing in either high or low, I will find big number. Otherwise, depending only on bigrams will give me a word frequency of 1 for each word.


texts: Understanding the generality of the NER problem

texts:
Named-entity recognition (NER) is a well-known problem in the NLP literature.
It typically addresses the problem to locate and classify named entities in text, e.g. Organizations and Products.

I'm trying to solve a similar problem but, in my view, a bit more general. Given an input text, I want to be able to comprehensively annotate the whole text; not only specific entities like Actors and Organizations but also higher-level concepts like Conditions of Applicability and Temporal Conditions, e.g.:

The added difficulty is that we have nested "entities", e.g. (from above):
<denotic> must, <temporal> within the specified period </temporal>, notify ... </deontic>

Can this still be formulated as a NER problem? If so, what would be the best type of model to solve this task assuming a dataset of ~ 50 K examples?


texts: Is the "training loop" used in AlphaGo Zero the same as an "epoch"?

texts:
I am confused about the training stage of AlphaGo Zero using the data collected from the selfplay stage.

According to an AlphaGo Zero Cheat Sheet I found, the training routine is:

Loop from 1 to 1,000:

Sample a mini-batch of 2048 episodes from the last 500,000 games
Use this mini-batch as input for training (minimize their loss function)

After this loop, compare the current network (after the training) with the old one (prior the training)

However, after reading the article, I did not see any mentions on how many epochs they used with those mini-batches.

Questions:

Are those 1,000 training iterations the actual epochs of the algorithm? The Keras code would then loosily be translated to:

network.fit(x_train, y_train, batch_size = 2048, epochs = 1000, ...)

Or do they actually have a for loop for the training? The Keras code would then loosily be translated to:

for _ in range(1000):
    x_train, y_train = sample_states_from_past_games(data_from_selfplay)
    network.fit(x_train, y_train, batch_size = ???, epochs = ???, ...)

If it is the second option, I would like to know how many batches and epochs they have used.


texts: Keras model evaluation accuracy vs. observation

texts:
I am a newbie here and trying to make sense out of the scores from model.evaluate from what I am actually seeing in model.predict

I have a created a CNN model for the Google Audio Set data and achieved a 99%+ accuracy on training.

Here is how I do the prediction

model = load_model('model_audioset.h5')
for x, y in unbal_generator:
  score = model.evaluate(x, y, verbose=0)
  pred_y = model.predict(normalized_x)

Here is what I am seeing for one specific iteration of x and y from model.evaluate

model.metrics_names = {list: 2} ['loss', 'acc']
 0 = {str} 'loss'
 1 = {str} 'acc'

score = {list: 2} [0.03851451724767685, 0.9905123114585876]
 0 = {float64} 0.03851451724767685
 1 = {float64} 0.9905123114585876

Here is a readable output from model.predict and comparing it to y

Wondering how Keras came up with an accuracy score of 99.05 for this output? Clearly, the predicted classes are not the same as expected

I am using a binary_crossentropy loss function and sigmoid activation in the predictions layer as classes are NOT mutually exclusive


texts:
I want to implement my own version of the CART Decision Tree from scrach (to learn how it works) but I have some trouble with the Gini Index, used to express the purity of a dataset.

More precisely, I don't understand how Gini Index is supposed to work in the case of a regression tree.

The few descriptions I could find describe it as :

gini_index = 1 - sum_for_each_class(probability_of_the_class²)

Where probability_of_the_class is just the number of element from a class divided by the total number of elements.

But I can't use this definition in the case of regression where I have continuous variables.

Is there something I misunderstood here ?


texts: Can the input and output be different in a variational autoencoder?

texts:
While the autoencoder and the variational autoencoder typically decode back to the same input space that was passed in, I am working on a more generic encoder-decoder structure, where the output is a different representation of the input. Particularly, the input to my system is a sequence of numbers, which is compressed into a latent 'z' vector, but it's decoded to an image. During training, the ground truth image for the input sequence is pre-constructed, and the loss between this 'expected' image and the reconstructed image drive the training.
This encoder-decoder structure does work in my case, but I am curious if I can get more control over the latent space, through a variational sense (example: the beta-VAE which constructs a disentangled latent representation). Does the VAE principle (compress input into a mean and variance, sample from normal distribution etc.) still apply to a case where the input and output are different? In my initial trials, I am having a lot of trouble training a VAE for my task because no matter the input, the encoder is always returning a zero mean and unit variance.


texts: Learning the distribution of a continuous variable using LSTM

texts:
I am trying to implement the following paper : https://arxiv.org/pdf/2006.10701.pdf. In order, to estimate the priors of the hidden states which have continuous values, the authors use a LSTM. I have never done this before, and I was wondering If:

It is possible to directly do this, or should I make an assumption about the distribution and use the reparameterization trick, or is estimating p(z_{i}|z_{i-1}) equivalent to estimating z_{i} ?
In the HalfCheetah environment, the z_{i} is a function of the episod. When trying to estimate p(z_{i}|z_{i-1}) doesn't the way of feeding data affect the learned distribution ?


texts: In the Keras Tokenizer class, what exactly does word_index signify?

texts:
I'm trying to really understand Tokenizing and Vectorizing text in machine learning, and am looking really hard into the Keras Tokenizer class. I get the mechanics of how it's used, but I'd like to really know more about it. So for example, there's this common usage:

tokens = Tokenizer(num_words=SOME_NUMBER)
tokens.fit_on_texts(texts)

tokens returns a word_index, which maps words to some number.
Are the words all words in texts, or are they maxed at SOME_NUMBER?
And are the dict values for word_index the frequency of each word, or just the order of the word?