texts: Is it possible for decision trees to consider less features than in my training set?

texts: I was looking at the decision tree algorithm and I wondered that for example if the training set has 20 features but only 5 features are important and classification can be done by using only them then will my end result decision tree has only 5 features? My understanding is that as decision tree is a kind of greedy based algorithm, so I believe it will only consider those features which can be used for the best fit of the training data and ignore the rest of them.


texts: Unsupervised text clustering using a driving list

texts:
I want to apply unsupervised clustering on a set of short texts, which I need to divide into 2 clusters.
Also I know that one of my clusters is likely to contain some words (non-exhaustive list) and I would like to use them for driving my machine learning.

I did not really find any simple way to do it (from any algorithm, library, tool...). Do you know one or have any solution to help me doing this?

I was thinking about applying some kind of k-means on my DTM matrix, using a specific metric e.g the euclidian distance between 2
texts which I would "artificially" reduce when the 2 texts words match my primary list of words which are likely to appear.

I did not test it yet and I was wondering if this solution could be relevant or not.


texts:
I tried sklearn SelectKBest for feature selection (feature scores) of supervised machine learning before. It looks good, but can only take numbers as inputs.
Although I can present some string columns into numbers, for examples:

Max/min length of strings
nunique of different strings, etc.

I still hope there's a feature selection method to read strings as input.
Please recommend some feature selection that be able to analyze string (*remark) columns. Or Python import xxx.

remark: Or maybe some special kinds of strings, e.g., IP addresses, web URL.


texts:
I am working on a deep learning project for face recognition. I am using the pre-trained model VGG16.

The dataset has around 100 classes, and each class have 80 images. I split the dataset 60% training, 20% validation, 20% testing. I used data augmentation (ImageDataGenerator()) to increase the training data.

The model gave me different results when I change ImageDataGenerator() arguments. See the following cases:

Case1:

 train_datagen = ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest')
validate_datagen = ImageDataGenerator()
Test_datagen = ImageDataGenerator()

Case1 result: High training accuracy and validation accuracy, but the training accuracy is lower. check the following image:

Case2:

train_datagen = ImageDataGenerator(
      rescale=1./255,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True)

validate_datagen = ImageDataGenerator()
    Test_datagen = ImageDataGenerator()

Case2 result: Overfitting. check the following image:

Case3:

train_datagen = ImageDataGenerator(
      rescale=1./255,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True)

validate_datagen = ImageDataGenerator(rescale=1./255)
    Test_datagen = ImageDataGenerator(rescale=1./255)

Case3 result: High training accuracy and validation accuracy, but the training accuracy is lower.. check the following image:

1-  Why does using augmentation in validation and testing data ImageDataGenerator(rescale=1./255) in case3 give different result than case2?

2- Is adding ImageDataGenerator(rescale=1./255) to the testing and validation better than not adding it?

3- Do you think there is a problem in the result of the first case?


texts:
I am learning about different input-vector representations for Neural Networks

One of the alternatives to sparse One-Hot encoded vector is the Multi-Hot encoding.

Do I understand correctly that a traditional binary approach to counting numbers is exactly what the Multi-Hot is?  We can imagine a byte as a vector of 8 components, and each entry is either 0 or 1.0

Second part of the question:

I would like to use Multi-Hot encoding to express up to 255 possible input values. If I use the binary-approach, will this be identical to Label-encoding? In a sense that network will figure out that 00000010 is "superior" to 00000001, that there is a "strong correlation and precedence"?

Or is it less exaggerated? For example, in Label-encoding, I could merely use 1 input neuron, just with a varied strength of 0 to 255, much like an enum. There the effect would be really exaggerated, as it presents a really obvious precedence of value $A$ over value $A-1$

This means I can't use it for 255 distinct categories (or for relatively unrelated categories)

Is the effect as bad in Multi-Hot encoding, in particular in a binary approach?

If it's the same as Label-Encoding (which only uses 1 input neuron), why would people ever consider Multi-hot, bloating the dimension of the input-vector?

This post points out we can use multi-hot when the input should contain $N$ concatenated one-hot vectors. For example to represent $N$ entities each of which can belong to $Z$ distinct categories.  Is there also another use?


texts:
I keep hearing people say something like:

  lets say you have a 1 dimensional vector of a person that just has his age.
  Then you add another dimension which is his height, so you have a 2 dimensional vector

But I also heard things like

  A 2d vector is a rank 2 tensor

Can someone clarify which is the correct one?

Is it

[1, 1]

or

[[1,1],
 [1,1]]


texts: Parameter tuning for machine learning algorithms

texts:
When it comes to the topic of tuning parameters, most of the time you read grid search. But if you have 6 parameters, for which you want to test 10 variants, you get to 10^6 = 1000000 runs. Which in my case would be several months of processing time.

That's why I was looking for an alternative. On the Kaggle website, I found a tutorial that uses a different approach. It almost works like this:

1) Test all 6 parameter individually with the other 5 parameters as default value and plot the results

2) Change the default values for alle 6 parameters to the best value of the associated plot

3) Test all 6 parameter individually with the other 5 parameters as last best value and plot the results

4) Repeat step 2 and 3 until the results does not change anymore

This approach has the advantage of requiring much fewer runs. Is this a scientifically accepted procedure? And does this approach have a name in the literature?


texts:
I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, Decision Tree).

I am performing a type of cross validation where I randomly select 67% of the data for use in the training set and 33% of the data for use in the testing set. I perform this for several iterations, say, 20.

Now, from each iteration I am able to generate a confusion matrix and compute a kappa. My question is, what are some ways to aggregate these across the iterations? I am also interested in aggregating accuracy and expected accuracy, among other things.

For the kappa, accuracy, and expected accuracy, I have just been taking the average up to this point. One of the problems is that when I recompute kappa with the aggregated average and expected average, it is not the same with the aggregated kappa.

For the confusion matrix, I have been first normalizing the confusion matrix from each iteration and then averaging them, in an attempt to avoid an issue of confusion matrices with different numbers of total cases (which is possible with my cross validation scheme).

When I recompute the kappa from this aggregated confusion matrix, it is also different from the previous two.

Which one is most correct? Is there another way of computing an average kappa that is more correct?

Thanks, and if more concrete examples are needed in order to illustrate my question please let me know.


texts:
I have installed cloudera CDH5 Quick start VM on VM player. When I login through HUE in the first page I am the following error

“Potential misconfiguration detected. Fix and restart Hue.”

How to solve this issue.


texts: Can I save prediction value in same csv file as a another column using panda python

texts:
I have csv data file and I design LSTM model to predict values. Then I want to save that prediction value in same csv file. Can I do that? I tried using one code then in my csv file only had prediction values and delete other columns. Can anyone give me a suggestion for that.

import pandas as pd
import numpy as np
data = pd.read_csv('data1.csv')
data = pd.DataFrame(data,columns=['x','x1','x2','y'])
data.columns = ['x', 'x1', 'x2','y']
pd.options.display.float_format = '{:,.0f}'.format
data = data.dropna ()
d = ['y']
y=data['y'].astype(int)
cols=['x', 'x1', 'x2']
x=data[cols].astype(int)
scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))
x = np.array(x).reshape ((len(x),3 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)
print("row",len(y))
n = data.shape[0]
p = data.shape[1]
fill_missing(data.values)
train_start = 0
train_end = int(np.floor(0.65*n))
test_start = train_end+1
test_end = n
x_train = x[np.arange(train_start, train_end), :]
x_test = x[np.arange(test_start, test_end), :]
y_train = y[np.arange(train_start, train_end), :]
y_test = y[np.arange(test_start, test_end), :]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))
seed = 20
np.random.seed(seed)
fit1 = Sequential ()
fit1.add(LSTM(
output_dim = 10,
activation='relu',
input_shape =(3,1)))
fit1.add(Dense(output_dim =1))
fit1.add(Activation(linear))
batchsize = 10
fit1.compile(loss="mean_squared_error",optimizer="adam")
fit1.fit(x_train , y_train , batch_size = batchsize, nb_epoch =10,   shuffle=True)
print(fit1.summary ())
pred1=fit1.predict(x_test)
pred1=fit1.predict(x_test)
real_test = scaler_y.inverse_transform(np.array(y_test).reshape ((len(y_test), 1))).astype(int)
pred1 = pd.DataFrame(pred1, columns=['pred1']).to_csv('data1.csv')


texts: Deciding Initial Weights In A Linear Classifier For Sentiment Analysis

texts:
I want to build a simple sentiment analysis classifier using logistic regression. I downloaded a list of positive and negative words from cs.uic.edu. There are more than 6000 words both positive and negative. Linear Classifier has the form: (Wikipedia Reference)

∑ wj*xj

where w is the weight of the vector x. So for example, if the weight of vector awesome is 3, then in the following sentence:

Food is awesome and music is awesome.

according to the formula, it will become:

3 * 2

where 3 is the weight of the vector awesome and 2 is the vector itself (denotes the number of times it occurs in a sentence)

My question is how do I decide the coefficients to start with? Will it be a manual process? There are more than 6000 words. What is the way to approach this?


texts: Passing contextual information to an RNN

texts:
I've been experimenting with RNNs for predictive typing at both the letter and word level in TensorFlow and Keras, with great success so far.

In some cases we know about the context in which a user is typing a message. For example the time of day, the frequency at which messages are being exchanged, or the gender of the recipient. Giving the RNN this information could produce better predictions.

All the example RNNs I've been working from have only passed the relevant series - times series, text, musical notes, etc. Is it reasonable to simply pass time, gender and frequency data into the RNN mixed with the text representations?

The orignal training data would look like this:

I may admit at the outset th
 may admit at the outset tha
may admit at the outset that
ay admit at the outset that
y admit at the outset that I

Could I just include contextual information with text?

M 12:09PM 27s I may admit at the outset th
M 12:09PM 27s  may admit at the outset tha
M 12:09PM 28s may admit at the outset that
M 12:10PM 29s ay admit at the outset that
M 12:10PM 30s y admit at the outset that I


texts: Bayesian test for classification problems?

texts:
I am using two classification algorithms in Weka i.e. Logistic Regression and Naive Bayes and want to know which algorithm has better performance? I need a statistical test like Bayesian, so which type of Bayesian test I need to perform?

Let me explain a bit. I have to perform regression based prediction on two different algorithms. The output is absolute residual I.e difference between actual and predicted values. For instance, algorithm A has results like 0.3, 0.12, 0.01, 0.7, 0.32 etc and algorithm B has values like 0.1, 0.9, 0.56, 0.32, 0.01 etc. The total instances could be more than 100. Now in weka, I got the results that algorithm A is better than B. Now I want to perform a statistical test that these results are not equivalent. In case of Wilcox test, for instance if I get p value less than 0.05, I say A is significantly better than B. How can I do this using baysian test or any other test which does not have p values?


texts:
I'm new to data analysis and ML in general. I'm working with some friends on this problem: We're trying to predict when a component of a machine will stop working properly so the client can change it in time to avoid production slowdowns. We have a dataset in which are registered:

Timestamp
Event description
Event type id
Machine id
List item

There are 2 types of events, each one with an id:

ID: 0 -> completed piece event, all ok (Event description: E_INFO_PIECE_COMPLETATED)
ID: 1 -> error event that indicates something went wrong

There are a lot of events with the error type, which are described by the Event description column, for example, E_ALARM_CUTTING_DRIVE_ERROR or E_ALARM_LOW_AIR_PRESSURE.

We tried to understand if there is a correlation between all these error events and if they repeat over time by following a specific pattern but ended with very poor results.

Some arguments we discussed are:

ARIMA
PCA
Time series anomaly detection

But we don't know if these models can be applied to our case or, if yes, how.
So the question is:

Is there a model that we can adopt using the current data (Event description and timestamp)? If not, do we need more data, like sensor values?


texts: Information Visualization: which parameters should a data scientist plot to best explain the work behind a Neural Network?

texts: I'm a beginner in the field and I wrote and trained my first Convolutional Neural Network from scratch. I would like to know what types of graphics and parameters a data scientist uses to plot to explain a neural network in the best possibile and mathematical way. As far as I know I use to plot loss and accuracy graphics using matplotlib.


texts: Model Parallelism not working in Inception v3 with Keras and TensorFlow

texts:
I have been stuck with a problem like this for a while now. I have an AWS setup with 500 GB of RAM and about 7 GPUs. Now the issue is that each time I try to run my Keras with TensorFlow as back-end code, it runs out of memory. I have found out the reason for this as well. The reason is that each GPU just has 12GB of memory, whereas my model needs more than that. So, how can I run the model such that it uses the memory of all the GPUs combined to load the model and not just rely on the memory of one GPU for loading the entire model and running out of memory? I have tried model parallelism with Keras and it seems to be set-up correctly as on printing the layers , each layer is assigned to the programmed GPU but the model is still trying to load into a single GPU's memory, i.e., just 11GB and soon runs out of memory.
Any idea what's going on?
with tf.device('/gpu:0'):
    x = conv2d_bn(img_input, 32, 3, 3, strides=(2, 2), padding='valid')
    x = conv2d_bn(x, 32, 3, 3, padding='valid')
    x = conv2d_bn(x, 64, 3, 3)
    x = MaxPooling2D((3, 3), strides=(2, 2))(x)

    x = conv2d_bn(x, 80, 1, 1, padding='valid')
    x = conv2d_bn(x, 192, 3, 3, padding='valid')
    x = MaxPooling2D((3, 3), strides=(2, 2))(x)

    # mixed 0, 1, 2: 35 x 35 x 256
    branch1x1 = conv2d_bn(x, 64, 1, 1)

    branch5x5 = conv2d_bn(x, 48, 1, 1)
    branch5x5 = conv2d_bn(branch5x5, 64, 5, 5)

    branch3x3dbl = conv2d_bn(x, 64, 1, 1)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 96, 3, 3)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 96, 3, 3)

    branch_pool = AveragePooling2D((3, 3), strides=(1, 1), padding='same')(x)
    branch_pool = conv2d_bn(branch_pool, 32, 1, 1)
    x = layers.concatenate(
        [branch1x1, branch5x5, branch3x3dbl, branch_pool],
        axis=channel_axis,
        name='mixed0')
    print(x)

with tf.device('/gpu:1'):
    # mixed 1: 35 x 35 x 256
    branch1x1 = conv2d_bn(x, 64, 1, 1)

    branch5x5 = conv2d_bn(x, 48, 1, 1)
    branch5x5 = conv2d_bn(branch5x5, 64, 5, 5)

    branch3x3dbl = conv2d_bn(x, 64, 1, 1)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 96, 3, 3)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 96, 3, 3)

    branch_pool = AveragePooling2D((3, 3), strides=(1, 1), padding='same')(x)
    branch_pool = conv2d_bn(branch_pool, 64, 1, 1)
    x = layers.concatenate(
        [branch1x1, branch5x5, branch3x3dbl, branch_pool],
        axis=channel_axis,
        name='mixed1')

    # mixed 2: 35 x 35 x 256
    branch1x1 = conv2d_bn(x, 64, 1, 1)

    branch5x5 = conv2d_bn(x, 48, 1, 1)
    branch5x5 = conv2d_bn(branch5x5, 64, 5, 5)

    branch3x3dbl = conv2d_bn(x, 64, 1, 1)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 96, 3, 3)
    branch3x3dbl = conv2d_bn(branch3x3dbl, 96, 3, 3)

    branch_pool = AveragePooling2D((3, 3), strides=(1, 1), padding='same')(x)
    branch_pool = conv2d_bn(branch_pool, 64, 1, 1)
    x = layers.concatenate(
        [branch1x1, branch5x5, branch3x3dbl, branch_pool],
        axis=channel_axis,
        name='mixed2')
    print(x)

Edit: Here's the link to the code.


texts:
Is there a database of phrasal verbs of similar semantics? Eg. one where querying ‘get in touch’ would return ‘get in contact with’, among other phrasal verbs of similar meanings?

If not, given a phrasal verb how would you obtain others of similar meanings?


texts: Find possible fearure values to predict a certain outcome

texts:
I have a dataset about patients waiting times in a health district. The data is aggregated by health provider, type of medical procedure, urgency of the procedure (3 classes) and reports the n. of procedures done with the average waiting time in a certain period.
Let's define each row of this dataset as waiting time.

I have these data for the whole 2018 and for the first 6 months of 2019. so I added two more columns, one for the year and the second for the number of the month covered.

For each waiting time, I have a target waiting time which is the global average of a subset of providers. So this target is unique for sub-groups of rows.

Given a ML model to predict the global average, is there a way to estimate a set of possible values for the n. of procedure and average waiting time for the rest of 2019 in order to have a global value at the end of the year which is lower of a defined threshold?

I was thinking about exploring a random grid of values, but I think I need also to give a distribution to these values, in order not to have impossible situations, based on the 2018 data.

Are there standard solutions for this problem (or libraries for R)?


texts: Use 1 or 2 norm for Voronoi vector quantization?

texts:
I have a script from a lecture. Basically it says that based on the Voronoi partitioning we identify the corresponding (nearest) class $w_k$ to a vector $x$ where $\left| {{w_k} - x} \right| = \mathop {\min }\limits_i \left( {\left| {{w_i} - x} \right|} \right)$ given the classes $w$.

The script uses the absolute value notation. This does not make much sense as we are using vectors. Which vector norms can or should be used? The 1 or 2 norm?
The 1 norm is faster to compute but maybe there are drawbacks I cannot currently think of.


texts: Can I force DecisionTreeClassifier to use integer conditions when the variable is integer?

texts:
I'm trying to visualize a decision tree in python for the purpose of explainability. I noticed that a condition like "NumGoals >= 1.23" could be quite vague for the user and I would much rather to see something like this: "NumGoals > 1".

So, I tried to switch the normalization off, and I even suspected that it might be a side effect the input matrix's dtype.

This is a simple code to reproduce the results.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt
import numpy as np

iris = load_iris()

clf = DecisionTreeClassifier()
clf = clf.fit(np.array(iris.data, dtype=int), iris.target)
plot_tree(clf)
plt.show()

So, is it even possible to tell the tree to treat integers as integers?


texts: Why are mBART50 language codes in an unusual format?

texts:
I am trying to use mBART for multilingual translation(about 30 languages) but I am facing an issue with using it as I am currently using langid to identify the languages then load mBART and translate all the words based on the language code that has been identified. But mBART uses this odd format for language codes for example:
en_XX -> English
hi_IN -> Hindi
ro_RO -> Romanian

Whereas Langid outputs them in this format:
af, am, an, ar, as, az, be, bg, bn, br

I cannot seem to find any documentation on how to interpret the mBART language code as even the research paper does not include it.


texts: what is the training phase in N-gram model?

texts:
Following is my understanding of N gram model used in text prediction case :
Given a sentence say, " I love my " (say N = 1 /bigram),  using N gram and say 4 possible candidates ( country, family, wife, school) I can estimate the conditional probability on each of the candidates and take the one with highest probability as the next word.
Question :

I understand the probability part of the model but to even get to the probability, we need the possible candidates ( next words, in this case family, wife, school, country). How does the model choose the candidates

Most of the articles online talk about the probability part but doesn't mention anything about training phase. What exactly is happening in training phase of this model?


texts: Determining which categorical data is beneficial in predictive modelling

texts:
I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all independant), and some historic data might look like:
JobID   Manager     City        Design          ClientType      TaskDuration
a1      George      Brisbane    BigKahuna       Personal        10
a2      George      Brisbane    SmallKahuna     Business        15
a3      George      Perth       BigKahuna       Investor        7

Thus far, my model has been relatively basic, following these basic steps:

Aggregate the historical data based on each category, calculating the mean, and counting how many times it occurs. From the previous example, the result would be:

Category        Value           Mean    Count
Manager         George          10.66   3
City            Brisbane        12.5    2
City            Perth           7       1
Design          BigKahuna       8.5     2
Design          SmallKahuna     15      1
ClientType      Personal        10      1
ClientType      Business        15      1
ClientType      Investor        7       1

For each job in the system, calculate the job duration based on the above. For example:

JobID   Manager     City        Design          ClientType
b5      George      Brisbane    SmallKahuna     Investor

Category        Value           CalculatedMean      CalculatedCount     Factor (Mean * Count)
Manager         George          10.66               3                   31.98
City            Brisbane        12.5                2                   25
Design          SmallKahuna     15                  1                   15
ClientType      Investor        7                   1                   7

TaskDuration    = SUM(Factor) / SUM(CalculatedCount)
                = 78.98 / 7
                = 11.283
                ~= 11 days

After testing my model on a few hundred finished jobs from the last four months, I calculated average discrepancies ranging from -15% to +25%.
I think the one of my issues is that I may be taking into account categories that actually have no effect on the build time, and are skewing my results. In reality, I'm taking 15 categories into account from ~400 completed jobs, and some of these categories might have results that only appear once or twice (for example, we might only have a single job in Perth).
How can I determine which categories are actually beneficial to the model, and which should be ignored?
Related question here.


texts: NLP: What are some popular packages for multi-word tokenization?

texts:
I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are splitted by whitespace, which may well cause accuracy problems in subsequent processing. So I want to get all the most interesting/informative collocations in these texts.

Is there any good packages for doing multi-word tokenization, regardless of specific programming language? For example, "He studies Information Technology" ===> "He" "studies" "Information Technology".

I've noticed NLTK (Python) has some related functionalities.

collocations Module:
http://www.nltk.org/api/nltk.html#module-nltk.collocations
nltk.tokenize.mwe module:
http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.mwe

What the difference between these two?

The MWETokenizer class in the nltk.tokenize.mwe module seems working towards my objective. However MWETokenizer seems to require me to use its construction method and .add_mwe method to add multi-word expressions. Is there a way to use external multi-word expression lexicon to achieve this? If so, are there any multi-word lexicon?

Thanks!


texts: How much is the Class Imbalance Problem rates?

texts:
I'm working on a data set and wanted to know is there a standard rate about Class Imbalance problem or not?

I have 47 samples in Class A and 150 Sample in class B , should I use Class Imbalance Technique or these rates are normal?


texts: Manually building and visualising a decision tree

texts:
I am currently working on a decision tree for predicting the average of a binary outcome with a small number of categorical features (think predicting the survival rate in the titanic dataset). I am aware of different R and Python libraries that allow to optimally calibrate decision trees on historical data. However, I'd like a more manual, interactive -ideally without code, ideally free- approach to build simple decision trees.
I am fully aware that this would not yield optimal results in many senses. However I have many practical constraints that would make that an interesting solution. This constraints include, but are not limited to : (for the design and the usage of the model) variables playing a key role in the industrial process should ideally be put at the top, need to involve non-coding people in the design of the tree, need to use limited and technically meaningfull splits so that user can print the tree and go trough it in less than 10 sec. and might even memorize parts of it). And, maybe most importantly in terms of modelling, the need to go further than calibration on past historical data and include prospective expert-based grouping of behavior in the model, as the current crisis is likely to change a lot of things.
So what tool, ideally free, ideally with a graphical user interface, and compatible with some well established DS / corporate langage (R/Python or Excel...) might be used for manually building some small non-binary tree / change splits / visualise the tree and outcomes ?


texts: Which model is better for incremental learning?

texts:
I'm trying to implement face recognition. I'm planning to use some model (like DeepFace) to extract discriminative features and then use a classifier to recognize the faces. I'm confused as to which classifier to use.

My setup is as follows:
Initially, I'll have a few labelled samples and I train a classifier. Then I start getting more labelled samples and I want to update my classifier with the new data. Also, the newer samples may contain new classes (faces). So, I want my classifier to be able to learn to classify new classes as well. My question is which classifier is better suited for this task. I would also prefer to get multiple predictions with a confidence score.

K Nearest Neighbors algorithm: Here I can use distance as some form of confidence score.
As presented here, I can use SVM: Train on the initial data and retain support vectors alone. Repeat the process as and when new data comes up.

But, I have no idea if the above 2 classifiers are suited for the task at hand or how they would perform. Any inputs regarding this is greatly appreciated. I'm also open to any other classifiers as well. Thanks!


texts: Is it normal when BLEU score on filtered data by length is greater than BLEU score on whole data

texts: I am creating 2 neural machine translation model (model A and B with different improvements each model) with fairseq-py. When I evaluate model with bleu score, model A BLEU score is 25.9 and model C is 25.7. Then i filtered data by length into 4 range values such as 1 to 10 words, 11 to 20 words, 21 to 30 words and 31 to 40 words. I re-evaluated on each filtered data and all bleu scores of model B is greater than model A. Do you think this is normal case?


texts: Is there a package for using SQL to manipulate Pandas dataframes in Python?

texts: Rather than learn a new package/language, I'd like to use my existing SQL skills to manipulate pandas dataframes in Python. Does anyone know of a way to do this, or perhaps a package that will allow me to do this?


texts:
I am trying to classify small strings into three categories.
Examples:
X(str column)                    y(str column)
abcd@gmail.com                   email_address
32                               age
@sadn                            invalid(since this is not a valid email address)
1234                             invalid(since this is not a valid age)

I am wondering which ML model needs to be used to achieve this case. And also what kind of text processing techniques need to be applied.


texts:
I am trying to make MXNet (C++ API) learn, with a common sample in C++, on multiple GPU.
According to this MXNet forum post, we need to aggregate manually the gradients that we fetch at the backpropagation time.
Now, if I separate the gradients of each GPU, both networks are training. If I concatenate the weights, it doesn't work. (Like this) :

gradValuesCombined.insert(gradValuesCombined.end(), gradValues1.begin(), gradValues1.end());
gradValuesCombined.insert(gradValuesCombined.end(), gradValues1.begin(), gradValues1.end());
NDArray combined1(gradValuesCombined.data(), combinedShape, ctx1);

However, if the gradients have the same batch_size, then summing the gradients works.

gradArray1[i] + gradArray2[i]

But, summing all the gradients like if it was a one-batch vector, doesn't work:

combinedOneDim[0] = 1;
int sizeOneBatch = combinedOneDim.Size();
gradValuesCombinedOneDim.insert(gradValuesCombinedOneDim.end(), gradValues1.begin(), gradValues1.begin() + sizeOneBatch);
for(int i = 1; i < curGradArrayShape1[0]; i++) {
     std::transform(gradValuesCombinedOneDim.begin(), gradValuesCombinedOneDim.end(), gradValues1.begin() + i * sizeOneBatch, gradValuesCombinedOneDim.begin(), std::plus<mx_float>());
}
for(int i = 0; i < curGradArrayShape2[0]; i++) {
     std::transform(gradValuesCombinedOneDim.begin(), gradValuesCombinedOneDim.end(), gradValues2.begin() + i * sizeOneBatch, gradValuesCombinedOneDim.begin(), std::plus<mx_float>());
}

At that stage, I figured that it was because I had to keep the initial shape of the gradient. But if I try to average the weights on the batch dimension, and copy the values to keep the same gradient shape, it should work. But it doesn't... And if I just sum everything on the first dimension and fill the rest with zeroes, it doesn't work either...

I verified, the synchronization of data is correct and working. So what do I miss?


texts:
I'm currently taking a course "Introduction to Machine Learning" which covers the following topics:
linear regression, overfitting, classification problems, parametric & non-parametric models, Bayesian & non Bayesian models, generative classification, neural networks, SVM, boosting & bagging, unsupervised learning. I've asked the course stuff for some reading material about those subjects but I would like to hear some more recommendations about books (or any other material) that give more intuition about the listed topics to start with and also some books that go deeper into the theory of those subjects (affter I'll gain some intuition)? (I guess every subject has many books to cover the theory of course, but any recommendations will be great)
Thank you.


texts: Should the type of Boolean categorical features be numerical or categorical after encoding?

texts:
There are categorical features which have two different value in my dataframe next to numerical features. I've converted these categorical values to 0 or 1.

I will apply correalation elimination on features after calculating correlation coefficients. Depending on type of features, methods are given below:

Numeric - Numeric: Pearson

Numeric - Categoric: Cramer_V

Categoric - Categoric: Correlation Ratio

That's why I could not be sure what should be type of converted categorical features? Numerical or categorical ?

Another reason to I asked this question is that when I create dummy features for the categorical features which have only two different values, it creates features contains 0 and 1 like how I did manually. So after this process it's taking these features as numerical. But still each value from the feature represents a class and I think feature type should not be numerical.


texts: How to Classify Game Stages Based on Bitrate Time Series Data

texts:
I need suggestions for my project and would be glad if you would give me a hand.
I have a dataset of frames obtained from the old-school game DOOM. Each frame in the dataset has the following columns:
time, frame buffer size in bytes, game stage (0: Exploration, 1: Combat)
Once collecting consecutive frames to calculate instant bitrate (kbps) for each time instance, I apply Exponential Moving Average on bitrate to smooth and here below the data I end up with:

I need to train a model by using this data and in the end, I should be able to classify game stages correctly. Currently, I have two approaches:

Calculating the average bitrate of each stage, and examining the % change from the former stage to the latter one (avg_latter - avg_former / avg_former * 100) in different baseline bitrate ranges: (W=9, 18, 35 below indicates Exponential Moving Average window lengths)

Result for transitions from Exploration -> Combat
Result for transitions from Combat -> Exploration

Calculating the bitrate variance in each stage change moment by looking 'N' frames back and forward:

Results of transitions from Exploration -> Combat and viceversa
According to the results above, I wonder which methodology would make more sense for game stage classification. Any idea is appreciated. Thanks!


texts: Keras ImageDataGenerator vs bounding boxes

texts:
I'm trying to implement custom object detection by taking a trained YOLOv2 model in Keras, removing the last layer and retraining it with new data. I'm confused about how to feed the data to Keras, though. I have annotated a bunch of pictures with bounding boxes using the YOLO annotation, and put them in two separate folders (images where the .jpgs reside and annots where the .txt annotations are).

I also removed the last layer from the model and added a custom one (I'm trying to predict bounding boxes for 2 classes).

I'm trying to pass my data with an ImageDataGenerator, as my dataset is quite small.

Since I am trying to create bounding boxes (and thus have categories AND bounding boxes as target variables), does the ImageDataGenerator also somehow skew the bounding boxes? Or in this case can/should I not use the ImageDataGenerator (because if it rotates the pictures but does not do so with the bounding boxes, it might mess up the learning)


texts:
I want to use the result of my PCA as an input for my LSTM model.

I began by Applying the MinMaxScaler and then did the PCA, (then I reshaped my data of course) :

sc = MinMaxScaler(feature_range=(0, 1))
data = sc.fit_transform(data)
pca = PCA()
data = pca.fit_transform(data)

The problem is, this give me a data between -1,23 and 1,33.
I know that the input of an LSTM must be between range 0 and 1.

So what can I do ? Should I apply one more time a MinMaxScaler on "data" ?


texts: Is it necessary to normalise data for XGBoost?

texts: MinMaxScaler in scikit_learn is used for data normalization (a.k.a feature scaling). Data normalisation is not necessary for decision trees. Since XGBoost is based on decision trees, is it necessary to do data normalisation using MinMaxScaler for data to be fed to XGBoost machine learning models?


texts: Augmenting the validation set in Ensemble Model

texts: I have 8 models which I have trained on 90% of my set (training set) and tracked its performance on the loss of the validation set (10% of the original set). I want to generate an ensemble model by blending the different models with a dense layer. However, the validation set only consists of 64 samples and I am concerned that this might cause severe overfitting. I was wondering whether it would make sense to augment the validation data, so that I can increase the number of instances and help in preventing overfitting.


texts: Data scientist vs machine learning engineer

texts:
What are the differences, if any, between a "data scientist" and a "machine learning engineer"?

Over the past year or so "machine learning engineer" has started to show up a lot in job postings. This is particularly noticeable in San Francisco, which is arguably where the term "data scientist" originated. At one point "data scientist" overtook "statistician", and I'm wondering if the same is now slowly beginning to happen to "data scientist".

Career advice is listed as off-topic on this site, but I view my question as highly relevant since I'm asking about definitions; I'm not asking about recommendations given my own career trajectory or personal circumstances like other off-topic questions have.

This question is on-topic because it might someday have significant implications for many users of this site. In fact, this stack-exchange site might not exist if the "statistician" vs "data scientist" evolution had not occurred. In that sense, this is a rather pertinent, potentially existential question.


texts: Pytorch Luong global attention: what is the shape of the alignment vector supposed to be?

texts:
I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. So that all makes sense.
My question is, what should the dimensions of the alignment score tensor be? If I have my data in batches, I basically compute a single score for each timestep in the hidden state, right. So should the alignment vector be of dimension [sequence length, 1], or something like that? I would then softmax this alignment vector and multiply it by every element in the batch to compute the context vector, right.
Again, my key question is what the dimensions of the alignment score vector or tensor should be. Thanks.


texts: I only have microsoft office, how can i write code to manipulate relational data sets?

texts: At work, I am in a role that does not have a lot of computer permissions. I do have the Microsoft office suite. I have skills with programming and would like to be able to use my experience to do more streamlined and substantial manipulation of large relational data sets. Considering I can't get python or even Bash, is there any good way to do this?


texts: How to automatically find the sentiment?

texts:
I have written a program that scrapes data from the web and I have in possession about 5k sentences which I want to analyze.

Part 1: I am just starting out in data science and wanted to know if there is such a thing as a library that will read through text and automatically classify words or sentences as being positive/negative/neutral. Is there such a thing that exists, perhaps in R or Python?

Part 2: From what I can tell, there isn't such a library, and I would have to do that myself by hand. It appears that the best way to analyze sentiment is to:

a) put the words in a corpus
b) do things like word stemming and
clean up the text
c) download a dictionary of predetermined
sentiment
d) run a program that vectorizes the text
e) compare the text from my corpus to the dictionary
f) ..... unsure what after this step.

Are these steps accurate, or am I totally off-base?


texts:
I am performing an online news classification. The idea is to recognize group of news of the same topic.
My algorithm has these steps:

1) I go through a group of feeds from news sites and I recognize news links.

2) For each new link, I extract the content using dragnet, and then I tokenize it.

3) I find the vector representation of all the old news and the last one using TfidfVectorizer from sklearn.

4) I find the nearest neighbor in my dataset computing euclidean distance from the last news vector representation and all the vector representations of the old news.

I have a problem using TfidfVectorizer because it weights more the special words that only appear in a few news, like Apple, and news that talk about Aple are grouped together even when they deal with different topics.

So, Is there a common approach more efficient than the one I am using?


texts: How can I extract the residual array from the Scikit Learn PCA routine?

texts: The Nipals PCA algorithm calculates the scores and loadings of a data array iteratively.  If only (say) 3 scores and loadings are calculated from a data array with more than 3 variables, there is a residual matrix created (called E).  How can I extract this matrix from the SciKit Learn PCA algorithm so that I can create contribution charts?


texts: How to model for choosing ‘safest’ outcome within range?

texts:
Let’s say you have a dataset of delivery service staff’s past performance:
deliver person id, percentage delivered on time for previous deliveries, percentage being late less than 5 min, percentage being late more than 5 min, median minutes late, std of minutes late, percentages as before but for X latest deliveries, area code, vehicle, weather, traffic, order price, etc. and label - was late to this delivery or not
for an upcoming delivery you want to assign the person with highest likelihood of being late less than 5 minutes (each time a person is late more than 5 minutes you give the customer a percentage discount of their order).

How would you model this? Specifically, what would you do to have high certainty around the 5 min threshold to reduce loss?

The way I thought to approach this was to train a classifier (random forest/xgbst) to predict late or not, and output a probability instead of label, then for a certain delivery task ’assign’ each person and get the probability for them - choose person with highest probability.

Is there a better way than just choosing the highest probable person? any suggestions on how to break ties?

Should the loss function be integrated somehow to the modelling/training? or it doesn’t matter since we are picking the highest probable person anyway and the loss is identical across them?

Any other features you'd recommending looking into?

An alternative method I thought was to predict for each person by how much they'll be late (using a regression model) and then choose the person with the least time. But I thought this might be a lot less accurate, and how would I get the probability of results being wrong?


texts: Why Is Overfitting Bad in Machine Learning?

texts:
Logic often states that by overfitting a model, its capacity to generalize is limited, though this might only mean that overfitting stops a model from improving after a certain complexity. Does overfitting cause models to become worse regardless of the complexity of data, and if so, why is this the case?

Related: Followup to the question above, "When is a Model Underfitted?"


texts: Using 2 nodes in the output sigmoid activation function for 2 mutually exclusive classes is somehow giving good results than softmax

texts:
I know for two mutually exclusive classes softmax is the best activation function in the output layer. However, somehow (2, softmax) and even (1,sigmoid) are giving average results and (2, sigmoid) as output layer is giving excellent results (in terms of validation and test set accuracy).
But, as per my understanding, using (2, sigmoid) in the output layer for mutually exclusive classes is not logically correct. I don't understand why in my case (2,sigmoid) is giving better results than (2, softmax). Have any one encountered similar results?
Also, an interesting observation is that my two mutually exclusive classes are very complex. i.e., although they are mutually exclusive but its very difficult to put them in any of the classes in the training data and there will be some instances where it could have been any of the two. Would (2,sigmoid) be justified given the complexity of the classes ?


texts:
I built a fairly standard backpropagation algorithm and just the process of forward propagating through a 5 layer x 5 nodes network using a data set of 10,000 observations of 39 variables takes almost 5 minutes for one iteration.

Do neural networks typically take many hours to train using data sets this size? My initial data set was 10x as long, but I couldn't wait an hour just for one forward pass to be completed.


texts: Different Values in Training and Testing set

texts:
If I am trying to build a classification model where Values for columns have different ranges: For Example: Column A in training set ranges from 0 - 30 but for testing set it ranges from 0 - 80.

Would a classification model still work?


texts: Which Graph Properties are Useful for Predictive Analytics?

texts:
Let's assume I'm building a content recommendation engine for online content. I have web log data which I can import into a graph, containing a user ID, the page they viewed, and a timestamp for when they viewed it. Essentially, it's a history of which pages each user has viewed.

I've used Neo4J and Cypher to write a simple traversal algorithm. For each page (node) I want to build recommendations for, I find which pages are most popular amongst other users who have also visited this page. That seems to give decent results. But, I'd like to explore alternatives to see which method gives the most relevant recommendations.

In addition to my simple traversal, I'm curious if there are graph-level properties I can utilize to build another set of recommendations with this data set. I've looked at SNAP, it has a good library for algorithms like Page Rank, Clauset-Newman-Moore community detection, Girvan-Newman community detection, betweenness centrality, K core, and so on.

There are many algorithms to choose from. Which algorithms have you had success with? Which would you consider trying?


texts: Recommended ways of building data science teams

texts:
I'm not sure if this questions belongs to this place. But I will be very interested in knowing the answer. I work for the Public Administration in Spain. We want to build a data science team, and in order to do it well, I want to know what in other large companies (like a bank, for example) are doing:

How many people usually work in the area of data science? And how are they distributed into teams?
What are their roles?
With respect to projects, how many people work for a particular data-science project?

If you work for a big company and can answer this questions, I will be really grateful.


texts: Is there any single disadvantage to use GPU in deep learning?

texts:
In most cases I frequently heard that to make a deep learning experiment, it is highly recommended to use GPU. It makes the computation blazingly faster than CPU, and sounds like a magical tool (although I can't use it since I use MacBook...).

However, is there any single disadvantage to use GPU, possibly except the higher monetary cost? For example the higher likelihood of getting crashed, etc...?


texts:
I am a noob in machine learning and trying to build a classifier using keras by following this tutorial machine learning mastery tutorial

I want to build a classifier based on MLP like in classification of MNIST using MLP for CIFAR-10 data set.Like in MNIST data set the 28*28 images to a 784 vector is given as a input to the neural network.
Like wise how should I frame the data in CIFAR-10 should it be changed in the 36*36 pixels rgb values to an array? or what type of array should I make to give as a input to the MLP network?


texts: How an input data flows through an lstm layer cells?

texts:
I make this sec2sec NN model for the purposes of learning:
model = Sequential()
model.add(LSTM(100, activation="relu", input_shape=(3,1)))
model.add(RepeatVector(2))
model.add(LSTM(100, activation="relu", return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(optimizer="adam", loss="mse")

It was trained on this sequence:
[10, 20, 30, 40, 50, 60, 70, 80, 90]
for using like this
x_test = array([70, 80, 90])
x_test = x_test.reshape(1, 3, 1)
model.predict(x_test, verbose=0)

I can not understand whats happen with input data inside all 100 LSTM cells when I invoke model.predict().
Explain me please what all this 100 units do if x_test array goes into one unit input.


texts: text elaboration with neural network non natural language

texts:
I'm trying to understand if it's possible to elaborate with the neural network 2 input strings and predict a result string. I'm doing this job manually and it's really time consuming: basically i have something like this
input 1 : "MEAS-CABC2S-AIR04-CO-AVG" (like a tag name of the signal) input 2 : "Scaled average value" (description of the signal)
the corresponding output should be the object associated to this signal output: CO-AIR-04 (sometimes is a substring of the tagname sometime is a substring of the description)
Mentally the steps i'm doing to understand what's the corresponding object is by mentally splitting (by "-") the input 1 and with the description understand what is the corresponding object.
My idea is to feed into the neural network the all substrings ['MEAS', 'CABC2S', 'AIR04', 'CO', 'AVG', 'Scaled', 'average', 'value'] and for the expected output ["CO", "AIR", "04"] or directly the entire string like so CO-AIR-04
Am i completely wrong about this idea?


texts: statsmodels ols does not include all categorical values

texts:
I am doing an ordinary least squares regression (in python with statsmodels) using a categorical variable as a predictor. There are 5 values that the categorical variable can have. However, after running the regression, the output only includes 4 of them.

Here is what I am running:

>>> from statsmodels.formula.api import ols
>>> model = ols("normalized_score ~ C(general_subreddit)", data=df_feature)
>>> results = model.fit()
>>> results.summary()

The output of the last command includes the following rows in the table:

I can check the count of each of the categorical variables as follows:

>>> from collections import Counter
>>> Counter(df_feature["general_subreddit"])
Counter({nan: 20,
     'community': 4159,
     'ending_addiction': 3819,
     'mental_health': 4650,
     'other': 6920,
     'relationships': 4318})

Ignoring the NaNs, why does the categorical value of "community" not appear in the model summary?


texts: paired t-test shows no difference between median and Wilcoxon test p value shows that there is a difference between median values ? How to interpret?

texts:
I have a dataset. I wanted to do paired t test on it. So I carried out normality test and it showed that it does not follow normal distribution. So I used Wilcoxon test in place of paired t test. The descriptive statistics of my data shows median value of 0 but when I ran wilcoxon test, it gave me a p value <0.05 stating to reject null. Could you please tell me why this discrepancy in my test results?
summary(Rel)
REL1.                REL2.          Diff
Min.   :1.000     Min.   :1.000    Min.   :-5.0000
1st Qu.:7.000     1st Qu.:6.000    1st Qu.:-1.0000
Median :7.000     Median :7.000    Median : 0.0000
Mean   :6.617     Mean   :6.289    Mean   :-0.3282
3rd Qu.:7.000     3rd Qu.:7.000    3rd Qu.: 0.0000
Max.   :7.000     Max.   :7.000    Max.   : 3.0000
Wilcoxon signed rank test
data:  REL1. and REL2.
V = 7993.5, p-value = 2.449e-13
alternative hypothesis: true location shift is not equal to 0


texts: Why Decision Tree boundary forms a square shape and SVM a circular/oval one?

texts:
I was going through a Udacity tutorial wherein a few data points were given and the exercise was to test which of the following models best fit the data: linear regression, decision tree, or SVM. Using sklearn, I was able to determine that that SVM is the best fit followed by decision tree. I got a very distinct decision boundary when these two algorithms were applied:

Is there any specific reason for the said shapes or does it just depend on the data sets?

The code was quite straightforward; just reading the CSV, separating the features and then applying the algorithms as shown below:

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

import pandas
import numpy

# Read the data
data = pandas.read_csv('data.csv')

# Split the data into X and y
X = numpy.array(data[['x1', 'x2']])
y = numpy.array(data['y'])

# import statements for the classification algorithms
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

    # Logistic Regression Classifier
    classifier = LogisticRegression()
    classifier.fit(X,y)

    # Decision Tree Classifier
    classifier = GradientBoostingClassifier()
    classifier.fit(X,y)

    # Support Vector Machine Classifier
    classifier = SVC()
    classifier.fit(X,y)


texts: Model Joint Probability of N Words Appearing Together in a Sentence

texts:
Assume that we have a large corpus of texts to train with. Given N words as input, I want to model the joint probability $p(x_1, x_2, ..., x_N)$ of these words appearing together in a sentence. More specifically, the N words are not required to be ordered or contiguous, and words other than given words can appear in the sentence. There is no restriction on the number of times each of N words can appear in the sentence. I did some research and below are some possible directions.

Popular RNN models like LSTM offer conditional probability $p(x_t | x_{t-1}, ..., x_0)$, which may shed light on the joint probability I am modeling. However, RNN models take sequential inputs and require the N given words to be ordered. Also, if I am to use RNN models, I need to use Markov chain rule to calculate the joint probability, the N words would need to be continuous, and no words other than given words would be allowed.
Topic models like LSA(Latent Semantic Analysis) or LDA(Latent Dirichlet Allocation) may help me find topics from corpus, and model joint probability based on topics that each of N words belong to. Words from same topic are assigned higher joint probability. These models require me to manually set the number of topics. As I am not really familiar with these models, I am not sure how good they will perform.
A neural network alternative to LDA is a Restricted Boltzmann Machine, where topics are learnt as hidden neurons. The input to this RBM would be a vector the size of my whole dictionary. Each entry is either 1(present), 0(not present), or -1(don’t know). During training the probability $p(h|v_{train})$ is learnt, where $h$ is my learnt topics and $v$ are training data. At test time, the test input $v_{test}$ will have all N given words’ value equal to 1, while all other words equal to -1.  In this way we can learn the probability $p(v_{test}|h)$.

I feel like this is a really fundamental problem in language modeling, but have not yet been able to find recent research on this. Could anyone please give any pointers on this problem? Does any of the above proposed solutions look feasible? Any help is appreciated!


texts: Will a larger data set in order to support your subset?

texts: Is it possible to utilize/create a larger data set with like data in order to be able to improve the Ml model optimization and/or ML model feature options that you can utilize on your subset of data? If so what should one take into account when building this data? i.e. when sourcing additional data is there mistakes to avoid and/or how many multiples of your subset data should you ideally add?


texts: How to train DDQN model in reinforcement learning?

texts:
I was reading through some RL code but couldn't understand one small bit.

here is the code:

# Apply Bellman Equation on batch samples to train our DDQN
    q = self.agent.predict(s)
    next_q = self.agent.predict(new_s)
    q_targ = self.agent.target_predict(new_s)

    for i in range(s.shape[0]):
        old_q = q[i, a[i]]
        if d[i]:
            q[i, a[i]] = r[i]
        else:
            next_best_action = np.argmax(next_q[i,:])
            q[i, a[i]] = r[i] + self.gamma * q_targ[i, next_best_action]
        if(self.with_per):
            # Update PER Sum Tree
            self.buffer.update(idx[i], abs(old_q - q[i, a[i]]))
    # Train on batch
    self.agent.fit(s, q)
    # Decay epsilon
    self.epsilon *= self.epsilon_decay

From my understanding, the Q function maps (state, action) pairs to rewards. However in the code above you assume that the Agent networks to return Q values. However a quick inspection of the Agent model, reveals that it actually returns actions, which makes sense since that's what the agent has to learn. Rewards are be calculated within env.step(action). So then, in the belmann equation we find this addition r[i] + self.gamma * q_targ[i, next_best_action].
Isn't q_targ[i, next_best_action] an action? so how can it be added to a reward?


texts: Interpreting decision tree results after target encoding

texts:
I am not sure how to interpret the results of my decision tree after I had used target encoding, could someone clarify? The example below doesn't need target encoding just for explanation of my confusion here.
For instance I am trying to classify if a fruit is rotten or not given its age and fruit type. I use target encoding for the fruit column:

I then get the following decision tree with default sklearn decision tree classifier parameters:

I believe after encoding I have lost information about fruit type and I can only say that if fruit_target <= 0.841 then the fruit is rotten if smaller, else not rotten. But then how do i interpret 0.841; what does it mean?


texts:
I'm currently pursuing a bachelor's degree in physics from a university in the UK. Most data science jobs here have a strong preference for people with PhDs in numerate degrees like physics and maths. I don't understand the point of spending 5 years of my life doing research in something I don't want to pursue a career in.

I'm confused about what master's degree I should choose if I want to apply for a job straight after graduating. A physics degree kind of leaves me hanging in the middle in terms of skills. I have done some analysis for experimental data using Python, but only very basic stuff. A bit of predictive modelling and C++, lots of maths, but barely any statistics. So I could either get a degree in CS to get better at programming or get one in statistics. Which one would give me the skills most relevant to data science?

Also, would you recommend that I get a diploma/certification in machine learning?


texts: What would be the equivalent of R's mboost in Python?

texts:
I am looking for the Python equivalent of R's mboost package ( mboost ).

Would that be xgboost?


texts: Which service could I use to train my networks?

texts:
My laptop's Intel i7 3630QM 2.4GHZ, 8Gb RAM and GXForce 670M are clearly not sufficient...

By reading some papers, I've written an SRGAN with Python Keras. At runtime there is no error but training only 2 images (324*324) with 1 epoch, batch size of 2 lasts too much...perhaps more than 1 hour of training...

Which service could I use to train my deep neural networks, see the console outputs, eventually see plots, and download the trained model?
Are there such paid services? Free services?

The idea would be that I query the compute server from my laptop.


texts: What is the dimension hopping problem in machine learning (occurring in convolutional neural networks and image recognition)? I have googled about it but all I get is information on the Physics of material shape deformation. It will be more helpful to me if some one explain it with an example related to machine learning. Can anyone help me out with this or point me toward resources that can?


texts: What does "Find pattern" mean in data science?

texts: I am doing certification and i have a project to complete. In project, they have said "Find Patterns". What does it mean? what steps should i carry out?


texts:
Would anyone have any advice on how to add model depth? This works below but I was hoping to experiment with adding in additional non-TimeDistributed layers.

# reshape from [samples, timesteps] into [samples, subsequences, timesteps, features]
n_features = 1
n_seq = 2
n_steps = 2

# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(filters=16, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=500, verbose=0)

For example, if I add this in:

# define model
model = Sequential()
model.add(TimeDistributed(Conv1D(filters=16, kernel_size=1, activation='relu'), input_shape=(None, n_steps, n_features)))
model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(50, activation='relu'))
model.add(LSTM(40, activation='relu'))
model.add(LSTM(30, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
# fit model
model.fit(X, y, epochs=500, verbose=0)

This will throw ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2

Any tips help! Sorry not a lot of Wisdom here but learning :)


texts:
I was wondering how useful the encoder's hidden state is for an attention network. When I looked into the structure of an attention model, this is what I found a model generally looks like:

x: Input.
h: Encoder's hidden state which feeds forward to the next encoder's hidden state.
s: Decoder's hidden state which has a weighted sum of all the encoder's hidden states as input and feeds forward to the next decoder's hidden state.
y: Output.

With a process like translation, why is it important for encoder's hidden states to feed-forward or exist in the first place? We already know what the next x is going to be. Thereby, the order of the input isn't necessarily important for the order of the output, neither is what has been memorized from the previous input as the attention model looks at all inputs simultaneously. Couldn't you just use attention directly on the embedding of x?


texts: Looking for an algorithm that correctly clusters visually separable clusters

texts:
I have visualized a dataset in 2D after employing PCA. As 2D visualization shows in figure, there is a good separation between points (A, B). Now, I want to use a metric which can separate these points (between these 2 PC components not in main dataset) too. I mean have separation between these PCA components without visualization. I used some clustering methods but they raise false positives. I mean they miss cluster many points.

Also, as shown in histogram there is a gap between points A,B. Does this help in devising any metric?

I will be so grateful if you can introduce me any method and algorithm to be able to do separation between A and B.


texts: Multiple categories within a variable in decision tree

texts:
This is more a conceptual question than related to the implementation on decision trees.

I've a feature vector say V1,V2,V3,target_variable

If the vector is a,b,c,true then, using normal decision trees, we can classify the data set.

But if the variable V1 is a set say {x,y,z},b,c,true how can I implement it?

I think of methods by using x,y,z as dimensions rather than categories but the problem is the number of categories are very huge in the order if millions. This solution won't scale.

Are there any efficient ways to deal with this problem?


texts:
I'm trying to use rvest to screen scrape headline news items from google and failing.

Having previously written a utility to screen scrape high level stats from DS.SE (not user info I have to say!), which runs successfully, I know that my technique works, but this produces nothing.

I'm using SelectorGadget to find the screen element needed (".r a") and the code snippet below should read the headlines from the google page for "education will" search term.

My code (reduced down to bare bones) is:-

library (rvest)

url = "https://www.google.co.uk/#q=%22education+value%22"
url

index_html <- read_html( url )

headline_tmp <- index_html %>%
  html_nodes (".r a") %>%
  html_text()
headline_tmp

Giving the output as

> library (rvest)
>
> url = "https://www.google.co.uk/#q=%22education+value%22"
> url
[1] "https://www.google.co.uk/#q=%22education+value%22"
>
> index_html <- read_html( url )
>
> headline_tmp <- index_html %>%
+   html_nodes (".r a") %>%
+   html_text()
> headline_tmp
>character(0)

When I run with the stackexchange URL, suitably modified code for the different URL, would give a vector of data.

The reading I've done about google stopping screen scrapers, suggests that they only stop people who abuse it and usually will do it with a Captcha or similar.

Any thoughts on a solution?


texts: Forward pass vs backward pass vs backpropagation

texts:
As mentioned in the question, i have some issues understanding what are the differences between those terms.

From what i have understood:

1) Forward pass: compute the output of the network given the input data

2) Backward pass: compute the output error with respect to the expected output and then go backward into the network and update the weights using gradient descent ecc...

What is backpropagation then? Is it the combination of the previous 2 steps? Or is it che particular method we use to compute dE/dw? (chain rule ecc...)


texts: How can I inverse a transposed convolutional layer?

texts:
I am interested in inversing a transposed convolution operation (or "deconvolution", and not straightforward convolution).

A transposed convolution usually maps data points from a smaller feature space (latent space) onto a much larger feature space (image space), so the inverse should not be ambiguous, or should it?

Is there an established mathematical framework for performing the inversion? If not, how do you suggest approach the problem?


texts: Transitioning from Math PhD to ML research

texts:
I am currently a Math PhD about to defend in January. I work in a field in functional analysis that uses a lot of measure theory (but no stats).

I have been considering transitioning careers since I don't enjoy my research area anymore. I have enjoyed taking a couple graduate courses in Machine learning/DS at my University and performed well in these. I have since had the itch to do research in these areas (I have been especially enjoying bandits and reinforcement learning in general due to their deep mathematical roots). All the research related jobs/internships I have looked at so far seem to require either a graduate degree in CS or publications in ML (which I do not have due to being new in this field)

My questions are as follows: Is it realistically possible transition into a job doing research in ML without a second PhD? What steps could I take to get some publications (are there research topics that are easier to transition into?)

Thanks very much for your help!


texts: Which classification algorithms can handle 24000 features? What are their pros and cons?


texts: Pre-process 'standardize' function not working correctly

texts:
I am running an Orange workshop with 10 participants. When I use the 'standardize' function in the 'pre-process' widget, the data distributions of only some participants is not standardized. The mean of the data is not 0. About 80% of participants have no issue (including me) and the mean of data is correctly at 0. The overall shape of the distributions are exactly the same, but the position of the mean is not the same.
Version Running: All users have downloaded the most current version of Orange (3.27).


texts: How can I run SVM on 500k rows with 81 columns?

texts:
I have about 500k rows of data. I'm new to data science and I'm trying to train a model utilizing support vector machines as part of my analysis. On my little macbook pro, it seems endless. Right now I'm using an RBF kernel and am not seeing an end to this computation.
I do however have access to 100+ compute nodes with 16 cores a piece. My problem is, I don't know how to utilize this to my extent as I am lacking knowledge on how I should approach this SVM. Right now I am using scipy.
SciPy Code
def makeModelandPrediction(trainData, trainLabel, testData):
    model = svm.SVC(
        C=1.0,
        cache_size=200,
        class_weight=None,
        coef0=0.0,
        decision_function_shape=None,
        degree=3,
        gamma='auto',
        kernel='rbf',
        max_iter=-1,
        probability=False,
        random_state=None,
        shrinking=True,
        tol=0.001,
        verbose=False,
        )
    model.fit(trainData, trainLabel)
    prediction = model.predict(testData)
    return prediction

I've already preprocessed the data and done a 70/30 train/test split on the data. Can someone point a beginner in the right direction?


texts:
I am working on a project that uses object detection. I have logo images that need to be detected in a video. I am doing this in keras. I followed this blog to convert the yolo weights to a keras model.

Now I would like to train this keras model with my dataset (transfer learning). I got right dataset with xmin, ymin, xmax, and ymax.

My Approach: remove the last layer and use the sigmoid function to define the classes. However, the above link says that this keras model predicts both boundary boxes and classes. I am not sure how to achieve these boundary boxes from the model itself.


texts: Machine learning worker performance features for optimum allocation of tasks to workers

texts:
Suppose we have an English to French translation task in a company, and there are 100s of workers who are proficient in doing this task, but each worker has its own unique attributes which enables them to do certain translations in a better way than the others (e.g., a translator who is a doctor would do better translations of medical documents). Consider 1000s of documents for translation received per day, and each document takes one day to translate. So, there is a two-fold problem:

Allocating the tasks to available translators using a queuing model or any other efficient allocation mechanism.
Learning the worker model based on the past performance of each (e.g., correctness scores, static parameters like skill set, qualifications, experience, etc.) using a machine learning system.

Is there a setup/system/model which solves both of the above problems in one cohesive system? From my reading, this problem calls for the application of the assignment problem, user modelling, queuing, and machine learning optimization. I am searching for existing models or frameworks which integrate all of this, can anyone give me some direction for this task of optimum allocation?


texts: If I have got different routes - a series of (lat, lng) points, how to get the similarity of different routes?

texts:
It is a real world use case. For example, a route from place A to place B can be different series of lat, lng points - the different trips though they are exactly the same sequence from Street x and then Road y then High way z.

The differences are the locations are reported in different time (for example each trip reports the location 1 minute) and the Vehicles appear in the different lane of the same street.

So, do you have some ideas about how to compute the similarity of two different trips belong to the same route (kinds of mapping different trips to the same route ).


texts: What do I do when my column has 50% data missing?

texts:
I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should i impute this? Do i replace ? with unknown or do i replace it with NaN.

Since my end goal is a classification problem i will be applying various algos on it such as NB classifier, random forest, decision tree, K neighbours. And sometimes it does not matter whether you impute or not since some algos do it for you


texts: Solve a regression equation algebraically in python

texts:
I fit a regression model using python statsmodels.formula.api package and obtain the coefficients/results. Here's the model:
Price ~ Availability + Income + unemployment + labor_force + HH

The coefficients are:
coef
Intercept   18.8172
Availability 0.0860
Income  9.213e-05
unemployment -1.562e-05
labor_force 5.78e-06

I'd like to calculate the value of Availability for when Price = 0 and for different values of independent variable which vary by time. Basically, solve for Availability when Price = 0.
0 = Intercept + b1 * Avail + b2 * Income + b3 * U + b4 * L
I need python to solve this equation for Avail.
I realize I can do this with a python function by simply solving the equation, ,however, I was wondering if there are any in built methods I could use to obtain the equation and solve?


texts: Validation loss increases and validation accuracy decreases

texts:
I have an issue with my model. I'm trying to use the most basic Conv1D model to analyze review data and output a rating of 1-5 class, therefore the loss is categorical_crossentropy. Model structure is as below
# define model
model = Sequential()
model.add(Embedding(vocab_size, 100, input_length=max_length))
model.add(Conv1D(filters=32, kernel_size=8, activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(10, activation='relu'))
model.add(Dense(5, activation='softmax'))

# compile network
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# fit network
model.fit(final_X_train, final_Y_train, epochs=5, batch_size=50, validation_data=(final_X_val, final_Y_val), callbacks=callback)

Total params: 14,977,717
Trainable params: 14,977,717
Non-trainable params: 0
Train on 212135 samples, validate on 69472 samples
Epoch 1/5
loss: 0.9452 - acc: 0.5781 - val_loss: 0.8440 - val_acc: 0.6309

Epoch 2/5
loss: 0.7707 - acc: 0.6711 - val_loss: 0.8234 - val_acc: 0.6433

Epoch 3/5
loss: 0.5807 - acc: 0.7657 - val_loss: 0.9144 - val_acc: 0.6345

Epoch 4/5
loss: 0.3736 - acc: 0.8575 - val_loss: 1.1982 - val_acc: 0.6194

Epoch 5/5
loss: 0.2285 - acc: 0.9173 - val_loss: 1.5770 - val_acc: 0.6073

Training acc increases and loss decreases as expected. But validation loss and validation acc decrease straight after the 2nd epoch itself. The overall testing after training gives an accuracy around 60s.
The total accuracy is :  0.6046845041714888

I've already cleaned, shuffled, down-sampled (all classes have 42427 number of data samples) and split the data properly to training(70%) / validation(10%) / testing(20%).

If you see any improvements to fix this problem, please let me know. :)


texts: Histogram alternatives for two sets of data combined

texts:
It's my first question here and I hope it's not too simple for a data science question.

I'm looking for a clear and less typical way to visualize my data. I have two data sets here - first one hatched and semi-transparent and the other one non-transparent fully-colored. I'd like to emphasize how the hatched histograms are widening while the fully-colored ones are not. I also want to keep something that shows that the data in hatched and non-hatched correspond with each other as I did in this histogram with the colors - red full-colored histogram data corresponds to red hatched data.
Any ideas how to visualize it differently? Is the visualization presented here good enough to show the things I described here?


texts:
How to plot two categorical variables in Python or using any library? I want to plot the Playing Role of a Cricketer (Batsman, Bowler, etc.) VS Bought_By (Franchise Names, e.g., CSK, DC, etc.). The logic here is to plot the cricket role vs franchise.
The Columns:
df.Playing_Role
df.Bought_By
One of these columns can be converted to continuous numerical, but is there any direct way without converting them?


texts: SGD data should be randomly selected or sequentially feed?

texts: When training neural network, is it better to randomly choose data for every batch or feed the data sequentially? Why?


texts: Pytorch XLA to solve the spawn problems in a Colab Env

texts:
As reference only, here is my code

It seems that torch.multiprocessing.set_start_method("spawn") can't be used in an Colab Env. Only 'fork' is allowed.

I have implemented A3C - Data Parallelism to solve the Breakout Atari Game. As I use multi-agents, I need to spawn several processes.

This is representing a single agent :

TotalReward = namedtuple("TotalReward", field_names="reward")

def data_func(net, device, train_queue, batch_size, entropy_beta,
              env_name, n_envs, gamma, reward_steps, **kwargs):

    env = GymEnvVec(env_name, n_envs)
    agent = Agent(net, batch_size, entropy_beta)
    exp_source = ExperienceSourceFirstLast(env, agent, gamma, reward_steps)

    for exp in exp_source:
        new_rewards = exp_source.pop_total_reward()
        if new_rewards:
            train_queue.put(TotalReward(reward=np.mean(new_rewards)))
        train_queue.put(exp)

and here is how I set and start several agents

  train_queue = mp.Queue(maxsize=params["process_count"])
  data_proc_list = []

  for _ in range(params["process_count"]):
      data_proc = mp.Process(target=data_func,
                            args=(net, device, train_queue),
                            kwargs={**params})
      data_proc.start()
      data_proc_list.append(data_proc)

Can I set multi-agents using Queue and Process in Colab? I have thought using spawn function from here using Pytorch XLA. What do you think?


texts: Neural Network is overfitting when using bigger dataset

texts:
I'm tring to train a model using CNN (supervised) to solve a binary classification problem. I have pretty big dataset containing 2 800 000 samples, each having 100+ features. Because training with such big dataset takes about 20 hours I've built my net with using only 25% of dataset (about 700 000 samples). And if model have promising resoults on smaller dataset, then use full dataset to train.

Here is something strange. I get pretty good resoults when using 25% of dataset

It looks realy good, so I decided to train a model with using full dataset, and i get this
which is awful. Model is clearly overfitting.
I don't have much experience with machine learning, but as far as I know, one way to solve overfit problem is to increase dataset. But here, problem with overfiting occures only when I use bigger dataset. Other way to solve it would be to decrease number of layers, or number of neurons, but the same CNN have trained a good model on smaller dataset.
My CNN looks like this
2 x Linear Layer
1 x Convolution
2 x Linear
4 x Convolution
6 x Linear

And before every linear layer there is a dropout 50%, and dropout 20% is on input. And after every Convolution there is also a Avg pooling layer.
Because dataset is pretty big, batch size is 256 for now.
One of my ideas is to increase a batch size to 512, or even 1024 when training with full dataset, or to add a batch normalization layers, but before I do that, I want to ask for your advices, because, as I said, training takes about 20 hours...

EDIT.
I collect data on every 1% of batches. So data is collected 100 times for each epoch. Model is trained on 100 epochs, so data are collected 10000 times, and this value is on x axis.
Also, to bad but I unfortunetly can not share the code.


texts: Algorithm or formula to measure happiness?

texts:
I am working on a project that takes in comments in a given Facebook page and determines the average happiness rating for it based on users of the page. My question is where would I find a formula, rating, or any other literature that would help me find a way to measure happiness using words and collections of words?

I'm looking for something similar to cosine similarity I suppose, but not to find similar words, but to find the average positivity or happiness related to a word or collection of words.

I'm not entirely sure this is the correct place for this question, but it has to do with data, and big data within Facebook so I am hoping to either get a question or be directed to somewhere that I may be able to find my answer. Thanks in advance for your help.


texts: What would you recommend to know before considering applying for a master in Machine Learning?

texts:
As someone who is about to finish a bachelor in Software Engineering still learning the basics of machine learning but would like to enroll in a master's course focused on machine learning.

What would you recommend to learn before considering applying for a master?

I'd like to hear experiences of people who have gone through ML or computer science with a focus on ML or have taught at that level, what is the expected knowledge you are suppose to have?


texts:
As we all know, XGBoost constructs trees based on gradient.
I wonder how does XGBoost define gradient of MAE loss, as MAE itself is not differentiable.

After some digging of the source code, I found the implementation of MSE loss here. But I can't find any implementation of MAE loss. The original paper didn't discuss MAE loss either.


texts: Neural network for a data with 5 inputs and 1 output of 3 different types


texts: How to train the generator in a recurrent GAN (Keras)

texts:
I am trying to train a Recurrent GAN that is meant to generate geospatial movement data (sequences of 3-tuples of latitude, longitude and time). You may simply consider it a sequences of vectors with 3 features.

I am using Keras on top of TensorFlow for this at the moment. The following are the two networks I have come up with. I know the hyper-parameters are probably terrible. I just want to get the general architecture working and then tune them later.

Generator:

generator = tf.keras.Sequential([
    tf.keras.layers.GRU(256, return_sequences=True, stateful=True, input_shape=(None, 3), batch_size=1),
    tf.keras.layers.TimeDistributed(tf.layers.Dense(3))
])

Discriminator:

discriminator = tf.keras.Sequential([
    tf.keras.layers.GRU(128, input_shape=(None, 3)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

For the sake of completeness, here is what my combined adversarial model looks like with discriminator.trainable = False:

adversarial = tf.keras.Sequential([
    generator,
    discriminator
])

The following is my training loop. Part 1 is for training the discriminator. I grab some real sequences and I run a loop to generate some fake ones using the generator, then I create labels for them and train the discriminator on both no problem.

I struggle to figure out how to train the generator, though. As you can see, I need to call the generator in a loop in order to generate the vectors of the sequence one by one. I can't really use train_on_batch
 with that.

for epoch in range(EPOCHS):
    for batch in range(n_batches):

        # --- PART 1: Train the discriminator ----------------------------------

        # Use the generator to generate a half a batch full of fake data
        fake_data = []
        for i in range(BATCH_SIZE // 2):
            length = df_train.sample(1).squeeze().shape[0]
            start = data_train[np.random.randint(data_train.shape[0])]

            generated = np.array([start])

            generator.reset_states()

            for i in range(1, length):
                input = generated[-1:]
                input = np.array([input])

                prediction = model.predict(input, batch_size=1)

                prediction = np.squeeze(prediction, axis=0)
                generated = np.concatenate([generated, prediction])

            fake_data.append(generated)

        fake_labels = np.zeros((BATCH_SIZE // 2, 1))

        # Get half a batch of real data
        real_data = df_train.iloc[batch * BATCH_SIZE:(batch + 1) * BATCH_SIZE].tolist()
        real_labels = np.ones((BATCH_SIZE // 2, 1))

        # Train the discriminator on half a batch of real and half a batch of fake data
        d_loss_fake = discriminator.train_on_batch(fake_data, fake_labels)
        d_loss_real = discriminator.train_on_batch(real_data, real_labels)
        d_loss = 0.5 * np.add(d_loss_fake, d_loss_real)

        # --- PART 2: Train the generator --------------------------------------
        g_loss = # HERE, I AM STRUGGLING

I'm open to any ideas, even changes to the entire approach, if you believe RNNs aren't the way go to about this. I decided to use RNNs because I need to generate sequences, so they naturally fit, obviously. Also, the sequences need to be of variable length, so using some sort of fully connected network would be possible but rather awkward.

I really hope someone can help, as I'm trying to figure this out for days now.

Please ask, if there is any information you are missing.


texts: Predict variable from time series when there are many observations in the same date


texts: How is this score function estimator derived?

texts: In this paper they have this equation, where they use the score function estimator, to estimate the gradient of an expectation. How did they derive this?


texts:
LDA: linear discriminant analysis

Suppose we have a classification problem. I understand that the data can be such that the features may have discrete values or continuous values.

Suppose our data contains continuous feature values. Then we can apply Naive Bayes using a distribution. Lets assume the data to be normally distributed and so use Naive Bayes with normal distribution. We can also apply LDA which also uses Normal distribution.

Using Naive Bayes we assume the features to be independent and by using LDA we assume the covariance to be same for all the classes.

How does these assumptions make these 2 models perform differently and which is a better model and in which conditions?


texts: What type of neural network could emulate a binary to HTML conversion tool?

texts:
I've got a problem, which I thought could be solved by using a neural network:

I've got a binary file and a tool that converts that file into a readable html file (probably a text file as well). How the tool is converting that file is a secret and it is too complex to reverse engineer (>100000 lines of assembler for this function) and the binary files are too big and complex to interpret for a human in a feasible time. My idea was to teach a neural network with those binary files and the known output to train the network, how to convert these files. The binary files differ extremely in size(from a few kb up to serval mb).

I've got some basic knowledge of neural networks but I'm not sure where to start to search for the right approach. A friend of mine (with a bit more basic knowledge) said this is definitely doable and suggested a seq2seq model in tensorflow, but after some research I'm not sure if this is the right one for my problem.


texts: Using keras with sklearn: apply class_weight with cross_val_score

texts:
I have a highly imbalanced dataset (± 5% positive instances), for which I am training binary classifiers. I am using nested 5-fold cross-validation with grid search for hyperparameter tuning.

I want to avoid undersampling, so I have been looking into the class_weight hyperparameter. For sklearn's decisiontree classifier, this works really well and is easily given as a hyperparameter. However, this is not an option for sklearn's neural network (multi-layer perceptron) as far as I can tell. I have been using Keras instead and I can apply class_weight with gridsearchCV, but not with cross_val_score.

Is there a way to use class_weights in keras with cross-validation?


texts: Does it make sense to train kalman filter on unscaled (Timeseries-)data, to clean it?

texts:
I have Timeseries data to clean and I got the tip to use a kalman Filter.
My Question is, does it make sense to "fit" this kalman-filters parameters to an unscaled and unnormalized (therefore raw) dataset of timeseries, with the purpose to clean it?
Or will I get far more accurate results when normailzing, and then denormalizing it?


texts: How to decided which test of normality to use

texts:
Given a data set with features, that you want to check for normality, one feature at a time w/o a multivariate normal test, how do you decided which test of normality to use? For example, using the python module scipy I could either use:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html

or I could use:

http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.mstats.normaltest.html

For example on the same feature I get these outputs:

scipy.stats.shapiro(data[:,0])
(0.9985173940658569, 0.77845299243927)

scipy.stats.mstats.normaltest(data[:,0])
NormaltestResult(statistic=1.492603328675163, pvalue=0.47411675723570479)

scipy.stats.anderson(data[:,0], dist='norm')
AndersonResult(statistic=0.17542490527580412, critical_values=array([ 0.573,  0.653,  0.783,  0.913,  1.086]), significance_level=array([ 15. ,  10. ,   5. ,   2.5,   1. ]))


texts: How to get the accuracy from merged models?

texts:
I am implementing hybrid RS using collaborative RS and content based RS. Collaborative RS with Artificial NN with MLP and content based RS with Skip-gram model. I have merged these to into a sequential model. But I did not have any idea how to get the accuracy from this merged model.

Here is my skip gram

skip_grams = [skipgrams(wid, vocabulary_size=vocab_size, window_size=10) for wid in wids]

num_users = int(articles.user_id.max())
num_articles = int(articles.id.max())

Here are my two models

latent_dim = 10

# Define inputs
article_input = Input(shape=[1],name='article-input')
user_input = Input(shape=[1], name='user-input')

# MLP Embeddings
article_embedding_mlp = Embedding(num_articles + 1, latent_dim, name='article-embedding-mlp')(article_input)
article_vec_mlp = Flatten(name='flatten-article-mlp')(article_embedding_mlp)

user_embedding_mlp = Embedding(num_users + 1, latent_dim, name='user-embedding-mlp')(user_input)
user_vec_mlp = Flatten(name='flatten-user-mlp')(user_embedding_mlp)

# MF Embeddings
article_embedding_mf = Embedding(num_articles + 1, latent_dim, name='article-embedding-mf')(article_input)
article_vec_mf = Flatten(name='flatten-article-mf')(article_embedding_mf)

user_embedding_mf = Embedding(num_users + 1, latent_dim, name='user-embedding-mf')(user_input)
user_vec_mf = Flatten(name='flatten-user-mf')(user_embedding_mf)

# MLP layers
concat = merge([article_vec_mlp, user_vec_mlp], mode='concat', name='concat')
concat_dropout = Dropout(0.2)(concat)
fc_1 = Dense(100, name='fc-1', activation='relu')(concat_dropout)
fc_1_bn = BatchNormalization(name='batch-norm-1')(fc_1)
fc_1_dropout = Dropout(0.2)(fc_1_bn)
fc_2 = Dense(50, name='fc-2', activation='relu')(fc_1_dropout)
fc_2_bn = BatchNormalization(name='batch-norm-2')(fc_2)
fc_2_dropout = Dropout(0.2)(fc_2_bn)

# Prediction from both layers
pred_mlp = Dense(10, name='pred-mlp', activation='relu')(fc_2_dropout)
pred_mf = merge([article_vec_mf, article_vec_mf], mode='dot', name='pred-mf')
combine_mlp_mf = merge([pred_mf, pred_mlp], mode='concat', name='combine-mlp-mf')

# Final prediction
result = Dense(1, name='result', activation='relu')(combine_mlp_mf)
model = Model([user_input, article_input], result)

model.summary()

#skip-gram model
word_model = Sequential()
word_model.add(Embedding(vocab_size, embed_size, embeddings_initializer="glorot_uniform", input_length=1))
word_model.add(Reshape((embed_size, )))

context_model = Sequential()
context_model.add(Embedding(vocab_size, embed_size, embeddings_initializer="glorot_uniform", input_length=1))
context_model.add(Reshape((embed_size,)))

model1 = Sequential()
model1.add(Merge([word_model, context_model], mode="dot"))
model1.add(Dense(1, kernel_initializer="glorot_uniform", activation="sigmoid"))

# view model summary
model1.summary()

Here is my merged model

merged_model = Sequential()
merged_model.add(Merge([model, model1], mode='concat'))
merged_model.add(Dense(1, kernel_initializer="glorot_uniform", activation="sigmoid"))
merged_model.compile(loss="mean_squared_error", optimizer="rmsprop")

merged_model.summary()

Please can anyone help me to get the accuracy from this merged model?


texts: I do feature engineering on the full dataset, is this wrong?

texts:
I am aiming to predict the number of days it takes to sell a given property, let's call this variable "DaysForSale" - in short DfS
Using the DfS I created a variable called "median_dfs_grouped_street_name" which returns the median days it takes to sell a property for the different streets available in the dataset. (The street names are all categorized).
After this, I do my train/test split and run my Random Forest method.
Using the feature_imporatances function I see that the new feature is the second most important, which makes me wonder if this is the correct approach?
I have two questions:

Is it wrong to develop features using the target variable?
Is it wrong to do feature engineering on the full dataset?


texts: Which Machine Learning Algorithm should be used to predict Waveforms from an already existing collection of waveforms ?

texts:
I am new to machine learning and I was wondering what algorithm to use to predict waveforms from an already existing waveform data samples.

I have the x coordinates and Y coordinates or (waveforms) for a few set of features for the different set of values. I want to be able to predict further waveforms for more values using this as training data. Which type of Machine Learning Algorithm should be used in such cases ?


texts:
I have two questions:

1) I have images and I must predict a continuous value.

What is the approach I must follow?

Use for example a pretrained network like this?

 last_layer = pretrained net
 x = Flatten()(last_layer)
 x = Dense(512, activation='relu')(x)
 x = Dropout(0.4)(x)
 predictions = Dense(1)(x)

So, just use a Dense layer at the end with 1 node?

and then procees to prediction as usual?

2) If I have again a numerical values to predict but it is ordinal.

If I want to go with regression in this case, can I work like in previous example?

And just assign some probabilities to the result?

If result >0 and <=0.5 then classify as 0
If result >0.5 and <=1 then classify as 1

and so on?


texts:
I have some test results in an NxN matrix which counts total fail results for that cell. I want to see if the failures generate any geometric patterns that can be identified automatically.

The approach I am planning is, I want to use the result matrix and use a Neural Network to Generate Classification. From the classification I want to see if it was actually able to find out some classes that the failure patterns can be identified spatially.

So I was planning to use kohonen SOM to classify the data. From kohonen SOM with automatic class detection how to find out the only the number of classes inferred?


texts: Building a content-based music recommendation system

texts:
I am trying to build a recommendation system in Python that recommends songs based on a playlist.

What I have is two datasets:
1. One dataset consists of 350 songs from my playlist and 13 acoustic features for each one like timber, energy, key, tempo, etc. that I extracted using the Spotify API
2. The other dataset consists of 340 000 songs and their acoustic features (got it from here: https://components.one/datasets/billboard-200/)
I've gotten both datasets in the same format and ready to be worked with.

Now what I am trying to do in Python is use my first dataset to get 30-40 songs with similar acoustic features from the second dataset but I have no idea how to approach this.
Should I use some machine-learning models or do something entirely else?
I thought about comparing the songs from my first dataset to the songs on my second dataset and pulling the ones with a similarity score of let's say over 70% or something like that but I feel like there is probably a much better way of doing this.


texts: Why is the softmax function often used as activation function of output layer in classification neural networks?

texts: What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?


texts: Sequence to sequence RNN model, maximum number of training size

texts:
So when running this example script from Keras repo (https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py), I found that we can easily run into out of memory for the input or output one-hot encoding in this code:

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
dtype='float32')

When I have a huge size of training samples, this one-hot does not fit into memory. Is there way to handle this issue?

UPDATE:
I changed float32 to uint, but thats about the smallest one hot array can get.