texts: Is the prediction algorithm absolutely the same for all linear classifiers?

texts:
Is the prediction algorithm absolutely the same for all linear classifiers and linear regression algorithms?

As known, any linear classifier can be described as: y = w1*x1 + w2*x2 + ... + c

There are two broad classes of methods for determining the parameters of a linear classifier (generative and discriminative): https://en.wikipedia.org/wiki/Linear_classifier

Linear Discriminant Analysis (or Fisher's linear discriminant), Naive Bayes classifier
Logistic regression, Perceptron, Support vector machine

Question: Is it true that linear classifiers differ only in the Learning algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2 + ... + c?

If I used one method for Training (for example SVM with linear kernel function), then can I use other method for prediction (for example Perceptron) with the same output result?
And can I do the same, if I used SVR (Support Vector Regression with linear kernel function) for Training, then can I use Perceptor as linear regression method for Prediction value?


texts: Unbalanced target variable in Orange, how do I balance it?

texts:
So I want to perform a predictive model to predict churn.

I have 2 datasets, one with churn and the other without (so I can later perform predictions).

The issue is that I think my Confusion matrix is kinda bad since my target variable is highly unbalanced:

which mostly leads to this confussion matrix:

(Similar values for both logistic regression and decision tree).

This is my workflow:

Is there any way to balance the data? I can't find it in the Orange documentation.


texts: Number of Parameters in Multidimensional rnn

texts:
I have the following equation for the hidden unit in a Multidimensional RNN and I need to calculate the total number of trainable parameters:

$$h^{i,j} = \sigma (W_{in}x^{i,j} + W_{left}h^{i-1,j} + W_{top}h^{i,j-1})$$
Assume that the input grid is $n \times n$, each input $x^{i,j}$ is a
$c$-channel vector, and the hidden state $h^{i,j}$ has dimension $h$.
How many trainable parameters does this architecture have?

So I want to know if I'm correct with the sizes of the W matrices:
\begin{aligned}
W_{in} = n \times h \\
W_{left} = h \times h \\
W_{top} = h \times h
\end{aligned}
If so, where does the $c$-channel vector take place? What am I doing wrong? Thank you


texts: Unexpected results from scikit learn regression decision tree

texts:
Apologies for this newbie question. I have a scikit learn DecisionTreeRegressor with muti-variable output. If the output is in the format  [ output_var1, output_var2 ], where each variable is a continuous number not an integer, why the result is [1, 1] instead of [1.5, 1.5] ? What needs to be changed in this model to get [1.5, 1.5] ?

from sklearn.tree import DecisionTreeRegressor

X = [ [1,1], [2,2], [3,3] ]
y = [ [1,1], [2,2], [3,3] ]

print('X:' , X)
print('-----------------------------')
print('y:' , y)
print('-----------------------------')

regr = DecisionTreeRegressor()
regr.fit(X, y)

X_test = [ [1.5, 1.5] ]
print('X_test:' , X_test)
print('-----------------------------')

y_result = regr.predict(X_test)
print('y_result:' , y_result )

Result:

X: [[1, 1], [2, 2], [3, 3]]
-----------------------------
y: [[1, 1], [2, 2], [3, 3]]
-----------------------------
X_test: [[1.5, 1.5]]
-----------------------------
y_result: [[1. 1.]]


texts: Transform single-label data set into multi-label data set

texts:
I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in.

When the dataset was put together, it was assembled under the assumption that each string of text can only belong to one group. In actuality, the text can belong to more than one group simultaneously.

Instead of going back to the drawing board and manually labeling the data again, I want to try and convert this single-label data set into a multi-label data set.

I've tried one method with questionable results. I built a linear regression that predicts each category individually, and appended those predictions to the original data. While this gave me data in the structure I needed, it yielded lackluster results. Most strings of text still only belong to one category (many should belong to multiple), and a good portion weren't assigned any label at all.

It seems that even if I can "Frankenstein" this data together, it may not serve as quality training data. I'm curious, is there's any great way of transforming this single-label data into multi-label data?


texts: Problem with elastic constraints in PuLP

texts:
(This is my first question on Stack Exchange)
I am working on a production allocation problem, whereby sales orders have to be allocated over three production plants. I am using PuLP in Python, and this works fine until I try to make the constraints "elastic". The elasticity is needed when the total of the sales orders is larger than the total capacity, and I have to increase the capacity of the plants, although with a "penalty". (Since producing in overtime increases the cost).
I got stuck in the lack of user-friendly documentation on this aspect of PuLP (sorry to say so) and encounter several difficulties. For applying "makeElasticSubProblem", I need to give the constraint a name. However,
a) When I use the traditional syntax of prob += lpSum([alloc[s][l]for s in so]) <= cap_dict[l], 'Myname', and when I use Myname in the next line, Python returns a message that Myname is not defined.
b) when I use the following syntax: Myname = LpConstraintlpSum([alloc[s][l]for s in so]), sense = 1, rhs = cap_dict[l]), the name is accepted, but the definition of the elasticity that follows for the name, seems to be "ignored" by the script.
Could someone give me a hint about what I am doing wrong ? Thanks!
Here follows a simplified version of my code (working with Python 3.8.2 64-bit):
from pulp import *

# define the locations and their production capacity
#---------------------------------------------------
location = ['locA', 'locB', 'locC']
capacity = [90, 60, 20]
cap_dict = dict(zip(location, capacity))

# define the sales orders (so) and their volumes (demand)
#--------------------------------------------------------
so = ['s1', 's2', 's3', 's4', 's5']
demand = [20,10,15,8,5]
order_dict = dict(zip(so, demand))

# define the problem
#-------------------
prob = LpProblem("Production_planning",LpMaximize)

# define the decision variables
#------------------------------
alloc = LpVariable.dicts("Alloc", (so, location), cat='Integer')

# set the objective function
#---------------------------
prob += lpSum(alloc[s][l] for s in so for l in location)

# define the constraints
#-----------------------
# 1) allocations should be positive
for s in so:
    for l in location:
        prob += (alloc[s][l] >= 0)

# 2) allocation limited by the capacity of the location
for l in location:
    prob += lpSum([alloc[s][l] for s in so]) <= cap_dict[l]

# 3) Location A should receive > 50% of all the allocations
prob += (alloc[s]['locA'] >= lpSum(alloc[s][l] for l in location)/2)

# 4) allocation limited by the amount ordered
for s in so:
    prob += lpSum([alloc[s][l] for l in location]) <= order_dict[s]

# solve the optimization problem
#-------------------------------
prob.solve()
print("Status :", LpStatus[prob.status])

for v in prob.variables():
    if "Alloc_" in v.name:
        if v.varValue > 0:
            print(v.name, v.varValue)

After the following definition of the capacity constraints, the specification of the 20% upward elasticity of the constraint for location C is simply ignored (the program allocates to location C any large amount if I increase the sales orders, without any limit):
# capacity constraints:
#---------------------
# same definition for locations A and B

for l in ['locA', 'locB']:
    prob += lpSum([alloc[s][l] for s in so]) <= cap_dict[l]

# different and specific definition for location C, where an elastic capacity increase needs to be introduced:

cap_locC = LpConstraint(lpSum([alloc[s][l]for s in so]), sense = 1, rhs = cap_dict['locC'])
elastic_cap_locC = cap_locC.makeElasticSubProblem(penalty = 100, proportionFreeBoundList = [0,0.2])


texts:
Let our data be a set of 2d points sampled from a line $ax+by=1$ (that we do not know) plus some noise $\nu$.

This is meant to represent a frontier of development - we can build a car with a powerful engine or very fuel efficient, but it is hard to do both.

How can I go about estimating the line $ax + by = 1$?


texts:
Google Trip is now performing automated clustering of travel items into trip bundles.

https://support.google.com/inbox/answer/6228360?hl=en-GB

So, if in your mail you have a flight confirmation mail and a hotel confirmation mail related to the same trip, Google will cluster them automatically into a trip bundle.

This means that that:

They scan all your e mail
They find the ones related to trips
They cluster those for the same trip into a bundle.

My question regards point 3. Which kind of algorithm is needed to perform such task?

Consider that the task is unsupervised, because the user can not interact with the system while creating the bundles..


texts: Multi-label classification for text messages (convert text to numeric vector)

texts:
Given a dataset of messages which are labeled with 20 features, I want to predict the value of each feature for a new message.

Dataset example:

message      feature1 feature2 feature3 feature3 feature4 ...
'hi'         1        0        1        1        0        ...
'i am bussy' 0        0        0        0        1        ...
...          ...      ...      ...      ...      ...      ...

Split data into train & test to train the model:

from sklearn.model_selection import train_test_split

x= df.iloc[:,0:1].values
y = df.iloc[:,1:-1].values
train_x, test_x, train_y, test_y = train_test_split(x, y, random_state=42)

Now, my train_x is an array of text values (impossible to fit into a train model), how could I convert them to numeric vectors?


texts: How to get a similarity vector from two vectors?

texts:
I want to make a classification model for 3 classes,  i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer.

inputs = Input(shape=(2,n_timesteps))
embedding_inputs = embedding_layer(inputs)

sentence1 = Lambda(lambda x: x[:,0,:,:])(embedding_inputs)
sentence2 = Lambda(lambda x: x[:,1,:,:])(embedding_inputs)

conv_sentence1 = Conv1D(filters=64, kernel_size=3, activation='relu',
input_shape=(n_timesteps,n_features))(sentence1)
conv_sentence2 = Conv1D(filters=64, kernel_size=3, activation='relu',
input_shape=(n_timesteps,n_features))(sentence2)

pooling_sentence1 = MaxPooling1D(pool_size=2)(conv_sentence1)
pooling_sentence2 = MaxPooling1D(pool_size=2)(conv_sentence2)

flat_sentence1 = Flatten()(pooling_sentence1)
flat_sentence2 = Flatten()(pooling_sentence2)

concat_senrences = concatenate([flat_sentence1,flat_sentence2])
dense_layer = dense(50)(concat_senrences)
dense_prediction = dense(3,activation='softmax')(dense_layer)

but i get an early overfetting, so i thought that the problem comes from "sentence 2", each observation has an unique "sentence 1", ,instead, "sentence 2" can be exists in several observations, in that case the neural network relies strongly on it, so i want to combine two sentences and apply an unique CNN layer, that's why i asked how to obtain a similarity vector.

thanks !!!


texts: Technical term for using regular expressions to classify text?

texts:
Background

I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext.
He wants to classify the claims as either having been "approved", "denied", or "remanded".
Each file has a section starting with the string "ORDER" followed by some sentences that explain the decision.
I suggested that we use regular expressions to:

extract the "ORDER" section,
separate the section into separate orders (as each case can involve multiple claims),
extract the claim from each order, and
classify each claim as approved or denied depending on whether the string "approved" or "denied" appears in the order regarding that claim.

Problem / Question

The researcher needs to describe his method in his paper and wants to know what the correct / technical term for this method of classification would be:

  I do want to be able to explain each step, including how you changed it to a standard format. Is there a genre of text analysis we can use to classify your ultimate regular expressions approach? It would certainly be data mining, right?


texts: Run python script using spark-submit on windows 7

texts:
I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it possible to run such a script using spark-submit? In the python script I included this block of code for spark context.

import findspark
findspark.init()
import pyspark
from pyspark import SparkContext
APP_NAME = 'kineticPCA'
conf = SparkConf().setAppName(APP_NAME)
conf = conf.setMaster('spark://192.168.xx.xxx:7077')
sc = SparkContext(conf=conf)

This is the how submit the job on command prompt, on the spark bin directory

spark-submit G:/guru orders/pca kinetics/files/kineticPCA.py

This is the error I get:

  Error: Cannot load main class from JAR G:/guru with URI G. Please
  s through --class.
  Run with --help for usage help or --verbose for debug output

What could be the cause of this error?


texts: Is my dataset unlearnable, or is my LSTM model not smart enough?

texts:
I have time-series data obtained from a video. The data is composed of bitrate and corresponding label pairs for each timestamp:

The distribution over the first 30 seconds is as follows:

I have built an LSTM model for this dataset to be able to classify the labels based on the bitrate. However, it seems that my model is not able to learn. Validation accuracy starts from approximately 0.3 (makes sense, since I have 2 classes (log2 = 0.3)) and it does not improve.
Do you have any idea about this? Is it normal considering this sample data distribution, or is something might be wrong with my model? Thanks!


texts: State-of-the-art Python packages that can evaluate language similarity

texts:
I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and calculating the perplexity values for a list of sentences.
I found that the package KenLM (https://www.aclweb.org/anthology/W11-2123/) was often used to do this task. However, it's kind of old (published in 2011).
On the other hand, I noticed that the two most famous state-of-the-art NLP packages, BERT and GPT-2, are both about pre-trained models.
I wonder if there is any package newer than KenLM suitable for this kind of likelihood evaluation task.


texts: The purpose of Autoregressive Flows and RealNVP in Normalizing Flows

texts: Do I understand correctly that the primary purpose of MAF/IAF/RealNVP etc. is to be able to train also the depth/complexity of the NF, besides other parameters?


texts: COVID: How is the R coefficient calculated?

texts:
I'm having a hard time trying to figure out what the formula for estimating R is.
I assume it's a coefficient used in some sort of regression type?
If that is the case is it an exponential, logarithmic or another type of regression from which I extrapolate this coefficient?
Thanks, hope someone can help this ignoramous :)


texts: In SVM Algorithm, why vector w is orthogonal to the separating hyperplane?

texts:
I am a beginner on Machine Learning.
In SVM, the separating hyperplane is defined as $y = w^T x + b$.
Why we say vector $w$ orthogonal to the separating hyperplane?


texts: Comparative Analysis of two sets of data

texts:
I have 2 sets of data which consist of marks of 60 students in a particular subject in year 1 and year 2 respectively.

Year 1 : 86, 76, 87, 67, 89, 95.... so on (60 students)

Year 2 : 82, 67, 99, 77, 65, 78.... so on (60 students)

I want to compare these two sets of marks and find if year 2 students performed better than year 1  students.

What kind of techniques are appropriate for this problem? I was told that the statistical techniques like ANOVA, hypothesis testing techniques are too common for this problem and I was asked to try something with Machine learning.

Does machine learning fit in the picture? If yes, what kind of algorithms could be used?


texts: Using the Datumbox Machine Learning Framework for website classification - guidelines?

texts:
A short while ago, I came across this ML framework that has implemented several different algorithms ready for use. The site also provides a handy API that you can access with an API key.

I have need of the framework to solve a website classification problem where I basically need to categorize several thousand websites based on their HTML content. As I don't want to be bound to their existing API, I wanted to use the framework to implement my own.

However, besides some introductory-level data mining courses and associated reading, I know very little as to what exactly I would need to use. Specifically, I'm at a loss as to what exactly I need to do to train the classifier and then model the data.

The framework already includes some classification algorithms like NaiveBayes, which I know is well suited to the task of text classification, but I'm not exactly sure how to apply it to the problem.

Can anyone give me a rough guidelines as to what exactly I would need to do to accomplish this task?


texts: How to create full graph from vertex data frame with igraph?

texts:
Here is a DF which I take as a representation of my future vertices:

Id   Name
01   Alpha
02   Beta
03   Delta
04   Epsilon

I want to get an igraph representation of a full graph with these nodes in R. As far as I know igraph's graph.full function cannot take DF as input and I cannot really want to create an intermediary DF with all possible conection combination. Any clever idea?


texts: Batch Normalization For Federated Learning

texts:
As you know, in a federated learning setting, clients train local versions of the joint global model with their Non i.i.d data and each submit an update to the global model which would be aggregated into the next joint global model.
The normalization which happens by Batch Normalization layers during the training phase is based on the local batch statistics. My question is, How should one aggregate these local statistics (batch normalization parameters) for the global model so they represent the global statistics of all the data? I am talking about beta, alpha, moving mean and variance for each batch normalization layer. Should we treat them like weight and biases of fully connected (or Conv) layers and simply average them?


texts:
I'm currently a student in a machine learning course studying for an upcoming exam. Here's a question I've been given for practice:
You have a very large dataset of employees and you'd like to investigate the drivers of employee salaries. Each instance of data contains the following information:

The occupation of the employee (e.g. data scientist or sales representative). Assume that there are thousands of distinct occupations.

Tenure (number of years the employee has been working at the company for)

Company size (number of employees at the company)

Years of experience (Number of years the employee has been in the workforce)

Title (assume there are 6 levels in every company; analyst, associate, manager, etc).

Now you want to build a linear model to incorporate these features to predict salary. The question asks me which of the following are good steps to take to build such a model:

Since occupations are categorial, create dummy variables for each occupation and use a one-hot encoding.

Convert title into an ordinal variable by assigning more senior titles larger values (i.e. map analyst to 1, associate to 2, etc).

Remove the "tenure" feature since it exhibits multicollinearity with the "years of experience" feature.

Create and name clusters of occupation types (e.g. data scientist and data engineer go together), then create dummy variables for each cluster and perform a one-hot encoding.

I think that $1$ is a bad idea since you'll increase the complexity (number of features) by a lot. The problem statement specifies that there are thousands of occupations. I think that $2$ is a good idea since it allows us to easily work with the rankings. I'm not sure about $3$. I don't have much experience with multicollinearity, but I think it sounds reasonable to remove the tenure feature, so I think it's a good idea. Finally, I don't really know about $4$, but I'd guess that it's also a good idea.
Can someone please give me some insight into this problem?
Thanks


texts: Weight-initialisation and model instability

texts:
Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with differents weight initialisations, despite similar overall performance, can yield very different individual prediction.

What could be the causes of such a behavior ? What would be the solutions ?

*The hyper-parameter tuning can't really be set aside, as regularisation may help getting the same result by reducing the number of variables. However, I observe similar behavior if I use a value significantly above / below the optimal hyper-parameter value.


texts:
I am stuck on what algorithm to use. I want to train my program on a dataset where i have an input image, and an output image which is a modified version of the input. The whole context of the image is important, and each pixel counts.

After training i want to input an image, and get an output similar to the training data.


texts: Difference between Non linear regression vs Polynomial regression

texts: I have been reading a couple of articles regarding polynomial regression vs non-linear regression, but they say that both are a different concept. I mean when you say polynomial regression, it, in fact, implies that its Nonlinear right. They why there is a difference in the Data Science world regarding both the concepts?


texts: Discrete choice questionnaire construction

texts:
I'm trying to conduct my first Discrete Choice experiment. To do this I have to construct a proper questionnaire to measure consumer preferences in a best possible way.

The experiment concerns measuring Willingness to Pay for air pollution reduction in Warsaw, Poland. There are 4 attributes, respondends will be asked in the questionnaire. The attributes and their levels are below:

mortality = [0, 2500, 5000] <- lifes saved in the effect of air pollution reduction
morbidity = [0, 250 000, 500 000] <- illnesses prevented in the effect of air pollution reduction
visibility improvement (in percentage) = [0, 50, 100] <- visibility improvement in the effect of air pollution reduction
ecological tax = [13, 25, 38] <- the price people have to pay to achieve some air pollution reduction

In the questionnaire people will be asked to choose between: A-scenario, B-scenario and Status-Quo. Both scenarios contain levels of attributes, which were desribed above. Using Python, I generated all possible A,B sets. And here is where my issue begins. There are 5184 possible sets.

Is there any algorithm I can use to create a reasonable subset of all this sets? Reasonable, I mean, not to get options like:
A = [2500, 250 000, 50%, 13]
B = [2500, 250 000, 50%, 25]
So the options, which don't really check preferences, but only reading skills of the respondend.

I hope, I presented my idea rather clear. If not, here is the article, my conception is based on: https://www.sciencedirect.com/science/article/pii/S0301479706004002

I will be really grateful for all your answers. Thanks in advance.


texts: How to append new image to train with existing image classification TensorFlow model?

texts:
I have 10 classes of images. Let's say 1 class has 500 images after training the model I want to add extra 100 images to the existing class. After adding extra images should I retrain all images 500 +100 or can I train only 100 images with the existing model?

same questing with adding a new class. after adding a new class, should I retrain all class images?


texts: How to resize image along with their mask?

texts: I have original images of the size 1935x1481. I am using labelme to annotate the images. I am creating polygons on the original image. Is there a way to resize the image along with their mask? I am planning to use TFOD mask-rcnn and I know it will resize the image but what happens to the mask?


texts: Text generation using Tensor Factorization

texts:
Text generation is well studied using Markov chains or NNs, but I am not aware of any works to word sequence prediction in terms of subspace learning.

Treating phrases or sentences as temporal data such as time series, it is possible to represent word sequences as a tensor $T = WS \times W \times K$, where $WS$ is the set of word sequences present in the corpus, $W$ represents the set of segmented words, and $K$ is the maximum length of observed sequences
 For instance, for a phrase, ws = word sequence prediction, then $T(ws, ``sequence", 2) = 1$

For an incomplete tensor, where entries s.t. prediction are missing, the reconstructed tensor after decomposition can then be used to generate texts, in terms of the observed word space.

My questions are as follows:

1) Is there any works using tensor factorization or factorization machines for word sequence generation?

2) How subspace learning models differ from those generative models, such as Recurrent Neural Networks or Belief Networks? What are the downsides of using subspace methods as compared to other established methods?

2) How to establish the threshold for the length of the predicted sequence? For example, can one look at the $WS_r \times K_r$ space, and use cross-validation to find the threshold for each word sequences?

Any pointers or answers to any of the above questions is highly appreciated.


texts:
I am working on this CSV file which is a collection of movie details from IMDB.
In this I have a genres column in the dataframe with all the genres of the movies seperated by a pipe (|)
What I need is to extract the first two genres from the genres column and store them in two new columns: genre_1 and genre_2.
And for the columns where there is only 1 genre, extract the single genre into both the columns, i.e. for such movies the genre_2 will be the same as genre_1.

I am sharing the screen shots of the code and results that I have got.

Now, I can create a new data frame with the genres created and can then remove the unwanted columns and can concatenate the remaining the with original data frame. But that looks pretty clumsy.
How can I crate split the column in my original data frame only and remove the unwanted expanded columns.
Any help is appreciated.


texts:
My question is rather simple what does the parameter scale_pos_weight in xgboost do? I know typically it should be $\frac{sum(negative cases)}{sum(positive cases)}$.
Does it oversample the minority class by that ratio or does it undersample the majority class by inverse of that ratio? Or something else?
Also I would like to know if during cross validation in xgbcv, does the sampling happen on the test part of the cross-validation also or only the train part is affected by scale_pos_weight? Because i've heard that sampling should never be applied to test as it gives over-optimistic results.


texts: When using Data augmentation is it ok to validate only with the original images?

texts:
I'm working on a multi-classification deep learning algorithm and I was getting big over-fitting:

My model is supposed to classify sunglasses on 17 different brands, but I only had around 400 images from each brand so I created a folder with data augmented x3 times, generating images with these parameters:

datagen = ImageDataGenerator(
    rotation_range=30,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest')

After doing so i got these results:

I don't know if it's correct to do the validation only using the original images or if I have to use also the augmented images for the validation, also is strange for me to get higher accuracy on the validation than the training.


texts:
A picture is worth a thousand words, so I decided to illustrate how I imagine the procedure of link prediction on network embeddings. In the figure below the LR model stands for the "Logistic regression model".

Train and test networks are composed of an equal number of positive and negative edges (training examples).

My question is: should I learn embeddings on a network defined by positive edges only or should I also consider negative edges in the network embedding process?

Any idea is appreciated.


texts:
I am running an XGBoost Regressor to predict electricity consumption (load) and further classify predicted values as peaks or not. As for dataset I started with hourly energy load data + hourly weather information.
After playing with different feature set, mean, std, slopes and rolling windows I ended up with certain model (saved as pickle file) that delivers reasonable results.
Yet after some feature turning, error went up, and prediction quality worsened. So we switched back to previous best yielding feature set, yet failed to reproduce the same model.
Old binary model, once loaded, still does good job predicting peak hours, yet exactly same feature set fails to train model to same state.
I know that boosting algorithm is stochastic in nature, yet is there a way to know the path the boosting tree within a binary model, to somehow hint model during train to repeat same choices it made before? (random state is set, numpy random seed also, hyper params were picked with grid search)
XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,
             colsample_bynode=1, colsample_bytree=0.7, gamma=0, gpu_id=-1,
             importance_type='gain', interaction_constraints='',
             learning_rate=0.1, max_delta_step=0, max_depth=5,
             min_child_weight=1, missing=nan, monotone_constraints='()',
             n_estimators=1000, n_jobs=8, num_parallel_tree=1, random_state=0,
             reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=0.7,
             tree_method='exact', validate_parameters=1, verbosity=None)


texts: How to create a classification model for multi output dataset?

texts:
I have a dataset where there are two target variables target-1 and target-2.
Both target variables are ordinal and thus I want to develop a multi-class classification model over these two target variables.
However, I am unsure as to where to start in the model creation process.
Therefore, if someone could give some pointers to this, that would be greatly appreciated.
Thanks in advance.


texts: Association between Categorical Variables and regression

texts:
We perform data analysis and build models. Say, for example, I built a regression model that has more than one predictor (multiple regression). We then check many things: normality, multicollinearity, etc. Specifically, we check for multicollinearity, for a numeric/continuous variable, VIF (Variance Inflation Factors) etc. If we find that there is multicollinearity, we then drop one of the highly correlated features.
My question is: what can be done with categorical variables? I mean if two categorical variables are correlated/associated does it mean I have to drop it? I am not clear how to handle categorical variables like we handle correlations between continuous variables.
What do we mean by two factor variables are correlated or dependent or independent? What if there is a collinearity? How do you identify that collinearity? How do you deal with it?


texts:
I got two classes namely positive and negative with 1500 samples on each a total of 3k. A sample sequence is like:
Period = 1min   Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 5min   Pattern = S1>M1>T1>B2>M2>S2>S3>T3>M3>B3
Period = 10min  Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 15min  Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 20min  Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 30min  Pattern = M1>S1>T1>B2>M2>S2>S3>T3>M3>B3
Period = 60min  Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3
Period = 120min Pattern = S1>M1>T1>B2>S2>M2>S3>T3>M3>B3

So for each sample, we have a total of 8 sequences and within each sequence there are 10 elements which are used only once. I assigned a number for each of the elements from 0 to 9. Later, stored positive and negative samples into numpy arrays with shapes of (1500,8,10). By using LSTM's, I was able to achieve around 96% accuracy, which I'm glad and sufficient for the task.
For now, my task is to create artificial samples and make the LSTM's classify it. I should mention that there are some rules for sequence generation like M1>T1>M2>T3>M3. By applying all of these rules as permutation constraints, I was able to get down to a 12960 sequences that could be used in any of the periods. This makes a total of 12960^8~= Multiple orders of Petabytes of data. Not feasible to store all samples. So I gave up generating each sample even by this means.
What I'm looking for now is a way to generate artificial data, by means of some GAN style LSTM or whatever I don't know. Which will take (1500,8,10) arrays of positive and negative samples and spit out a positive or negative candidate with a label. Thanks.


texts:
Let's assume that I have a classification problem and all my features are categorical data.

I have missing data (and I do not want to do any imputation).

Also, I know that I will have some unseen data (at my test data) at some of my features.

My question is the following:

Should I encode the missing and unseen data (of the test set) into the same class or to different ones?

Which is the most common practice and why?


texts: Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

texts:
I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization or normalization should be done after splitting the data into train and test subsets, but I am not still sure about the imputation process. For the classification task, I am planning to use Random Forest, Logistic Regression, and XGBOOST (which are not distance-based).

Could someone please explain which should come first? Split > imputation or imputation>split? In case that split>imputation is correct, should I follow imputation>standardization or standardization>imputation?


texts: Best way to build gensim WdmSimilarity for document data

texts:
I'm building an application that searches for queries in OCR data. My documents are numerous and have many pages.
I'm using Wmd Similarity to query my data with gensim
instance = WmdSimilarity(text_data_preproc, model, num_best=10)

Given that I'm working with a large dataset of OCR data from documents, what should be fed into this function to get better results among the following options?

The text each page only (one page at time)
The text of the full document
The text of all documents togheter

I'm using GloVe as my model.


texts: Why to adjust class weights instead of simply finding the best threshold?

texts: In a binary supervised classification where classes 1 and 0 have different number of samples in training, it’s very common to find tutorials about adjusting class weights, over and under sampling for imbalanced data sets. In a situation where there are enough samples for both classes (e.g. not an anomaly detection), why would one adjust class weights or balance the training data if in the end you’ll have to adjust a threshold anyway?


texts:
What are some good websites which  provide quiz questions on supervised learning and machine learning in general?
I have a quiz coming up and I would like to be prepared for it.


texts: How to detect blocks of texts in document images

texts:
I am planning to detect texts from document text images like below:

GOAL:

WORK DONE:
I have tried to solve this with some scene text detection algorithms like EAST Text detector and PixelLink. But it only provides result in such a way it detects each and every word individually as below, which is obvious:

What method can help me detect blocks of texts as mentioned under GOAL.

EDIT :

I don't want extract all texts via OCR. What I want instead is to detect texts based on their visual positional arrangement. See in the image, texts positioned together are detected as blocks. And my result should contain all the bounding box co-ordinates of all the detected text blocks.


texts: Elimination of features based on high covariance without affecting performance?

texts:
I ran into a question where the answer ran me into a big doubt.
Suppose we have a dataset $A=${$x1,x2,y$} in which $x1$ and $x2$ are our features and $y$ is the label.
Also, suppose that the covariance matrix between these three random variables are as follows:
$$\begin{array}{|c|c|c|} \hline
    &x1&x2&y\\  \hline
x1&a&d&e\\
   \hline
x2&d&b&f\\\hline
y&e&f&c\\\hline
\end{array}$$
When $|d| \gg 0$, someone may ignore either of the two features, without losing any performance.

The claim was marked as a true question. Is there any idea why the
mentioned proposition is True and performance was not affected?


texts: Tensorflow/Keras, How to convert tf.feature_column into input tensors?

texts:
I have the following code to average embeddings for list of item-ids.
(Embedding is trained on review_meta_id_input, and used as look up for pirors_input and for getting average embedding)

 review_meta_id_input = tf.keras.layers.Input(shape=(1,), dtype='int32', name='review_meta_id')
 priors_input = tf.keras.layers.Input(shape=(None,), dtype='int32', name='priors') # array of ids
 item_embedding_layer = tf.keras.layers.Embedding(
     input_dim=100,      # max number
     output_dim=self.item_embedding_size,
     name='item')

 review_meta_id_embedding = item_embedding_layer(review_meta_id_input)
 selected = tf.nn.embedding_lookup(review_meta_id_embedding, priors_input)
 non_zero_count =  tf.cast(tf.math.count_nonzero(priors_input, axis=1), tf.float32)
 embedding_sum = tf.reduce_sum(selected, axis=1)
 item_average = tf.math.divide(embedding_sum, non_zero_count)

I also have some feature columns such as..
(I just thought feature_column looked cool, but not many documents to look for..)

  kid_youngest_month = feature_column.numeric_column("kid_youngest_month")
     kid_age_youngest_buckets = feature_column.bucketized_column(kid_youngest_month, boundaries=[12, 24, 36, 72, 96])

I'd like to define [review_meta_id_iput, priors_input, (tensors from feature_columns)] as an input to keras Model.

something like:

 inputs = [review_meta_id_input, priors_input] + feature_layer
 model = tf.keras.models.Model(inputs=inputs, outputs=o)

In order to get tensors from feature columns, the closest lead I have now is

fc_to_tensor = {fc: input_layer(features, [fc]) for fc in feature_columns}

from https://github.com/tensorflow/tensorflow/issues/17170

However I'm not sure what the features are in the code.
There's no clear example on https://www.tensorflow.org/api_docs/python/tf/feature_column/input_layer either.

How should I construct the features variable for fc_to_tensor ?

Or is there a way to use keras.layers.Input and feature_column at the same time?

Or is there an alternative than tf.feature_column to do the bucketing as above? then I'll just drop the feature_column for now;


texts: Where can Batch Normalization be used? CNNs or everywhere?

texts: Should BatchNormalization be used only in CNNs or can they be used in Fully Connected Networks, Recurrent networks as well?


texts:
>>> from sklearn.ensemble import RandomForestClassifier

>>> clf = RandomForestClassifier(n_estimators=10, random_state=1)
>>> Y=[0,1]

>>> X = [[3,2,1,0], [7,6,5,4]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

>>> X = [[0, 1, 2,3], [4,5,6,7]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

>>> X = [[3,2,1,0], [7,6,5,4]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

>>> X = [[3,1,2,0], [7,5,6,4]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

Assume the features have names.
When I shuffle/change the listed order of features specified in the training data set, the importance for each feature changes.
That means the resulted random forest classifier also changes.
Note that I have rule out the effect of randomness, by fixing the random seed.

Why does the listed order of features specified in the data set matter to the random forest classifier, given that the random seed is fixed? Thanks.


texts: How to extract information from plot images?

texts:
Are there any free tools or libraries that can understand a plot image file automatically? Things like type detection (line, bar, scatter), as well as label and axis scaling detection.

In the tools I have found so far, one has to do manual configuration. I'm asking if it is possible in our AI driven world to let a machine do this.

If there are none, I will start a project for myself as it seems like a challenging but instructive AI task. Is this effort doomed from start? Are there any helpful libraries in C++ or Java and theoretical algorithms one could consider?

Thanks,
Tomas


texts: Normal distribution instead of Logistic distribution for classification

texts:
Logistic regression, based on the logistic function $\sigma(x) =
\frac{1}{1 + \exp(-x)}$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic distribution where the p.m.f is

  $f(x) = \frac{\exp(-x)}{[1 + \exp(-x)]^2}$

and the c.d.f is

  $F(x) = \sigma(x) = \frac{1}{1 + \exp(-x)}$

The hypothesis to test is

  $H_0: x \text{ isn't positive} \hspace{2.0cm} H_1: x \text{ is positive}$

The test statistic is $F(x)$. We reject $H_0$ if $F(x) \geq \alpha$ where $\alpha$ is the level of significance (in terms of hypothesis testing) or classification threshold (in terms of classification problem)

My question is that why they don't come up with the Standard normal distribution, which truly reflects the "distribution of nature", instead of Logistic distribution ?


texts: Feature Interactions vs Feature Importances

texts: Can somebody explain in-detailed differences between Feature Interactions and Feature Importances? And their applications?


texts: How to integrate out hyperparameters of Gaussian process for Bayesian optimization?

texts:
I read this paper (https://arxiv.org/pdf/1206.2944.pdf) discussing about practical issues of Bayesian optimization and they mentioned that integrating out hyperparameters of Gaussian process using fully-Bayesian treatment would be sensible and give more stable results.  The way they do that is to use following equation (Eq. 6 in page 4 of the paper):

$$\hat{a}(x;\{x_n,y_n\})=\int a(x;\{x_n,y_n,\theta\})p(\theta|\{x_n,y_n\})d\theta$$

Does anyone know how this integral is implemented?  I know that $a(x;\{x_n,y_n,\theta\})$ can be computed using Gaussian process regression, and in theory $p(\theta|\{x_n,y_n\})$ can be expressed as

$$p(\theta|\{x_n,y_n\})=\frac{p(\{x_n,y_n\}|\theta)p(\theta)}{p(\{x_n,y_n\})}$$

and I expect $p(\{x_n,y_n\}|\theta)$ should be tractable (it is just a multi-variate Gaussian), but what is prior for $p(\theta)$ and how to compute $p(\{x_n,y_n\})$ and more importantly how to combine them to implement the complete integral?

Thanks!


texts: Library/package/tool for geographical data visualizations

texts:
I am looking for a good free existing tool which visualizes geographical data (let's say in the form of coordinates) by plotting them on a map.

It can be a library (see this question on StackOverflow, which suggests a Python library called basemap, which is interesting but not dynamic enough, namely it does not allow for interactivity) or a complete toolkit.

Existing things I found are oriented towards realizing web pages, which are not my ultimate goal (see Exhibit or Modest Maps). I'd like something to feed with data which spits out an interactive map where you can click on places and it displays the related data.


texts:
Tried to forecast the target variable using Holt's Winter Additive model.
Its a univariate dataset with 6409 data points.
I took a subset of 100 points.
Getting best RMSE is 48220. Min value of target var in training data set is 7.674931979e-07 and max is 111848.10654880891. Is it ok to go with such high value of RMSE?
Also, AIC score is 1710 for the model achieved above.
Thanks in advance


texts: Number of internal nodes in a Decision Tree

texts: The number of internal nodes in a decision tree is not equal to the size (n) of the input feature vector? It is always less than n or the upper bound on the number of decision nodes will be n? How to draw a relation?


texts:
I've recently conducted a k-prototypes R routine on some mixed data.
In particular, the data is health data concerning a certain public health intervention, with categorical variables for health scores and numerical demographic data such as age
The utility scores were measured at different points in time, with one sample at week 1, and one sample at week 10.
At the moment, I have only conducted clustering analysis on one of these samples. However, I wondered if there is a recognised routine for making inference on the clusters between week 1, and week 10.
All feedback would be appreciated. I recognise that it is not quite longitudinal clustering, but more comparing two different clustering states.
Thank you


texts: TensorFlow - Resume training in middle of an epoch?

texts:
I have a general question regarding TensorFlow's saver function.

The saver class allows us to save a session via:

saver.save(sess, "checkpoints.ckpt")

And allows us to restore the session:

saver.restore(sess, tf.train.latest_checkpoint("checkpoints.ckpt"))

Inside the TensorFlow documentation, there is an example code (with an added epoch loop, and restore):

# Create a saver.
saver = tf.train.Saver(...variables...)
# Launch the graph and train, saving the model every 1,000 steps.
sess = tf.Session()
saver.restore(sess, tf.train.latest_checkpoint("checkpoints.ckpt"))
for epoch in xrange(25):
    for step in xrange(1000000):
        sess.run(..training_op..)
        if step % 1000 == 0:
            # Append the step number to the checkpoint name:
            saver.save(sess, 'my-model', global_step=step)

The problem is that if we stopped the training loop at epoch=15, and execute again, then if we would start at epoch=0 again, but the model is trained up to epoch=15.

Is there a way to resume from epoch=15?


texts:
I would like to ask for help regarding my model.

I have a dataset of preprocessed images and I performed a binary classification with SVM on Python.
I tuned the value of the c parameter from 0.001 to 1 with 10-fold CV and I obtained the following results.

My questions are:

Does this mean that the optimal value of C is 0.001?
The rest means that there is overfitting?

The first value is not 0.000 but 0.001

Thank you.


texts:
I have a dataset of diabetes patients and I am trying to predict the next blood glucose level. I have attached an image below and I have about 1600 records in that csv file containing data of 10 patients. Each patient is uniquely identified by the Id column and the Glucose_t-1 means the glucose value in which the patient had before the current reading(Glucose_t) likewise this apply to Glucose_t-2 and Glucose_t-3. And same applies to the Insulin_t-1,Insulin_t-2. The event column is the Glycemic event in which the current reading of the blood glucose value falls to. For example,

blood glucose value <= 70 then 0,
70< blood glucose value <=180 then 1,
blood glucose value > 180 then 2.

I have applied different regression algorithms like Logisitic Regression, Random Forest Regression and so on but I was unable to predict the Glucose_t value accurately. The accuracy comes to 0.008.. which is very depressing :(. Please any help to improve the accuracy of will be greatly appreciated. Thanks.


texts: Catastrophic forgetting in linear semi-gradient RL agent?

texts:
I've been working through the Sutton + Barto RL text, implementing a number of the algos + running them in the OpenAI gym. One phenomenon that I seem to come across quite regularly is that agents who, at certain points during their training appear to be making good progress towards learning a plausible state-value/state-action functions, "catastrophically forget" the insights they glean and subsequently never recover.

To make this more concrete, here's the (smoothed) reward history of my implementation of a semi-gradient expected SARSA agent with linear function approximation and binary features running on the MountainCar environment.

Problem Details

The problem has a bounded, continuous state space and a discrete, 3-action action space.
The learner receives a reward of $-1$ at each timestep. The episode ends when the car makes its way all the way up the hill OR it reaches 200 timesteps without success.
Prior to learning, I tile-coded the state space using an 8x8 grid with 8 overlapping tilings, generated using Sutton's own tiles.py script.

Learner Details

The learner is an implementation of the semi-gradient SARSA algorithm with linear function approximation for the Q value.
The agent begins with all Q weights initialized to 0.
The learning rate for the agent was set at $\frac{1}{10} \times (\text{# tiles})^{-1}$, where $\text{# tiles}$ in this case was $8 \times 8 \times 8 = 512$
During learning, the agent selected its actions using an $\epsilon$-soft policy, where $\epsilon$ is set to 0.10
The agent's temporal discount factor is set to $\gamma = 0.95$ (though I realize it could have been 1 given the episodic nature of the task)

My Question

One possibility I want to rule out is that my implementation of the learning agent is incorrect. To that end I am curious to know whether others experience this kind of "forgetting" behavior (even on simple problems like this), and if so, how it might be reduced.


texts: How can we perform STS(Semantic Textual Similarity) on UnSupervised dataset using Deep Learning?

texts:
How to implement STS(Semantic Textual Similarity) on unlabelled dataset. Dataset column contains Unique_id, text1(contains paragraph), text2(contains paragraph).

Ex: Column representation:  Unique_id   |    Text1    |     Text2

Unique_id 0

Text1 public show for reynolds suspension of his coaching licence. portrait sir joshua   reynolds portrait of omai will get a public airing following fears it would stay hidden because   of an export wrangle.

Text2 then requested to do so by  spain   s anti-violence commission. the fine was far less than the expected amount of about Â£22 000 or even the suspension of his coaching licence.

Unique_id 1

Text1 groening. gervais has already begun writing the   script but is keeping its subject matter a closely guarded secret. he will   also write a part for himself in the episode. i ve got the rough idea but this is the most intimidating project of   my career.

Text2 philadelphia said they found   insufficient evidence to support the woman s allegations regarding an alleged   incident in january 2004. the woman reported the allegations to canadian   authorities last month. cosby s lawyer walter m phillips jr said the   comedian was pleased with the decision.

In the above problem, I've to compare two paragraphs texts i.e. Text1 & Text2 and then I've to compare semantic similarity between two texts. If they are semantically similar then it will print '1' if not then '0'

Any refrence implementation link or any suggestions !

Thanks in advance


texts: Building a predictive algorithm in python using an inspection sensor data of an assembly line (manufacturing)

texts:
We have a manufacturing assembly line. At the end of it, there is a sensor that can measure the final product. There is a set engineering tolerance so the product will be inside the limit (OK) or outside (NOT OK).

What I want to do is using all the previous measured data, try to predict if the next products are going to be outside this tolerance limit.

Is there a model that can be adapted to achieve this?

If it is possible, can it tell me when? (Based on a tendency maybe?)

I am pretty new to all this so thanks for the help!


texts:
I am somewhat confident this is the correct approach but I need a little sanity check before setting out on this.

I want to record a numerical value for a category over time. There will be many categories and may appear as follows:

1  cat_1
4  cat_2
10 cat_3
4  cat_4
10 cat_5
8  cat_6
99 cat_7
4  cat_8

Say, the count (on the left) on the left will be different each time I poll for the data.

I will want to run analysis on this later on.

My plan is to store the data each time I poll for it in a JSON encoded object in S3 (along with the timestamp). This means it will be in raw form.

Later on I will want to query that information to see the values per category over time.

Will this be sufficient and easy to analyse later?


texts:
What's the difference of calculating features directly vs having a CNN interpret it?

If I train a CNN, then it will predict, whether a given input is of the recognized type.

However, what if my data is such that I can interpret, whether it's of a certain type by doing direct measurements on the data? That I don't need to "compare" it to something. Is CNN pointless in this case?

Or perhaps it's that CNN becomes viable when there are so many measurable parameters that trying to infer all of them by manual algorithm becomes laborious?


texts: Predict the probability that a customer buy today

texts:
My company sells a single product that is a commodity. If you buy it, I can be sure that you will buy it again in the near future from me or from my competitors. The demand is affected by the weather. If we have high temperatures my product sales decrease, and vice-versa.

I have this data per sale:

Customer
Average temperature in the day of the sale
Amount of product bought in a transaction
Size of the customer warehouse
Price at the moment of the purchase
Timestamp of the sale

The problem:

I want, based on historic time series data, to get the Probability that a customer requires my product today. I would like to have a %. Customer X have a 70% of probability of require your product today based on the historic. With that I could get a list of all the customers that today have 80% or more probability of buying and take action with that.

The complexities I see:

I have a lot of data because I have a lot of customers but for a single customer I have an average of only 5 purchases.
If the customer is not loyal and gets the product from my competitors I have holes that make sales separated by a irregular time spans.

What would be a good algorithm to tackle this problem?

I thought about regressions using time series but I won't get a probability. Maybe transform the timestamp of re-purchase per customer in my training set and try to cluster based on days between purchases.


texts: If a time series has random time events, how to detect patterns?

texts:
My app receives messages with a random number of bits at a random time. But two weeks ago I started to notice some almost regular patterns on the metrics of my app. I suspect they are some bots sending artificially generated data to my app. Specifically, I'm looking for sequential subsets of messages in a time series where messages has almost the same number of bits.

I read about some methods but they use data where time is not a random variable. I appreciate any help you can provide, including books, web pages, tutorials (in Python if possible), etc.


texts: How to cluster histograms or density distributions?

texts:
I have no idea where to start when it come to cluster distribution and finding out similar the similar one.
Is there a package in R that does the job?


texts: Appropriate naive benchmark for class recall in binary classification for unbalanced dataset

texts:
I have an unbalanced dataset with 3969 rows of customer data. The labels are whether or not they subscribed for a loan (yes or no). There are 3618 no cases (91.2%) and 351 yes cases (8.8%). I am more interested in predicting yes than no.
My question concerns finding an appropriate benchmark or naive prediction against which to compare my models. (I am mainly working with decision trees, but I don't think that is relevant to my question.) I understand that because the data is heavily unbalanced, the naive prediction would be to assign all cases as no, which would give a prediction accuracy of 91.2%; thus, any model I develop must have greater than 91.2% accuracy. Here are the results of my initial analysis (of a 30% validation set after model development on a 70% training set of the data):

Here, the overall accuracy of 89.75% is less than 91.2%, so I consider this a bad model overall. However, I am primarily interested in the predictions of the yes cases. For this initial model, the class recall for yes is 11.43%. That is, the original dataset had 8.8% yes data, and this model can predict yes 11.43% of the time. Is this good or bad? Is it even relevant to my intention of identifying yes cases? Or does "good or bad" depend entirely on how much weight or cost I assign to finding yes versus missing no?
As a second step, I know that my poor accuracy results are because many algorithms struggle with unbalanced data. So, I undersampled the training set (not the entire dataset) to obtain a 50-50 proportion across yes and no cases and then built a new model on that undersampled data. Here are the results of my analysis of this balanced dataset (the 30% validation set after model development on the undersampled training set:

Since this data is balanced, I understand that the naive benchmark for accuracy is 50%. So, the accuracy of 70.5% is a good overall result. The class recall for yes of 54.29% is also a good result, and so this second model is undoubtedly superior to the first. However, my second question again concerns the benchmark for comparison of the class recall for yes.  Am I supposed to compare that against the initial dataset distribution of 8.8% yes, or is there some other naive benchmark for this  value? I am fairly certain that I should not be comparing the class recall with 50% because the validation dataset was not balanced; only the training dataset was.
So, in summary, what is the appropriate naive benchmark against which to compare class recall results in binary classification for an unbalanced dataset?


texts:
I am working with a relatively small (~1000 samples) dataset that has some very messy text data (i.e lots of things missing, no real structure, etc.). I am trying to preprocess it and just going through each data sample and preprocessing it by hand is a viable option. I think that writing an algorithm to automate the process would actually end up being more tedious and take longer than just getting my hands dirty with the data.
So, my question is: is it common for data scientists/academics/professionals to preprocess a relatively small dataset by hand if it is the best and quickest option, or is anything done without automation looked down upon?


texts:
So I am starting a project that has some code in it that has to do with machine learning (for example recommender system).
At the moment I am thinking that if I want to deploy this application, should I look into the VM´s that are dedicated to Machine Learning applications or can a regular VM run these kinds of operations as well? What are the differences between them?

Thanks for answers!


texts:
Queries regarding feature importance for categorical features:

Context:
I have almost 185 categorical features and these categorical features have either 2 or 3 or 8 or 1 or sometimes 4 categories, null's also. I need to select top 60 features for my model.  I also understand that features needs to be selected based on business importance OR feature importance by random forest / decision tree.

Queries:

I have plotted histograms for each feature (value count vs category) to analyse. What is the approach to select whether feature is important?
What is the standard practice followed across data science industry to get the feature importance from categorical data?
Is there basic and elegant way to select the top important features?
How do I engineer these categorical features?


texts: Build an Autocomplete model for document titles

texts:
I want to build an autocomplete model using RNN where input is article names (documents title).

  X: ['Billing', 'Loan status', 'Filling loan application', 'Contact Info', ...]

The article name can be 1-5 word long and there is an even distribution of article names among word count i.e. no. of 1 word articles names is almost same to no. of 2 word article name.

I want my model to start predicting article names based on first 3 typed characters (as I do not want to miss one word article names) and then make predict call based on each next typed characters.

I am not able to decide what should be the target output (y) here while training the model? Should I right shift my input sequence by 3 characters and train the model like below?

PS: Please mind my drawing.


texts: One hot encoding error "sort.list(y)..."

texts:
I'm trying to do one hot encoding on a data set containing 4 categorical features in R. Unique levels per feature:  400, 60, 6, 5, respectively. I get the following error during the first call to the ohe function (acm.disjonctif):

Error in sort.list(y) : 'x' must be atomic for 'sort.list'
Have you called 'sort' on a list?

I can't do sorting on a per column basis because it will remove the data relationship between columns.  What am I missing in understanding?  I searched on the web but couldn't find anything helpful.

Here is the code I'm running in RStudio:

ohe_feats = c('RSN_CD', 'RSN_CAT', 'ELIG_SYS_CD', 'PROD_CD')
for (f in ohe_feats) {
    df_all_dummy = acm.disjonctif(as.factor(nodups[f]))   # package = ade4
    nodups[f] = NULL
    nodups = cbind(nodups, df_all_dummy)
}


texts: how can i plot probability distribution of my classes in the way below?

texts:
All,
I would like to plot the following:

I have a binary classification problem where I am using xgboost as my 'model' below:
y_pred = model.predict_proba(x_test)[:, 1]

probabilities = pd.DataFrame({
            'prediction': y_pred,
            'output': y_true
        })

        colors = ['g', 'r']
        for output in [0, 1]:
            probabilities[probabilities['output'] == output]['prediction'] \
                .plot(kind='kde', ax=ax2, legend=True, color=colors[output])
        ax2.set_xlim(prob_xlims)
        ax2.set_xlabel('Predicted probability')
        ax2.set_title('Predicted probabilities and true labels (color)')

        # Legend
        handles, labels = ax2.get_legend_handles_labels()
        ax2.legend(handles, ['Class 0', 'Class 1'])

Using the above how can I get a plot similar to the above? Correct me if wrong as well, but predict_proba returns $\text{probability(class x | data sample)}$ where data sample is a single row of data..?


texts: What Are Stats Metrics To Analyze How Well One Column Predicts Another Column

texts:
I have the following table with three columns:

Type    Category    Actual
A       1           47
A       1           90
A       1           74
A       2           64
A       2           25
A       3           55
A       3           55
A       3           19
B       6           80
B       6           28
B       2           60
B       2           49
B       2           7

Next, I'm given a static table that provides predictions by Type and Category.  I will be using the "Prediction" column values as the actual predictions.

Type    Category    Prediction
A       1           83
A       2           48
B       6           54
B       2           57

So once I combine the two tables by Type and Category, I have the following table:

Type    Category    Actual  Prediction
A       1           47      83
A       1           90      83
A       1           74      83
A       2           64      48
A       2           25      48
B       6           80      54
B       6           28      54
B       2           60      57
B       2           49      57
B       2           7       57

I want to evaluate my data by distinct Type and Category combinations.  So my output table could be something like this:

Type    Category    Avg_Actual  Avg_Prediction
A       1           value       value
A       2           value       value
B       6           value       value
B       2           value       value

My question is: What are different metrics I can use to evaluate how predictive the "Prediction" column is against the "Actual" values given my data set.

Please be specific with your metrics and the more you can provide (with details on how to derive them) the better.  So far, the only one I came up with was calculating RMSE.

I would like all the metrics at a distinct Type-Category level.  Additionally, is there a way to incorporate confidence intervals into this analysis?

Any help is greatly appreciated.


texts: Predicting sequence element based on the previous M and the following N elements

texts:
I have an array of sequences of equal length, each sequence contains 300 numbers (M=300). Each element in a sequence is a number from 1 to 9:
13571398...2455 # 300 numbers
33344467...1143 # 300 numbers
...
...
...
66118859...2121 # 300 numbers

My task is to build a model that predicts an element (number) at sequence positions from 180 to 190 based on the first 179 elements and the last 110 elements in a sequence.
In other words, given elements at positions from 0 to 179 and from 191 to 299 predict elements in a sequence at positions from 180 to 190.
I am thinking about the following steps to solve this task with Keras BiLSTM model:

Split all sequences into train / validation / test sets
Train BiLSTM on a train set to predict next number anywhere in a sequence
In test and validation sets randomly replace K elements at positions from 180 to 190 with 0 (a number that does not exists in original sequences).
Use pre-trained BiLSTM to predict true values of '0' elements in validation and test sets

Please help with the following questions:

How should I represent data and classes for BiLSTM in this case? It looks like my data and classes are the one and the same thing. Both 1...9 numbers are data and corresponding classes to BiLSTM.
What data structures, encodings in this case should I create to train and predict with Keras BiLSTM?
How to evaluate quality of this model on a train and test sets ?

Any other ideas of using other models, in particular Transformers (PyTorch, Tesnsorflow) are very welcome, thanks!


texts: Gbm prediction distribution different to training data

texts: I’m doing a regression on a dataset using lightgbm. For the training data the response variable has a non normal distribution which is multimodal. However, the predictions out-of-fold have are normally distributed and have a single mode. Is there an assumption of normality made by light gbm ? Is there a way to make predictions which better match the required distribution?


texts: Is there any point in learning Hadoop in 2018?

texts:
I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing framework.

Hadoop had created a lot of fuss in around 2013-2014. As per my limited knowledge on the subject, Apache Spark has improved upon Hadoop multi-fold in all aspects.

So, apart from the obvious case when you need to maintain legacy Hadoop applications, is there any reason as of today to prefer Hadoop over Apache Spark?


texts: Reinforcement learning with sparse acting agent

texts: I'm working on a problem where the optimal policy involves the agent "doing nothing" most of the time, and "doing something" during rare critical moments. Is there any literature or best practices when it comes to solving problems like these where actions are quite rare? I tried changing the random action choice to choose inaction over action with a certain probability to bias the agent to do nothing, but that didn't really affect learning. I think part of the issue is that the agent only receives reward feedback when he acts, so this makes learning the optimal sparse action policy quite difficult. I've tried to avoid reward hacking (agent acting frequently and gathering many small rewards) by weighing the rewards in such a way that rewards given by infrequent but correctly timed action far out-weight rewards given by frequent, incorrectly timed action. But I'm not sure if that's a good solution, or even a solution at all. Currently using a dueling DQN network structure, but open to trying anything (policy gradients?). Any help would be much appreciated.


texts: How to choose between classification Vs regression approach?

texts: I need to predict the profitability of the products of a retailer. I can either predict the absolute value of the profit the products will make (continuous outcome) or predict whether the products will make a profit or not (categorical outcome). Is there any advantage of approaching this as a classification problem rather than a regression problem or vice versa? Why?


texts: How Sklearn-crfsuit interpret text features

texts: As we see here, to build an NER model we can pass text features (parts of the word, pos tag, structure of the word etc.) to Sklearn-CRF. I was wondering how does this package convert the text features to numerical features? What strategy do they use - any specific type of embedding?


texts: Visulazing a specific value of a column with the corresponding values

texts:
I have recently started using data studio to make visualizations before diving into analysis and data prep!

I have a column named NAME OF STORE that good different names and other columns like Date, Number of fans, Number of Visitors, etc. For each store, at each date, I got different values for the other column.

How, can I choose in data studio what store I wanna make visualization for?


texts: Validation accuracy does not increase for binary classification using GNN

texts:
I am trying to perform graph classification with GINConv model of GNN. I have tried everything from varying dropouts to weight decay (for L2 regularization), learning rate (1e-6 to 1e-3), batch normalization to varying batch sets. It still doesn't show any change. My dataset is a set of 8000 graphs (with varying vertices and edges) that contain label 0 or 1. There are 4000 graphs with label 0 and 4000 with label 1. I am using Adam optimizer.There are 6 layers in my model. Here is my code :
def __init__(self):#,init_eps=np.random.uniform(1.5, 99.5),learn_eps=True):
    super(Net, self).__init__()

    self.dropout = 0.3
    self.first_h = []
    self.nns = []
    self.convs = []
    self.linears = []
    self.pooling = global_mean_pool

    #layer 0
    self.first_h = Sequential(Linear(60, 50), BatchNorm1d(50), ReLU(),
            Linear(50, 50), BatchNorm1d(50), ReLU())
    self.linears.append(Linear(50, 2))
    #rest
    i = 50
    while(i >= 20):
        j = i - 10
        self.nns.append(Sequential(Linear(i, j), BatchNorm1d(j), ReLU(),
            Linear(j, j), BatchNorm1d(j), ReLU()))
        self.convs.append(GINConv(self.nns[-1], train_eps=9))
        self.linears.append(Linear(j, 2))
        i = j

    self.nns.append(Sequential(Linear(10, 2), BatchNorm1d(2), ReLU(),
        Linear(2, 2), BatchNorm1d(2), ReLU()))
    self.convs.append(GINConv(self.nns[-1], train_eps=9))
    self.linears.append(Linear(2, 2))

    self.nns = torch.nn.ModuleList(self.nns)
    self.convs = torch.nn.ModuleList(self.convs)
    self.linears = torch.nn.ModuleList(self.linears)

def forward(self, data, flag):
    out = 0
    x, edge_index, batch = data.x, data.edge_index, data.batch
    for layer in range(6):
        if layer == 0:
            x = self.first_h(x)
            out += F.dropout(self.pooling(self.linears[layer](x), batch), p=self.dropout)
        else:
            x = self.convs[layer-1](x, edge_index)
            out += F.dropout(self.linears[layer](self.pooling(x, batch)), p=self.dropout, training=self.training)
    return out

Optimizer and train/valid/test splits :
train_loader = DataLoader(data_list[0:8000], batch_size=8000, shuffle = True)
valid_loader = DataLoader(data_list[8000:9000], batch_size=1000, shuffle = True)
test_loader = DataLoader(data_list[9000:10000], batch_size=1000, shuffle = True)

Train code :
for epoch in range(1000):
    if(validVal < 0.4 and trainVal < 0.3):
        break
    trainVal = 0
    model.train()
    for batch,label in train_loader:
        optimizer.zero_grad()
        batch = batch.to(torch.device(device))
        out = model(batch,1)
        label = label.to(torch.device(device))
        loss = F.cross_entropy(out, label)
        trainVal += loss.cpu().detach().numpy()
        loss.backward()
        optimizer.step()
    model.eval()
    for batch,label in valid_loader:
        batch = batch.to(torch.device(device))
        out = model(batch,0)
        label = label.to(torch.device(device))
        loss = F.cross_entropy(out, label)
        validVal = loss.cpu().detach().numpy()
        if(minValid > loss.cpu().detach().numpy()):
            minValid = loss.cpu().detach().numpy()

I have tested by varying number of epochs as well. I have tried almost every possible scenario to prevent overfitting, but it does not seem to work. Any suggestions how to go about it?
This is my obtained curve :


texts: Choice of replacing missing values based on the data distribution

texts:
I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has missing values so I want to replace them.

I made some general calculation to get some understand of the disruption of the data and to help me choose the value which would replace the missing values,

Assume that I have attribute A with the following:
mean = 121.68676278
std = 30.51562426
median = 117
mode =
min = 44
max = 199
[in all the calculations, I ignored the missing values]

If I were to choose between mean, median, or mode which one would be most suitable?

and there is something else which was very confusing for me, the std is very large and when I asked about it I was told that this could be normal based on the range of my data but I did not understand what that means?


texts: SVM Cost function change to improve its computational efficiency

texts:
While listening to Andrew Ng's course of Machine Learning he said that the
SVM's cost function term $\frac{\Theta^T\Theta}{2}$ is usually changed to $\frac{\Theta^TM\Theta}{2}$, where matrix $M$ depends on the kernel choice.

In his words, this allows that the computational efficiency of the optimization algorithm can be improved, even when the training set is large.

Can someone point me to some materials that support that claim? How do we choose $M$, and how can we change the cost function/its partial derivatives so that the gradient computation is feasible even when the training set is large?

Also given that we include a kernel-specific weighing matrix, I assume that different kernels then lead to different kinds of decision boundaries? Is this correct or does matrix $M$ have no influence on the decision boundary?


texts: Why n-split is not possible for a dataframe with KFold?

texts:
On running below code on python 3.7, I am getting the following response:
'DataFrame' object has no attribute 'n_splits'.
How to get rid of this?

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import sklearn.model_selection as train_test_split
from sklearn.tree import DecisionTreeClassifier
%matplotlib inline
import seaborn as sns
sns.set()
from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

train=pd.read_csv(r"C:\Users\Pranjal Pathak\Desktop\kaggle-titanic-master\Digit recognizer kaggle\train.csv")
test=pd.read_csv(r"C:\Users\Pranjal Pathak\Desktop\kaggle-titanic-master\Digit recognizer kaggle\test.csv")

train1=train.drop('label',axis=1)
test1=train['label']

Kfold=KFold(n_splits=30,shuffle=True,random_state=0)

clf=KNeighborsClassifier(n_neighbors = 20)
scoring='accuracy'

score=cross_val_score(clf,train1,test1,cv=KFold,n_jobs=1,scoring=scoring)


texts: Orange (Data Mining) : How to start using "Orange" from Python Anaconda Environment?

texts:
I have install "Orange Data Mining v3.4.1" in Anaconda Python v3 environment using commands : "conda install orange3" sucessfully.

However, I do not know how to call "Orange" as Application to start using it. There is also NO icon on the desktop as well.

Please help. Thank you very much in advance

Regards
Pearapon S.


texts:
I am working on a project that should predict Alarm based on the input data. I am trying to use a supervised learning algorithm. But I do not have the exact target value based on the input for feeding the learning algorithm. I am trying to generate the target value manually based on the input data (Current Target = previous input[frequency] +constant). But the plot of the target is not a  smooth curve. A sample is given below.

In the graph given below the yellow line plots the current alarm (manually set), pink is the generated target alarm value and red is the input (frequency).

As the pink plot isn't a smooth curve, it cannot be used as a efficient target output for learning. I have also tried normalization of the graph which gives a really undesired output.

Question

What method should be used to generate the target value (There are no fixed rule for the alarm value)?

How can the curve be smoothed desirably?

Project background

Working on Knime with a dataset of 6000 rows.


texts: What is the better model architecture and setting when using merge layers?

texts:
I am building a deep learning model with dense, dropout, and merge layers. The inputs will be N sentences' feature encoded by BERT (768 dim) and then each will go into the same dense layer as the first downstream then following by dropout. N inputs are done at this moment and since assuming maximum only 1 label will exist in every sentence but multi labels for the sentences set.  I use the merged layer at the end, hope to summarize the results from the results.

An example of input features and label as below:

Features                                             Labels
Vector 1, Vector 2, ..., Vector N (1*768 for each)   A, B
Vector 1, Vector 2, ..., Vector N (1*768 for each)   J
Vector 1, Vector 2, ..., Vector N (1*768 for each)   H, Y, Z

Currently, the model I used is like

class TestModel(keras.Model):
    def __init__(self, input_dim=(768,), max_sents=6, num_cls=1200):
        super(TestModel, self).__init__()

        self.Hidden_1 = keras.layers.Dense(256, activation='relu', name='BERT_DOWSTREAM')
        self.Hidden_2 = keras.layers.Dropout(.25)
        self.LayOut   = keras.layers.Dense(num_cls, activation='sigmoid', name='OUTPUT')
        self.Combines  = keras.layers.Maximum()

        self.inputs = [keras.layers.Input(shape=input_dim) for i in range(max_sents)]
        tmp = [self.Hidden_1(inpt) for inpt in self.inputs]
        tmp = [self.Hidden_2(inpt) for inpt in self.inputs]
        tmp = [self.LayOut(inpt)   for inpt in self.inputs]
        self.outputs = self.Combines(tmp)

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
BERT_DOWSTREAM (Dense)       (None, 256)               196864
_________________________________________________________________
dropout (Dropout)            (None, 768)               0
_________________________________________________________________
OUTPUT (Dense)               (None, 1200)              922800
_________________________________________________________________
maximum (Maximum)            (None, 1200)              0
=================================================================
Total params: 1,119,664
Trainable params: 1,119,664
Non-trainable params: 0
_________________________________________________________________

My questions are

Is this good to use sigmoid in self.LayOut or softmax?
Is this the correct way to set the merge layer (maximum) there? or any better way to do it?
(A bit out of question) Any better way to solve this kind of problem?

Many thanks for any ideas, suggestion or relative research sources posted.


texts:
There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he compares his own definition of dataframe-tidy to other possible ways of achieving tidiness and mentions array-tidy in the following paragraph (but does not give any further explanations or examples):

  Fortunately, because there are many efficient tools for working with high-dimensional arrays, even sparse ones, such an array-tidy
  format is not only likely to be quite compact and efficient, it should
  also be able to easily connect with the mathematical basis of
  statistics. This, in fact, is the approach taken by the pandas Python
  data analysis library (McKinney 2010).

What does array-tidy data mean and why does he imply that this is somehow the default for the pandas library in Python?


texts: Confusion matrix : Train data accuracy lower than test data accuracy and 95%CI meaning

texts:
Let's say that I have training data and test data. I trained the model with the training data and then constructed a confusion matrix of the predictions of both. I am getting that the prediction of training data is lower than the test data.
This my training data prediction:

This is the test prediction confusion matrix (bottom screen). What does it mean? Is my model bad because my training accuracy is lower than my test? What about my 95% CI which is lower in the test data than in the train data prediction? I do not really understand everything even after searching on Google. I am using linear SVM method.
If anyone knows something I would be happy to listen to it.


texts: Geolocation Based Anomaly Detection in IPs Using Isolation Forest

texts:
I'm trying to detect anomalies based on geolocation from IP addresses on a server access log file. I have created two features country and geo_velocity, using the IP address and the timestamp of each request. However, since all the requests are from stationary clients and all the clients are from one country in the log file I have, my dataset ends up looking something like this.
| Country     | geo_velocity|
| ----------- | ----------- |
| USA         | 0           |
| USA         | 0           |
| USA         | 0           |

Basically, If I plot the whole dataset in a scatterplot, it would condense onto a single point. Therefore, literally, every other value to these features should be an anomaly for this dataset.
I used Isolation Forest with GridSearchCV to tune hyperparameters. And for the scoring parameter in GridSearchCV, I used a custom scoring function as shown in the code below. The problem is, the model classifies everything as inliers even though they should be clearly outliers.
from sklearn.ensemble import IsolationForest
import numpy as np
import pandas as pd
from sklearn.model_selection import GridSearchCV

# this is similar to my dataset after label encoding the "country"
data = np.zeros((5000,2),dtype=int)
dataset = pd.DataFrame(data, columns=['country','geolocation'])

params = {'n_estimators':[70,80,100], 'max_samples':['auto'],
     'contamination':[0,0.001,0.010,0.1,0.5], 'max_features':[1,2],
     'bootstrap':[True,False],'n_jobs':[-1],
     'random_state':[None,1,], 'warm_start':[True]}

def scorer_f(estimator, X):
      return np.mean(estimator.score_samples(X))

isolation_forest = GridSearchCV(IsolationForest(), params, scoring=scorer_f)
model = isolation_forest.fit(dataset)

best_model = model.best_estimator_

predictions = best_model.predict([[0,0],[100,100]])
print(predictions)

output : [1,1]

The dataset generated in the above code is similar to the dataset I have after encoding. Even though the second point should clearly be an outlier, the model classifies it as an inlier.
What seems to be wrong with this approach? Thanks in advance!


texts:
I tried below code where I used method as 'mxnet':

classifier = train(form = Survived ~ ., data = training_set_scaled, method = 'mxnet')

From this code I got output:

Neural Network

713 samples
  8 predictor
  2 classes: '0', '1'

No pre-processing
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 713, 713, 713, 713, 713, 713, ...
Resampling results across tuning parameters:

  layer1  dropout  Accuracy   Kappa
  1       0.00     0.6220221   0.234515267
  1       0.35     0.6220221   0.234515267
  1       0.70     0.6220221   0.234515267
  3       0.00     0.3931104  -0.002267395
  3       0.35     0.3931104  -0.002267395
  3       0.70     0.3931104  -0.002267395
  5       0.00     0.3798166  -0.119003962
  5       0.35     0.3798166  -0.119003962
  5       0.70     0.3798166  -0.119003962

Tuning parameter 'layer2' was held constant at a value of 0
Tuning

Tuning parameter 'momentum' was held constant at a value of 0.9

Tuning parameter 'activation' was held constant at a value of relu
Accuracy was used to select the optimal model using the largest value.
The final values used for the model were layer1 = 1, layer2 = 0, layer3 =
 0, learning.rate = 2e-06, momentum = 0.9, dropout = 0 and activation = relu.

But the problem with this code is, its not giving high accuracy.

Is there any solution to this to find out the optimal hyperparameter of neural network in R ?


texts: Are there any tools to make text labeling faster?

texts:
I saw the labelme tool, and I was wondering if there is a similar tool for labelling short documents ?

preferably in python


texts:
From the comments in my previous question, I'm trying to build my own custom weight initializer for an RNN. Based on the code given here (careful - the updated version of Keras uses 'initializers' instead of 'initializations' according to fchollet), I've put together an attempt.

import numpy as np
import pandas, math, sys, keras
from keras import optimizers
from keras import initializers
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from keras.regularizers import l2
import numpy as np

def rnn_model(hid_dim=10, ker_reg=0.01, rec_reg=0.01, optimizer="sgd", w_mean = 0., w_var = 0.05):
        my_init = lambda shape: initializers.TruncatedNormal(mean=w_mean, stddev=w_var)
        model = Sequential()
        model.add(SimpleRNN(units=hid_dim, activation='relu', kernel_initializer=my_init, recurrent_initializer=my_init, input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))
        model.add(Dense(Y.shape[1], activation='softmax'))
        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])
        print 'fitting a model'
        return model

When I call rnn_model later, I get an error:

    model = rnn_model(hid_dim=hid_val, ker_reg=ker_reg_best, rec_reg=rec_reg_best, optimizer=optim, w_mean=ave_weights, w_var=var_weights)
  File "rnn.py", line 187, in rnn_model
    model.add(SimpleRNN(units=hid_dim, activation='relu', kernel_initializer=my_init, recurrent_initializer=my_init, input_shape = (X.shape[1], X.shape[2]), kernel_regularizer=l2(ker_reg), recurrent_regularizer = l2(rec_reg), return_sequences = False))
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/models.py", line 430, in add
    layer(x)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 257, in __call__
    return super(Recurrent, self).__call__(inputs, **kwargs)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 551, in __call__
    self.build(input_shapes[0])
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/layers/recurrent.py", line 478, in build
    constraint=self.kernel_constraint)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 384, in add_weight
    weight = K.variable(initializer(shape), dtype=K.floatx(), name=name)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 288, in variable
    v = tf.Variable(value, dtype=_convert_string_dtype(dtype), name=name)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 197, in __init__
    expected_shape=expected_shape)
  File "/user/pkgs/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 274, in _init_from_args
    initial_value(), name="initial_value", dtype=dtype)
TypeError: __call__() takes at least 2 arguments (1 given)

Does anyone know how to initialize a Keras model using custom parameters for the initializer?


texts: How to find out if someone cheated by training on the test set?

texts:
Do approaches exist to see if somebody (e.g. authors of a scientific publication) unfairly improved their results by using (parts of) the test set also for training?
Any ideas? Using a new and independent test set might be an option but in the end it does not proof that they cheated - it just shows that the model is bad?


texts: prepare email text for nlp (sentiment analysis)

texts:
I have text of emails, which also contains disclaimers, phone numbers, email addresses, file attachment names, addresses, greetings etc.

At the moment I blindly pass this text through an OOTB sentiment analyser called Vader with poor results (i.e. if I open an email marked as negative, my human understanding does not confirm the sentiment - looking at the core English text).

I could use regex etc. to strip out email addresses, file names etc. but other text components (e.g. addresses, disclaimers) are harder to remove. Btw, disclaimers are often negative ...

Anyway, I wonder if anyone is aware of text prep methods for my scenario - to extract core human text. Google searches were moderately successful. Thanks!


texts: Discovering important topics in corpus of text using metadata and text content

texts:
I am working on a system to classify documents into important/non-important. I have a large (200,000) sample set of documents which have been pre-labelled and using Naive-Bayes I have achieved 95% accuracy in classifying documents using an 80:20 split of training:test data. This result is amazing and I am confident I can tune the system to achieve better results.
However, the system must be able to be used for an indefinite amount of time. This raises the issue that the content of the documents being classified, or what constitutes important/non-important will change over time, so the model must evolve. Thus I need to find a way to train the model without using pre-labeled data. I have only been able to come up with one potential method for this and it didn't work.
The first method I have tried is using LDA topic modelling to create a set of topics from the corpus. I then attempted to find a subset of topics which fit the important documents and a subset which fit the non-important topics. Unfortunately this method did not work as the content of the important/non-important documents is too similar to create obviously distinct topics.
Does anyone have a suggestion for how I could achieve this or any resources that could point me in the right direction?


texts: Predictive models with class value belonging to a set of observations

texts:
I would like to know whether it's possible to build a predictive model where I could define a set of rows with their attributes, and a class belonging to that set of rows, instead of having the typical model one observation - one class.

What I'm trying to do is to create a model which should be able to predict the future level of a rider in professional cycling - more likely to be used to forecast young riders' level . The input data are the results of each rider (race,category of race, type of parcours, position, and so on). The class is represented with the value (from 1 to 10) each "known" rider has been assigned for each skill (mountain, flat, time trial, sprint, etc.), which means that in principle, I would make one predictive model for each skill (what means for each class). For example, for mountain class I would have the following:

Rider   Category    Race Type   Parcours    Distance    Position    Class(Mountain)
-----------------------------------------------------------------------------
Froome  GrandTour   Stage       Mountain    185         1           10
Froome  WorldTour   Stage       MidMountain 210         25          10
Nibali  Contin.Tour Stage       Mountain    170         15          9
Nibali  WorldTour   Classic     MidMountain 245         3           9

I see here two main challenges, apart from the fact of dealing with both numerical and categorical attributes.

A rider gathers a lot of results, and each of them will result in the same class value. For example, a victory in a Grand Tour category stage from Chris Froome would be rated with a 10 in mountain, but he would also be rated with a 10 in mountain when he ended up in a bad position in a lower level stage - just as in the example. Ok, it's supposed that with thousands of rows the model (regression model probably, rather than a classification algorithm, but not sure of that either) should be able to detect the "common" pattern (a good performance in mountain from a good climber like Froome), so it could be overcome. I guess it'd be a matter of testing and finding the best-fitted models.
I find the predictive part the most problematic one, because what I need isn't to predict the class value for each observation (which is represented by each result), but the class value for each rider. That's why I said at the beginning of the thread that I would need to somehow treat all the results of a rider as a unique observation (something like a set or group of observations), and assign the class value to that set. A workaround could be to take all the observations of the rider, predict the class for each of them, and then calculate the average, but it doesn't look very smart.

So, any idea of how to build this kind of models?

Thanks in advance.


texts: Q learning Neural network Tic tac toe - When to train net

texts:
This is another question I have on a q learning neural network being used to win tic tac toe, which is that im not sure i understand when to actually back propogate through the network.

What i am currently doing is when the program plays through the game, if the number of game sets recorded has reached the max amount, every time the program makes a move, it will pick a random game state from its memory and back propagate using that game state and reward. this will then continue every time the program makes a move as the replay memory will always be full from then on.

The association between rewards and game state and action from history, is that when a game has been completed, and the rewards have been calculated for each step, meaning that the total reward per step has been calculated, the method i use to calculate the reward is:

Q(s,a) += reward * gamma^(inverse position in game state)

in this case, gamma is a value predetermined to reduce the amount that the reward is taken into account the further you go back, and the inverse position in game state means that if there have been 5 total moves in a game, then the inverse position in game state when changing the reward for the first move would be 5, then for the second, 4, third 3 and so on. this just allows the reward to be taken less into account the earlier the move is.

Should this allow the program to learn correctly?


texts: Does out-of-bag sampling make Random Forests inherently less robust than other classifiers?

texts:
All of the popular evaluation metrics (ROC-AUC, Confusion Matrices, etc.) require two lists as parameters: a list of the actual y labels associated with some arbitrary group of training examples (x's), and a parallel list of the predicted labels given to those x's by the model.

To construct such lists, you must separate a testing/validation data set from the training set you feed the model on. However, random forests automatically partition 1/3 of the training set you give it to calculate the out of bag score. I don't believe you can stop the bagging process that causes this, since I think it is vital to how the random forest functions. Because the RF model never sees 1/3 of the training set (due to bagging), do RFs create a less thorough image of the dataset then, say, a neural network would whenever a testing set is reserved for evaluation?


texts: Facing this issue while predicting "CountVectorizer - Vocabulary wasn't fitted"

texts:
#these are classifier and  vectorizer

vectorizer = CountVectorizer(tokenizer = spacy_tokenizer, ngram_range=(1,1))
classifier = LinearSVC()

I have created a Pipeline as shown below

# Create the  pipeline to clean, tokenize, vectorize, and classify
pipe = Pipeline([("cleaner", predictors()),
                 ('vectorizer', vectorizer),
                  ('classifier', classifier)])

while predicting the test dataset, im facing this issue.

sample_prediction = pipe.predict(X_test)

ERROR : sklearn.exceptions.NotFittedError: CountVectorizer - Vocabulary wasn't fitted.

Can anyone help ?