texts: Can I conduct independent t-test when data is infested with outliers ? and how to interpret the t-statistics?

texts:
I am working on 2 sample independent t-test. I have conducted analysis on test group vs control group and I have to write a report but I have few questions.

Do we have to take out the outliers and then perform t-test?

Once I perform t-test- can anybody explain the t-test output? The explanation should not be in terms of statistical terms but in such a way that non business person can also understand. I need simple explanation for confidence intervals and difference in means of the two samples.

What kind of charts can we draw to represent our results?


texts: Recurrent neural network producing same predictions

texts:
I am trying to train a recurrent neural network that I built in keras on timeseries data to predict number of sales for next 10 days. For this, I've created my dataset as -

var(t) -> var(t+1)
var(t+1) -> var(t+2)
var(t+2) -> var(t+3)
var(t+3) -> var(t+4) and so on

I did Min-Max scaling on this data and the RNN code is as follows -

model = Sequential()
model.add(LSTM(20, input_shape=(1, look_back),activation='tanh',bias_initializer='ones'))
model.add(Dense(1, activation='linear',bias_initializer='ones'))
opt=adam(lr=0.1)
model.compile(loss='mean_squared_error',optimizer=opt)
model.fit(xtrain, ytrain, epochs=100, batch_size=1, verbose=2)

But the plot I am getting is when I did predictions on xtrain (green = ytrain, blue =  ypred) -

The rnn isn't learning anything at all. Its producing same results for each dataset. I've tried adding hidden layers, increasing number of neurons, changing parameters (learning rate, momentum), optimizers (sgd, adam, adagrad, rmsprop), lstm activation fxn (tanh, softsign). I got little fluctuations in some cases in the graph. But the output is mostly constant.
Also, I've only 200 datasets.

Can someone please guide me what I am doing wrong here. What else I can try. Will small sized data not work using RNN at all ? If so, is there any other way to solve this problem (except ARIMA model) ?

EDIT - Increased batch size to 100 and epochs to 1000. Received some better results. Also, I did mean normalization [(x-mean)/std_dev] instead of MinMax scaling.


texts: Any clear winner for Data Science in Scala?

texts:
I have found a number of libraries and tools for data science in Scala, I would like to know about which one has more adoption and which one is gaining adoption at a faster pace and to what extent this is the case. Basically, which one should I bet for (if any at this point).

Some of the tools I've found are (in no particular order):

Scalding
Breeze
Spark
Saddle
H2O
Spire
Mahout
Hadoop
MongoDB

If I need to be more specific to make the question answerable: I'm not particularly interested in clusters and Big Data at this moment, but I'm interested in sizable data (up to 100 GB) for information integration and predictive analytics.


texts: Understanding how to use ConvLSTM for multistep ahead forecasting

texts:
I have a problem where I have transaction data for many banking accounts. The task is to train a model on historical debit/expense transactions and then forecast expense transactions for the next n days.

The data for each account is aggregated daily - I have total expense values for each account per day. I have done some research and found that the ConvLSTM model (proposed here) is powerful for these multistep ahead forecasts. A example of this model to forecast the next 7 days of power consumption of a household is shown at the bottom of this tutorial.

I am struggling to understand how the convolutional unit is used within the LSTM unit to make these forecasts. Specifically I am having trouble understanding the shape that my input should be in to feed it into TensorFlow's ConvLSTMCell (I think I will need to use the Conv1DLSTMCell sub-class?) and also what format the output/predictions will be returned as.


texts: Use predicted data to improve Multinomial Naive Bayes model for text classification

texts:
For a small project, I am making use of Naïve Bayes Multinomial Model to do some text classification. It has shown some very promising results, especially since I don't have a lot of Training data.
Currently, it happens that some predictions are false, but would it be possible to reuse the text and the corrected class to "improve" the model ?
E.g.: My model has to classify text as either sports, or politics. Let's image that the model classifies text t = "Ronaldo plays at Juventus." as being politics, which is not the case. Would it be possible to "reinject" text t with the correct class into the model, in order to improve it ?
Is it good practice within Machine Learning ?
My model is a basic Multinomial Naïve Bayes (using Sklearn) and makes use of TFIDF and SpaCy to remove unnecessary words from the text (like stop-words, etc.).


texts: Will flattening multivariate time series data before clustering make the results meaningless?

texts:
I have a large number of financial time series that I wish to do cluster analysis on. Each time series has the same length and spans multiple years of daily data (returns, volatility, etc.). As part of my research, I wanted to compare the performance of K-means with the performance of more sophisticated clustering algorithms. I decided to simply stick with Euclidean distance as the similarity measure for K-means.

My issue is that I was unable to find any examples of how to represent such multivariate time series data when you do K-means clustering with Euclidean distance. My solution was to simply "flatten" each time series so a new variable was created for every variable at each time step. For example, for time series $s$, the open price at time step $t$ would become a new variable $o^{s}_{t}$.

Is this a meaningful approach or will the results of my cluster analysis be meaningless? I understand that if you were to, for example, do price forecasting, then flattening a multivariate time series like this would kill the temporal structure of the data. But since I do not care about forecasting and only wish to cluster the data, does my approach not make sense?


texts: Classifying ultrasound videos with a small dataset

texts:
I have a small dataset of ~300 ultrasound clips, about evenly divided between 3 classes. Due to the nature of the data (medical) it's difficult to get more.

I need to train a model (or an ensemble) to classify these clips.

Feature reduction is a necessary first step. I've considered using HOGS (for example) on a cropped part of the video to create relatively small set of time-series features.
However, even a "dumbed down" version of HOGS would give at least 100 time series features.

That being said, lets assume for a minute I'm able to reduce the feature space even more while maintaining variability between classes.

Which model(s) would be best suited for this problem?


texts:
I have enabled sympy on Anaconda to use it to solve basic linear equations, but whenever I try to type something up it gives me an error when defining the variables:

from sympy import *
x,y=symbols(’x,y’);
solution=solve((4*x-3*y-17,7*x+5*y-11),x,y);
P=(solution[x],solution[y]);
print(P);

This code gives me this error message:

  x,y=symbols(’x,y’);
                 ^
SyntaxError: invalid character in identifier

Does anyone know how I could fix this?


texts: Confidence interval interpretation in linear regression when errors are not normally distributed

texts: I've read that "If the error distribution is significantly non-normal, confidence intervals may be too wide or too narrow" (source). So, can anyone elaborate on this? When are the confidence intervals narrow and when are they wide? Does it have anything to do with skewness?


texts: What is the scikit learn Non-negative Matrix Factorisation Coordinate Descent algorithm?

texts:
What is the scikit-learn Coordinate Descent (CD) algorithm for Non-negative Matrix Factorization (NMF)?

The sklearn implementation of NMF has two different solvers, Coordinate Descent and Multiplicative Update. These are introduced in the user guide (here) but not described in detail, except to link to the source papers. For CD, the documentation links to Cichoki & Phan+09 (link is broken in current docs, but you can find it here), but there are 5 different algorithms described therein. The sklearn source code (cython) is given here but I'm struggling to understand the procedure, and link with any of those in Cichoki & Phan.

Could someone describe (and preferably provide a source for) the CD algorithm used in sklearn?


texts: Backpropagation with multiple different activation functions

texts:
How does back-propagation handle multiple different activation functions?

For example in a neural network of 3 hidden layers, each with a separate activation function such as tanh, sigmoid and ReLU, the derivatives of each of these functions would be different, so do we just compute the output error of each of these layers with the derivative of the activation function of that layer?

Also will the gradient of the cost function used to compute the output error of the last layer change at all, or is the activation function of that layer used to compute the gradient of the cost function?


texts:
It is a well known fact that a 1-layer network cannot predict the xor function, since it is not linearly separable. I attempted to create a 2-layer network, using the logistic sigmoid function and backprop, to predict xor. My network has 2 neurons (and one bias) on the input layer, 2 neurons and 1 bias in the hidden layer, and 1 output neuron.
To my surprise, this will not converge. if I add a new layer, so I have a 3-layer network with input (2+1), hidden1 (2+1), hidden2 (2+1), and output, it works. Also, if I keep a 2-layer network, but I increase the hidden layer size to 4 neurons + 1 bias, it also converges. Is there a reason why a 2-layer network with 3 or less hidden neurons won't be able to model the xor function?


texts: Predicting house price using linear regression

texts:
I'm trying to predict a house price using linear regression method. I gather the real data from a real estate website. I have some features and two numerical value in which the price is the target variable to be guessed. I have about 3000 data in which first column is provinces, area field of the house as meter square, following how many salon + rooms, and the other features as 0 or 1. What I try to obtain is a formula(coefficients). However, the Orange Toolkit which I use shows far strange guessing. Is there any wrong step or omitted step(s)? Can be the guesses improved? By the way, via the Box link the dataset can be downloaded.

https://app.box.com/s/0tjroz2tn8h710n6l1q5n5w4y0htn8lt


texts: MCA and FAMD bad scores for UCI census dataset

texts:
Attempting dimensionality reduction on the Census-Income (KDD) Data Set.
The dataset is a mixed dataset with continuous and categorical features.
PCA works fine for continuous variables, reduced down to 2 PCA variables.
[0.84686919 0.14737952]
But having difficulties with categorical variables and cont/cat together.
There are 2 options that are not working out.

multiple correspondence analysis (MCA - for categorical variables)

[0.022592513816454877, 0.022074753604733162, 0.01779832534650477, 0.01644525961669972, 0.013596646212654651]

factor analysis of mixed data (FAMD - for continuous and categorical variables).

Will add these later but they are also very low - not picking up variance in data
Any ideas/tips ?


texts: Using One Hot Encoding VS Label Encoder for Football Match Outcomes

texts:
I am working on a project to predict football matches using bookmakers' odds. The 6 columns I have are Home Team, Away Team, Home Win Odds, Draw Odds, Away Win Odds, Outcome.

The possible football match outcomes are Away Team Win (A), Draw (D) and Home Team Win (H). I have been using label encoders to represent the variables (0, 1, 2 respectively) but it occurred to me that the outcomes may not be ordinal (since there is no order/rank/preferences to it).

Questions

In this case, should I continue with label encoders or should I use one hot encoding? Or it depends on the model used (like Tree based or linear etc)?
If I should change to one hot encoding, how should I ensure that my prediction vector outputs two 0s and one 1 (since it has to be either A, D or H) because sometimes it gave me a vector of three 0s?
I have used one hot encoding for the football teams (20 teams each for both Home and Away). Since a team must play against a unique team, there exists one 1 and nineteen 0s in the first 20 columns and one 1 and nineteen 0s in the next 20 columns. Does this relationship/restriction across the columns affect the models in any way?
Should I apply dimensionality reduction techniques like PCA to it since I now have 20+20+3=43 features as compared to the 5 features previously? I have read that if a tree model is built, the 3 numeric features (bookies' odds) will be rarely used. Or can I just increase the max_features for RandomForestClassifier?

Thanks in advance!


texts: Normalising energy data using weather data?

texts:
I'm currently doing some research into energy use, and normalising it is critical for comparison across years.

I'm considering ways to normalise the daily data (kWh) so that I can run statistical tests, however the standard Heating Degree Day method is lacking for daily normalising (better weekly and monthly).

Has anyone explored any alternative ways of normalising energy use?

Eds. with further information:

There's a linear relationship between kWh and outdoor air temperature, so when comparing energy use (dependent on the occupats behaviour rather than weather) across years it's necessary to normalise for a comparison (otherwise colder winter leads to more energy use).
Scale for outdoor temperatures range between -5 and 35C.
Due to energy use being so linked to weather, a statistical analysis of change in kWh between periods A, B and C needs to be normalised to be comparable.


texts: Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN

texts:
Given MNIST dataset in keras,the challenge is to develop a CNN neural net model with less than 10k parameters with 99% validation accuracy.

I tried making the model for the same but am getting accuracy as 98.71.
Please find below the code for the same.

    def create_model():
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay1=Conv2D(2,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay1)
        lay1=Conv2D(2,kernel_size=(1,1),activation='relu',padding='same')(lay1)
        lay1=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay1)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay2=Conv2D(4,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay2)
        lay2=Conv2D(4,kernel_size=(1,1),activation='relu',padding='same')(lay2)
        lay2=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay2)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(inputs)
        lay3=Conv2D(6,kernel_size=(7,7),strides=(2,2),activation='relu',padding='same')(lay3)
        lay3=Conv2D(6,kernel_size=(1,1),activation='relu',padding='same')(lay3)
        lay3=MaxPooling2D(pool_size=(7,7),strides=(2,2),padding='same')(lay3)
        fc=concatenate([lay1,lay2,lay3])
        fc=Flatten()(fc)
        fc=Dense(10,activation='relu')(fc)
        outputs=Dense(10,activation='softmax')(fc)
        model=Model(input=inputs,output=outputs)
        model.compile(loss=keras.losses.categorical_crossentropy,
          optimizer=keras.optimizers.Adadelta(),
          metrics=['accuracy'])
        return model

The total parameters coming are 8,862 and the batch size used for the above is 32 and the number of epochs are 10.

Can you please suggest ways to improve the model with the constraints on the number of parameters so that the validation accuracy is 99% or above?


texts: How should one deal with null values in time stamp data types?

texts: I have a data set where certain features are time stamps for certain important events like time stamp of when a user logged in etc. How should I deal with null values in such cases. Should one just replace null values with current or most recent time stamp. Are there any standard techniques followed ?


texts: Predicting algorithm elapsed time using a Gaussian Process

texts:
I am new to the Gaussian Processes. I have a simple problem: given some training data of an algorithm performance given its problem size, I want to predict how long it will take (in seconds) to complete.
I have as an input 2 integer parameters that define the problem the algorithm tries to solve and should hint something about the difficulty of the problem. The output is the algorithm elapsed time. I have around 2k runs of that algorithm with different values of the input parameters. The input parameters (let's name them x1 and x2) can have the following values: for x1 it can either be 3, 5, 10, 15, or 20. For x2 it can either be 0, 1, 2, 3, 4, or 5. The output is time in seconds.
Note that I have simplified the details of the problem to something simple to describe. It might not make sense as an application of GP but that's because I have simplified the problem. In principle, the algorithm is stochastic and it's not clear whether it will find a solution or not. Therefore, starting with an easy problem of predicting the time it will take to finish is a good start for me.
I am using Sklearn and its GaussianProcessRegressor object.
I will provide the code I have so far:
# Training data and test data split
training_x, training_y = get_training_data() # 95% of the data
testing_x, testing_y = get_test_data() # 5% of the data
kernel= RationalQuadratic(alpha=1, length_scale=1)
model = GaussianProcessRegressor(kernel)
model.fit(training_x, training_y)
prediction, err = model.predict(testing_x, return_std=True)
print(model.score(testing_x, testing_y)

The score I get back is 0.39 which I find quite bad. I was expecting something higher given that the input space is not large and that the output (time in seconds) is not varying much from run to run.
I am posting this question here in case you can provide me with some pointers to what I should look up or if you spot something I do fundamentally wrong.
Thank you in advance for your time.


texts: Connection between Regularization and Gradient Descent

texts:
I would like to understand regularization/shrinkage in the light of MLE/Gradient Descent. I know both concepts but I do not know/understand whether both are used to determine coefficients of a linear model. If so, what are the steps followed?

To further elaborate, regularization is used to reduce variance which is accomplished through penalizing coefficients of a linear model. The tuning parameter, lambda, is determined through cross-validation. Once, lambda is determined the coefficients are automatically determined, right? Hence, why do we need to minimize (RSS + regularization term) to find coefficients? Are the steps the following:

Find lambda through cross-validation
Minimize (RSS + regularization) through MLE or GD
Find coefficients
Penalize coefficients to decrease variance
We are left with a small subset of coefficients


texts:
I want to compute a confidence interval for each sample for a lightgbm model I've trained.
If the model was a random forest, it'd be quite easy, just take all the trees and compute the standard deviation of the predictions.
I wonder if a similar think can be done with lightgbm. In particular:

Does it make sense to compute the standard deviation of a boosting model by taking the predictions of all trees, or something similar?
Can it be done in scikit-learn?
Are there alternatives to obtain confidence intervals for lightgbm predictions, appart from the alternative that would be training quantile models?


texts: How does the Naive Bayes algorithm function effectively as a classifier, despite the assumptions of conditional indpendence and bag of words?

texts:
Naive Bayes algorithm used for text classification relies on 2 assumptions to make it computationally speedy:

Bag of Words assumption: the position of words is not considered
Conditional Independence: words are independent of one another

In reality, neither of those conditions often holds, yet Naive Bayes is quite effective. Why is that?


texts: Machine Learning library in Python, list or numpy or pandas

texts:
Background: We are trying to build a customized ML library in Python 3 to tackle analysis we often repeat, in a general fashion. But it would not be nearly as general as sklearn. In fact, we are prepared to break some interfaces if that give us enough performance boost in return.

The basic starting point would be constructing a Learner by feeding it X and y, and predict on new input X0

learner = Learner(X, y)
y0 = learner.predict(X0)

One of the design decision is what data type to use for X and y, here are 3 choices, along with some rudimentary brainstorm advantages for each respectively:

native Python list: X being a list of lists, y being a list. Would this have better performance for being 'closer to metal'?
numpy: X being an ndarray of (n, p), y being an ndarray of (n, ). This would benefit from the richer functionalities in numpy/scipy. This is also the data type choice of sklearn.
pandas: X being a DataFrame, y being a Series. This can utilize more Data Analysis (read: dirty work) functionalities from pandas. This way we can also refer to variables with their names instead of just integer indices. But performance would be the worst?

Please share your thinking of pros and cons for each choice from both tech and math perspectives. Thanks in advance!

PS: I thought about whether this should be a StackOverflow question, but still feel this is more Data Science.


texts: How to plot a wordCloud for essay text from a confusion matrix false positive rate count?

texts: I have an essay of text(BOW) and I have modelled it using let's say any model and plotted the confusion matrix and that I have got FPR, I need to plot a word cloud which shows the words due to which the FPR has occurred.


texts: What does Machine Learning Paradigms means, and what are they?

texts:
Please, help me to properly answer the following (main) question:

  What does machine learning paradigms means? And, what are they?

I tried to answer this questions myself, but two completely different answers come up when searching for it, they are represented by the following links:

https://medium.com/datadriveninvestor/learning-paradigms-in-machine-learning-146ebf8b5943
https://www.quora.com/What-are-the-five-major-paradigms-of-machine-learning

I understand that in order to answer the question above, we would also need to answer this additional question:

  Which of these two links is the proper answer for the question above?

Also, I was wondering how come those two very close terms, on those links, could have what seems to be different meanings. Hence, I also have this second additional question:

  Is the term "machine learning paradigms" actually different from "learning paradigms in machine learning"?


texts: What is the encoders job and how does it do that LSTM

texts:
I am studying LSTM-RNNs and ran into a problem! I am not sure what exactly the Encoder's job is, or better: how he does it?
I know that the Encoder is transforming the input in a way, that the Decoder can pull out some useful information of it, but what exactly is it doing? Does it have any weights?  How does it transform the input?


texts: How do I predict new/unknown data in Bayesian linear regression?

texts:
This is my first question on this forum. I just got started with Bayesian statistics. While I do understand the motivations behind Bayesian methods, I am a little unclear on what the predictions even mean. Consider a standard regression problem of predicting the price of a house given its area in square feet. Assume the optimal parameters (slope and intercept) of the model have been found. The price of any new house (x_new) is just the number - {intercept +  slope * x_new}.

In Bayesian linear regression, we work with the so-called posterior predictive distribution (abbreviated PPD). But what is the PPD anyway?

1) Is it a probability density function (pdf) with some parameters like mean and (co)variance? If so, how do I obtain a single value of the house price from this density function? Should I just take the mean of this distribution or are there sophisticated techniques available?

2) Is it a real number, given by the equation - intercept +  slope * x_new? If so,
a. Are intercept and slope sampled from the posterior distribution of slope and intercept?
b. Or are the posterior mean values of slope and intercept used for computing the price of the new house?


texts: Dividing data between test, learn and predict

texts:
I was posting on stats.stackexchange but perhaps I should be posting here.

Context. Subscription business that charges users a monthly fee for access to the service. Management would like to predict "churn" - subscriptions who are likely to cancel. Management would like to create an email sequence in attempt to prevent high risk accounts from churning, perhaps with a discount code of some sort. So I need to identify those accounts at risk of leaving us.

I have a dataset with say 50k records. Each line item is an account number along with some variables. One of the variables is "Churned" with a value of "yes" (they cancelled) or "No" (they are active).

The dataset I have is all data since the beginning of time for the business. About 20k records are active paying customers and about 30k are those who used to be paying customers but who have since cancelled.

My task is to build a model to predict which of the 20k active customers are currently likely to churn.

Here is where I have tied my brain in a knott. I need to run the model (Predict) on the 20k records of active customers.

How do I split my data between training, test and predict?

Does predict data have to be exclusive of train and test data?

Can I split the entire dataset of 50k into 0.8 train and 0.2 test, build a model and then predict on the 20k active accounts? That would imply I'm training and testing on data that I'm also going to predict on. Seems "wrong". Is it?


texts: implementing skip connection in neural nets in pytorch

texts:
I'm trying to implement skip connections in neural nets for tabular data in pytorch

code:
class EmbedNet(nn.Module):
def __init__(self, num_inputs=None, num_net_outputs=None, **kwargs):
    super(EmbedNet, self).__init__()
    self.layers = layers
    self.num_inputs = num_inputs
    self.num_net_outputs = num_net_outputs
    self.cat_cols = cat_cols
    self.numeric_cols = numeric_cols

    self.batchNorm1 = nn.BatchNorm1d(num_inputs)
    self.dropout1 = nn.Dropout(0.15)
    self.dense1 = nn.utils.weight_norm(nn.Linear(num_inputs, 512))

    self.batchNorm2 = nn.BatchNorm1d(512)
    self.dropout2 = nn.Dropout(0.15)
    self.dense2 = nn.utils.weight_norm(nn.Linear(512, 512))

    self.batchNorm4 = nn.BatchNorm1d(512)
    self.dropout4 = nn.Dropout(0.15)
    self.dense4 = nn.utils.weight_norm(nn.Linear(512, num_net_outputs))

    self.wide_block = nn.Linear(num_inputs, num_net_outputs)
    self.wide_block2 = nn.Linear(num_inputs, 512)

def forward(self, x):
    wide = self.wide_block(x)
    wide2 = self.wide_block2(x)
    x = self.batchNorm1(x)
    x = F.relu(self.dense1(x) + wide2)

    x = self.batchNorm2(x)
    x = self.dropout2(x)
    x = F.relu(self.dense2(x) + wide2)

    x = self.batchNorm4(x)
    x = self.dropout4(x)
    x = self.dense4(x)
    x = nn.Sigmoid()(wide + x)

    return x

This model is getting same results every time. It doesn't seem to learn anything.
what am I doing wrong here?
Is there any better way to implement skip connections in pytorch?
what is the best format to implement skip connections for tabular data?


texts: Whether Interaction terms should be included in Linear Regression analysis?

texts:
I am working on a linear model with 6 independent variables and when thinking about including an interaction I got lost.
An interaction exists if the level of one independent variable is affected by another independent variable. Doesn't that therefore mean that if an interaction exists there may also be collinearity problems? Similarly, if the correlation is low between the two variables, then that should imply there is no interaction?
I hope my question makes sense and that someone can help me clear up this confusion. I am now confused how we can ever get to having an interaction between two independent variables ?


texts: Could someone explain to me how back-prop is done for the generator in a GAN?

texts:
I'm not very familiar with neural networks, however, I though I understood the concept of back propagation as starting from the error in the output layer. Say, we have 3 neurons in the output layer and their respective values end up being:

[1 0.5 0.3]

And we wished to obtain values

[0 1 0]

So we can compute a vector of errors between the two:

[-1, +0.5, -0.3] (Not necessarily with the - operation, but you get the point)

And back propagate from there. However, in a the generator of a GAN, it seems to me that the output layer has a bunch of neurons (representative to whatever size the entity we want to generate is) but the error is based only on a % of images the discriminator classified wrongfully. So how exactly do we do back-prop for the generator ?

The only human readable examples of GAN I've found use frameworks and libraries like autoGrad that kind of obfuscate the problem for me :/

Is there a simple way to explain how this backprop can be done ? (Like, in code, not with an "intuitive example" or "simple 4 variable equation where each of the 4 variable packs 50 years of mathematics behind it")


texts: Algorithm for segmentation of sequence data

texts:
I have a large sequence of vectors of length N. I need some unsupervised learning algorithm to divide these vectors into M segments.

For example:

K-means is not suitable, because it puts similar elements from different locations into a single cluster.

Update:

The real data looks like this:

Here, I see 3 clusters: [0..50], [50..200], [200..250]

Update 2:

I used modified k-means and got this acceptable result:

Borders of clusters: [0, 38, 195, 246]


texts: Re-training regression model on covid data

texts:
I am trying to re-train a regression model (XGB regressor) which was used in the pre-covid times (Feb 2020).
The dependent variable for the model is the number of bookings done, and due to covid, the bookings have been affected ,hence it will affect the accuracy of my model.
Any suggestions on how I can manage to re-train my model on covid data and still manage to get decent accuracy?


texts: How could a considerable increase in loss leads to an improvement in accuracy?

texts:
I'm experimenting with NLP and at the moment, I'm trying to come up with a translator model for converting English sentences to French counterparts. I'm using this dataset (not that it's relevant):

https://github.com/udacity/deep-learning/raw/master/language-translation/data

which is composed of more than 137K sentences. My model is an encoder-decoder LSTM with attention implemented in Keras. And here are my plotted validation loss and accuracy charts:

The two accuracy metrics are custom ones developed by myself. But they are based on the same categorical_accuracy from Keras.

Now, my question is why I'm getting an improvement for the accuracy while the loss value is getting wrose?

Also, is such a model trustworthy?


texts: How to perform cross validation with ktrain

texts: Is there a valid code to perform cross validation with ktrain models ?


texts: Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

texts:
I am using a Dice Coefficient based function to calculate the similarity of two strings:

def dice_coefficient(a,b):
    try:
        if not len(a) or not len(b): return 0.0
    except:
        return 0.0
    if a == b: return 1.0
    if len(a) == 1 or len(b) == 1: return 0.0
    a_bigram_list = [a[i:i+2] for i in range(len(a)-1)]
    b_bigram_list = [b[i:i+2] for i in range(len(b)-1)]
    a_bigram_list.sort()
    b_bigram_list.sort()
    lena = len(a_bigram_list)
    lenb = len(b_bigram_list)
    matches = i = j = 0
    while (i < lena and j < lenb):
        if a_bigram_list[i] == b_bigram_list[j]:
            matches += 2
            i += 1
            j += 1
        elif a_bigram_list[i] < b_bigram_list[j]:
            i += 1
        else:
            j += 1
    score = float(matches)/float(lena + lenb)
    return score

However, I am trying to evaluate the best match out of a large possible list, and i want to use list comprehension/map/vectorize the function calls for a whole series of strings to be matched to make this computationally efficient. However, I am having difficult getting the run time into a reasonable ballpark for even medium sized series (10K-100K elements).

I want to send two input series into/through the function, and then get the best possible match from all candidates on dflist1 against a second series: dflist2 . Ideally, but not necessarily, the return would be another series in the dflist1 dataframe return the best possible score also. I have an implementation of this working (below), but it's incredibly slow. Is it also possible to parrelelize this? I think this would be a hugely valueable problem to solve as it would perform the same function that reconcile csv currently does.

dflist1 = pd.read_csv('\\list1.csv', header = 0,encoding = "ISO-8859-1")
dflist2 = pd.read_csv('\\list2.csv', header = 0,error_bad_lines=False)
dflist1['Best Match'] = 'NA'
dflist1['Best Score'] = '0'
d = []
start = time.time()
for index, row in dflist1.iterrows():
    d=[dice_coefficient(dflist1['MasterList'][index],dflist2['TargetList'][indexx]) for indexx,rows in dflist2.itertuples()]
    dflist1['Best Match'][index]=dflist2['TargetList'][d.index(max(d))]
    dflist1['Best Score'][index]=max(d)
    print('Finished '+str(index)+' out of '+str(len(dflist1.index))+' matches after '+str(round(time.time() - start))+' seconds.')

Any help would be appreciated very much!


texts: I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to address them. I found summarize:  + the text to summarize. I also try to find an overview in the paper (https://arxiv.org/abs/1910.10683) there are for instance examples of question-answering in the appendix but without instructions how to tell T5 to answer a specific question. The huggingface.co documentation did not provide any further information besides the summarize: declaration.


texts: What is the difference between ICR and OCR?

texts:
I've just found the term "Intelligent Character Recognition" (ICR) on Wikipedia and other pages. According to Wikipedia:

In computer science, intelligent character recognition (ICR) is an advanced optical character recognition (OCR) or — rather more specific — handwriting recognition system that allows fonts and different styles of handwriting to be learned by a computer during processing to improve accuracy and recognition levels.

Is this just a marketing stunt or are there actually techniques which are specified as OCR and other techniques which are ICR? Are there Conferences / Journals about OCR which distinguish between OCR and ICR? (If yes, please add a reference)


texts: Stats degree from 25 years ago: What do I need to learn now to get into Data Science?

texts:
Having got a bachelors degree in math and a masters degree in statistics back in the mid 90s and then having gotten old and despondent in information technology, I have caught the data science bug, and I'd like to make a switch over the next 2 years.
I have forgotten pretty much all statistics I learned, but I'm sure it'll all come back once I start studying again. I know that I'll have to learn R and/or Python, but real my question is, what statistics do I need to learn and in what order?
I could look at some standard university curriculum, but there is a danger of going crazy and learning all kinds of things that I won't need. So, what books do I really need to read so I don't look stupid on day 1 in a new job, two years from today?


texts: Does SQL Server support the Poisson distribution?

texts: I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson natively within SQL?


texts: How do i cluster binarized categorial data, without knowing the number of clusters?

texts:
I have a dataset of categorical data, and I need to cluster it without knowing k. I know algos for clustering data without knowing the number of centroids, like G-mean, but none works for categorial data.
I think that DBSCAN is also bad because of density.
My dataset is typically a marketing dataset, each row is a customer, and each column corresponds to an attribute, like eyes color. So, I need to binarize this dataset before any algorithm. Any ideas??


texts: To remove Chinese characters as features -

texts:
I have created document-term matrix using TfIdfVectorizer, but just noticed the feature contains Chinese characters. Is it possible to remove them using Python's regex?

I believe these characters are one of reason for lower prediction accuracy of my model.

Currently I use the below for pre-processing my data-

   # Pre-processing the data
    def text_preprocess( data ):
        # Changing to lower case
        data = data.lower()
        # Removing special characters
        data = re.sub("(\\d|\\W)+"," ",data)
        return data

Also, please note I used stopwords='english' in my TfidfVectorizer.

Please let me know if any information required. (New here, still learning)


texts: Prediction interval for very small dataset

texts: Does it make sense to calculate prediction interval for very small dataset (about 60 samples)? I know it's easy for linear regression. But it seems like for tree-based methods or non-linear methods it doesn't make any sense. Am I wrong?


texts: How can I reduce the volume of data (No column to be dropped)?

texts:
I have a data set of 80,000 samples (40k 3 axis accelerometer and 40k Gyro data). I am trying to implement KNN and Random Forest for activity recognition on ESP8266 Node MCU. The limited memory of the MCU is the bottleneck of the process.
Is there any method that can reduce the dataset to, say 5000 records without losing any vital information and without affecting overall accuracy? Dimensionality reduction, as I could understand with my non-mathematical background, is reducing the data by dropping less important columns. However, in my case, I cannot drop any column (only 6 columns are there which are x,y,z values of accelerometer and gyroscope).
Sample data:


texts: Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

texts:
Suppose I've trained a RandomForest model with 100 trees. I then have two cases:

I drop the first tree in the model.
I drop the last tree in the model.

Would the model performance be less in the first or the second case?
As the last tree should be the best trained one, I would say that the first scenario should be less performant than the last one.
And what if I was using another model like a Gradient Boosting Decision tree?  I guess it should be the same.
I am okay with some math to prove it, or any other way that might prove it.
Update
I tried with two different learning rates 0.1 and 8. With 0.1 I get:
# For convenience we will use sklearn's GBM, the situation will be similar with XGBoost and others
clf = GradientBoostingClassifier(n_estimators=5000, learning_rate=0.01, max_depth=3, random_state=0)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:, 1]
# "Test logloss: {}".format(log_loss(y_test, y_pred)) returns  0.003545821535500366

def compute_loss(y_true, scores_pred):
    '''
        Since we use raw scores we will wrap log_loss
        and apply sigmoid to our predictions before computing log_loss itself
    '''
    return log_loss(y_true, sigmoid(scores_pred))

'''
    Get cummulative sum of *decision function* for trees. i-th element is a sum of trees 0...i-1.
    We cannot use staged_predict_proba, since we want to manipulate raw scores
    (not probabilities). And only in the end convert the scores to probabilities using sigmoid
'''
cum_preds = np.array([x for x in clf.staged_decision_function(X_test)])[:, :, 0]

print ("Logloss using all trees:           {}".format(compute_loss(y_test, cum_preds[-1, :])))
print ("Logloss using all trees but last:  {}".format(compute_loss(y_test, cum_preds[-2, :])))
print ("Logloss using all trees but first: {}".format(compute_loss(y_test, cum_preds[-1, :] - cum_preds[0, :])))

which gives:
Logloss using all trees:           0.003545821535500366
Logloss using all trees but last:  0.003545821535500366
Logloss using all trees but first: 0.0035335315747614293

Whereas with 8 I obtain:
clf = GradientBoostingClassifier(n_estimators=5000, learning_rate=8, max_depth=3, random_state=0)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:, 1]
# "Test logloss: {}".format(log_loss(y_test, y_pred)) returns 3.03310165292726e-06

cum_preds = np.array([x for x in clf.staged_decision_function(X_test)])[:, :, 0]

print ("Logloss using all trees:           {}".format(compute_loss(y_test, cum_preds[-1, :])))
print ("Logloss using all trees but last:  {}".format(compute_loss(y_test, cum_preds[-2, :])))
print ("Logloss using all trees but first: {}".format(compute_loss(y_test, cum_preds[-1, :] - cum_preds[0, :])))

gives:
Logloss using all trees:           3.03310165292726e-06
Logloss using all trees but last:  2.846209929270204e-06
Logloss using all trees but first: 2.3463091271266125


texts:
I'm using this dataset : https://archive.ics.uci.edu/ml/datasets/optical+recognition+of+handwritten+digits
a dataset that consists of 65 columns , the last column is the label for 10 classes i.e 0,1,2,3...9. The other 64 columns are the features of each digit. Reshaping the features to (8,8) matrix and plotting it, results in the image of the digit. For example like the one in the picture.
I'm trying to use decision tree regression for training and afterwards I want to give only the first 48 features and let the regression model predict the next 16 pixels.
I think I have to create 16 decision tree regression models, using the sklearn library, I'm not sure what I have to start with. I appreciate any help. Thanks in advance.


texts:
I am trying to visualise a transition from one state (work in origin country) and another (work in destination country). I think transition plot is perfect but I can't get how does it work?

My data frame is called (Cimad) and my variables are EMPLOIPRE (origin) and EMPLOIFR (destination). Both are factors, the first count 20 levels, the second 15 levels (is it a problem, do I have to make them all match?) and I have 400 observations.

I hope you'll be able to help me because I am completely lost !
Thank you in advance
Caroline.


texts:
I am new to CNNs, and I'm trying to follow along with a Pytorch DCGAN tutorial by reimplementing it in Keras. Clearly there are some differences in the frameworks, but in particular I am struggling to understand how the input noise vector is handled per the following image of the generator architecture:

The code goes on to define the first layer of the network as:

# input is Z, going into a convolution
nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
nn.BatchNorm2d(ngf * 8),
nn.ReLU(True),
# state size. (ngf*8) x 4 x 4

Where nz = 100 (size of noise vector) and ngf = 64.

How does that work? Is the noise vector implicitly reshaped by that first convolutional layer to a (?, 4, 4) tensor, as suggested by the diagram? How would that even work? Is there an implicit dense connection between the noise vector and the convolutional layer? How does that first layer result in a tensor of shape (64*8, 4, 4) per the comment?


texts: Which would be an ideal model to get a specific sub string from a bigger string?

texts:
I have a corpus of documents whose some lines have information like this:
wt 210 1b 14.4 oz (98 kg)
or
weight: 219 lb (99 kg), height: 5' 1.9" (157 cm)
The format of occurrence of such information varies from document to document.
I need the value or the substring corresponding to weight and weight only.
Here are my questions regarding the problem:

I have certain regexes that can get the weight value for labeling the lines. However, I do not know how to provide a string as the y-axis, should I convert it to TFIDF vector? Won't that make y-axis hyper-dimensional?

My first intuition is to use Extractive summarizer trained on many other such lines. Is there a better way to handle that?

Thank you.


texts:
I am quite confused because a colleague of mine recently told me that he preferred using SVD instead of PCA (by eigendecomposition) because, contrary to the latter, the former is non-linear so it can identify also some non-linear patterns.

However, I cannot see exactly in what way SVD is non-linear since I have the impression that it simply applies a series of linear matrix multiplications (see also this StackExchange answer).

I know that t-SNE is certainly non-linear and for this reason it is sometimes called as non-linear PCA.

Is SVD non-linear while PCA (by eigendecompostion) is linear?


texts: Create a binary-classification dataset (python: sklearn.datasets.make_classification)

texts:
I would like to create a dataset, however I need a little help. The dataset is completely fictional - everything is something I just made up.

Since the dataset is for a school project, it should be rather simple and manageable.

I would like a few features could be something like:

x1: temperature

x2: color

x3: moisture

and then I would have to classify with supervised learning whether the cocumber given the input data is eatable or not.

I am having a hard time understanding the documentation as there is a lot of new terms for me.

import sklearn.datasets as d           # Python #

a = d.make_classification(n_samples=100, n_features=3, n_informative=1, n_redundant=1, n_clusters_per_class=1)

print(a)

n_samples: 100 (seems like a good manageable amount)

n_features: 3 (3 is a good small number)

n_informative: 1 (from what I understood this is the covariance, in other words, the noise)

n_redundant: 1 (This is the same as "n_informative" ? - well, 1 seems like a good choice again)

n_clusters_per_class: 1 (forced to set as 1)

Would this be a good dataset that fits my needs? If not, how could I could I improve it?


texts: What parameters should be searched with grid search

texts:
In neural networks exist the  parameters
lr, error, iterations.
I used this Library in my code:
trainer.train(trainingSet, {
            rate: getParametersAnn(tipo,parametros).lr,//0.003
            iterations: 20000,
            error:getParametersAnn(tipo,parametros).error,//0.005
            shuffle: true,
            log: 10,
            cost: synaptic.Trainer.cost.CROSS_ENTROPY,

it is recommended that lr be too small to settle in the solution

what parameters should be searched with grid search?
or should I use the default parameters of the library?
with this parameters
rate: 0.001,
            iterations: 2000,
            error: 0.1,
            shuffle: true,
            log: 1,
            cost: synaptic.Trainer.cost.CROSS_ENTROPY

the result is:
iterations 1 error 0.5826960676804478 rate 0.001
iterations 2 error 0.24274412100103793 rate 0.001
iterations 3 error -0.1024786307448624 rate 0.001
iterations 1 error 0.5714651392530429 rate 0.001
iterations 2 error 0.30301059323863505 rate 0.001
iterations 3 error 0.062228013069539526 rate 0.001

.
.
.etc.
and with
rate: 0.1,
iterations: 20,
error: 0.1,
shuffle: true,
log: 1,
cost: synaptic.Trainer.cost.CROSS_ENTROPY

iterations 1 error -7.352525710025565 rate 0.1
iterations 1 error -11.137117275284693 rate 0.1
iterations 1 error -8.64864011321306 rate 0.1
iterations 1 error -11.165450848337727 rate 0.1
iterations 1 error -11.087407126740619 rate 0.1
iterations 1 error -11.157093359867917 rate 0.1
iterations 1 error -11.139330057051765 rate 0.1
iterations 1 error -11.111847588899709 rate 0.1

see the iterations of the second example, it does not advance because, how should the iterations be?


texts: How to fit a KNN and then a linear regression with those neighbors?

texts:
How do I fit a KNN to get the $k$ nearest neighbors and then aggregate the those neighbors into a fit using a linear regression (instead of a weighted average) in Scikit-Learn?
I've tried creating a pipeline using KNeighborsTransformer and then LinearRegression but that doesn't appear to be doing the right thing.


texts: From pairwise comparisons to ranking - python

texts:
I have to solve a ranking ML issue. To start with, I have successfully applied the pointwise ranking approach.

Now, I'm playing around with pairwise ranking algorithms. I've created the pairwise probabilities (i.e. probability of item i being above item j) but I'm not sure how I can transform this to rankings.

For the historical data (let's assume these are queries), I have their pairwise probs AND the actual ranking (the ideal one). I want a solution that will provide a ranking for a new query as well (i.e. the ideal ranking is what I'm looking for here).

Any python package that has, at least partially, the functionality I'm looking for?

EDIT: I have the pairwise probs for for all possible pairs of i and j.


texts: Transformation of Dependent and Independent Variables

texts:
I have a few Independent variables that's normal and a Dependent variables that's skewed , I pick log(feature+SHIFT) to correct skewness. The procedure I follow to get prediction is just take exp(predictions)-SHIFT.

Now how do I get my predictions for the following cases.

1) I have Normal - dependent variable and the independent variables are skewed , I transform only a few of them using same log(feature+SHIFT) where the shifts are unique for every feature.

2) Transformations on both dependent and independent variables with different SHIFT values.

Suitable code examples will be highly appreciated. Thanks


texts: Training a model where each response in the observation data has a different known varience

texts:
I have a dataset where each response variable is the number of successes of N Bernoulli trials with N and p (the probability of success) being different for each observation. The goal is to train a model to predict p given the predictors. However observations with a small N will have a higher variance and higher N.

Consider the following scenario to illustrate better: Assume coins with different pictures on them have a different bias and that the bias is dependent on the picture on the coin. I have a large number of coins each with a different picture on them and each with a different bias p. I want to create a model that can predict the bias of a coin given only the picture on the coin. I flip each coin a different number of times and record the number of successes and total number of flips. So my data set consists of each picture and its estimate p=successes/flips.

So my question is when training my model how should I handle this. It seems more weight should be given to observations with a higher sample size(number of flips). I don't think it makes sense to include number flips as a predictor variable because the point is to build a model which predicts p using only the picture on the coin so this difference in variance for the response for each observation should be taken into account when training the model.

I am using several types of model but mainly working with keras and xgboost


texts: Deep neural networks for mostly zero valued data

texts:
I have some data with around 1000 features. Problem is most of the features are 0. In each row usually around 100 features have values and rest of the features are 0. A sample example data is given below

ID  f1  f2  f3 ........ f1000  label
 1  45  0    0            32     1
 2  23  0    0            0      2
 3  0   40   0            0      1
 4  0   0    23           24     3
 5  78  0    64           0      2

As you can see there are lot of 0 in each row. Which deep neural networks method should I use? I was thinking about RNN or CNN but I am not sure if these are the best choices. I was thinking about some methods which will ignore zero values i.e does not consider the zero values in the calculation. I think RNN can do this. But I am not exactly sure.


texts: Neural networks: which cost function to use?

texts:
I am using TensorFlow for experiments mainly with neural networks. Although I have done quite some experiments (XOR-Problem, MNIST, some Regression stuff, ...) now, I struggle with choosing the "correct" cost function for specific problems because overall I could be considered a beginner.

Before coming to TensorFlow I coded some fully-connected MLPs and some recurrent networks on my own with Python and NumPy but mostly I had problems where a simple squared error and a simple gradient descient was sufficient.

However, since TensorFlow offers quite a lot of cost functions itself as well as building custom cost functions, I would like to know if there is some kind of tutorial maybe specifically for cost functions on neural networks? (I've already done like half of the official TensorFlow tutorials but they're not really explaining why specific cost functions or learners are used for specific problems - at least not for beginners)

To give some examples:

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y_output, y_train))

I guess it applies the softmax function on both inputs so that the sum of one vector equals 1. But what exactly is cross entropy with logits? I thought it sums up the values and calculates the cross entropy...so some metric measurement?! Wouldn't this be very much the same if I normalize the output, sum it up and take the squared error?
Additionally, why is this used e.g. for MNIST (or even much harder problems)? When I want to classify like 10 or maybe even 1000 classes, doesn't summing up the values completely destroy any information about which class actually was the output?

cost = tf.nn.l2_loss(vector)

What is this for? I thought l2 loss is pretty much the squared error but TensorFlow's API tells that it's input is just one tensor. Doesn't get the idea at all?!

Besides I saw this for cross entropy pretty often:

cross_entropy = -tf.reduce_sum(y_train * tf.log(y_output))

...but why is this used? Isn't the loss in cross entropy mathematically this:

-1/n * sum(y_train * log(y_output) + (1 - y_train) * log(1 - y_output))

Where is the (1 - y_train) * log(1 - y_output) part in most TensorFlow examples? Isn't it missing?

Answers: I know this question is quite open, but I do not expect to get like 10 pages with every single problem/cost function listed in detail. I just need a short summary about when to use which cost function (in general or in TensorFlow, doesn't matter much to me) and some explanation about this topic. And/or some source(s) for beginners ;)


texts: How does one go about feature extraction for training labelled tweets for sentiment analysis?

texts:
I am currently working on a sentiment analysis project where the end goal is to try and predict who wins the Nigerian 2019 presidential election. My goal is to create a model that analyses tweets and detects the tone of a tweet (whether positive, negative or neutral). This model can in turn be used to predict who wins. Here's what I have done so far.

I have a program running on a server streaming tweets daily
I have labelled about 5000 tweets manually into different categories
(positive, negative and neutral tweets).
I previously used text blob to determine sentiment on each tweet. But it
was not accurate enough.
I was then advised to train a model on the way Nigerians speak on twitter,
which can sometimes be a little different from normal English

From my research, I found out I have to use feature extraction before I can train the model with my labelled data. My problem now is that I don't understand how to go about these feature extraction. I have read materials about what people use (bag of words, bigrams, unigrams, part of speech tags etc). I don't understand the explanation I got so far, how it all comes together or which to use. Please could anyone explain these in layman's terms or point me to resources that explains this in layman's terms. Thank you


texts:
I want to know of any repositories that contain complete experimental design in R covering basic test and analyses?
I want to take a top-buttom approach to learn step by step through a real project how that works. Do you know of any places to find that, such as an R notebook?


texts: Network design for extracting MFCC features from audio samples

texts:
I currently trying to train a neural network capable of  mapping a input (400 input features) to a output vector of 13 elements.

The input features is an fixed length audio sample, and the output is a feature vector extracted from it.

I made a simple network consisting of one layer:

model.add(Dense(output_dim=13, input_dim=400, init="normal",activation="relu"))

Trained it for 10 epochs gives me these results:

The training shows that the it does become better, but how do i improve it ?

Updated model:

Training and testing:

Plotting (predicted_output - actual_output)[0]
Plot  showing the different between the predicted_output and the actual_output for only the first feature.

overlap plot:


texts: Can I say that a trained neural network model with less parameters requires less resources during real world inference?

texts:
Let us imagine that we have two trained neural network models with different architectures (e.g., type of layers). The first model (a) uses 1D convolutional layers with fully-connected layers and has 10 million learnable prameters. The second model (b) does use 2d conv layer with and has only 1 million paramerts in total. Both model achieve equal scores on the same input data set.
Can I say that model b with less parameter is more favourable because it has less (trainable) prameters and for this reason it requires less resources (GPU-Power, Memory ...) if it would used in a real world scenario? Or does this only show that I is able to learn more effectively in constrast to model a? Are the required resource during real world inference also influenced by the layer types? How can I measure the resource utilization during inference
Just for information: I use tensorflow / keras for both models.
Thanks for your opinon.


texts: Density estimation of hashed strings grouped by events

texts:
Beforehand, sorry for might not use the correct terms, but this is my first adventure in the Data Science world.
Said that this is what I'm searching for: I have data (strings) that will be hashed. Every single string can be classified according to the event the generates it. So imagine that there are the events yellow, green and purple. Then, those strings are grouped according to their similarity, forming a density area in each event. The first question is which model could be suitable to group this data? Note that, two groups could contain the same string (generated twice), once that what distinguishes between them is exactly the event.
Finally, created this density (see image below), once introduced a new string (black dot), we could measure the Hamming distance between this new string and the centroid of each independent cluster.


texts:
I'm using agnes to group terms that frequently appear with one another in a set of documents. I get a dendrogram which verifies that things are working as expected. From this, I'd like to to retrieve the cluster of each member. I'm using cutree to cut the tree at a specified height, but it is returning one extra cluster, I'm not sure why.

length(flex75$order.lab)
[1] 132
length(cutree(flex75, h = 1.5))
[1] 133

It occurs for varying heights, even up to h = max(flex75$height) and all clusters = 1. So I cannot bind the labels to their clusters.

EDIT:

I found the issue to be when I reorder flex75$height to be in ascending order (I was previously receiving an error that: the 'height' component of 'tree' is not sorted (increasingly)). I was also reordering the $order and $order.lab components using the $height order, and turns out that height is of length n-1 and order/order.lab is of length n.

My question now is: What is the relationship between the agnes object's $height vector, and the $order/$order.lab vectors? Can I permute the heights, as I have, and feed that object into cutree? Should any of the object's other properties be correspondingly changed?


texts: Accuracy of KFold Cross Validation for Neural Network

texts:
I have a neural network that Im evaluating using 10 -Fold cross validation. The validation accuracy for a fold changes alot during training in the range of -+10%

So for example the validation accuracy of a fold would range between 80% and 70%.

My question is which number should I consider to be this fold's accuracy.

Should I just take the maximum validation accuracy reached while training or should I just run the training for a certain number of epochs and take the last number (The second approach's result will depend on luck)?

Thanks


texts: Best way to fix the size of a sentence [Sentiment Analysis]

texts:
I am working on a project that is about Natural Language Processing. However I am stuck at the point which is I have a ANN that has fixed size of input neurons.

I am trying to do sentiment analysis with using Imdb movie review set. To able to do that, firstly, I calculated the word embeddings for each word with creating a word-context matrix then applied SVD. So I have the word embedding matrix. But I do not know the best way to compress sentence's vector (which contains embeddings for each word in the sentence) into a fixed size to be able to feed the neural net. I tried PCA but result was not satisfying.

Any help?


texts: Changing colors on Orange Scatterplot widget?

texts:
This is probably has an obvious solution, but I cannot seem to figure it out.
I have this scatterplot below:

I know that there is a drop down menu on the left that allows you to decide how to color the data. However, I cannot seem to find a way to change which color the data points appear as. As you can see from the image above, the colors are not exactly the best for visually displaying the this data.


texts: Identifying if the sentence if it comprise information about education

texts:
Given a sentence I am trying to classify if the sentence contain information about education. For example:

sentence1 = "Require minimum four years of professional degree."
sentence2 = "no degree required for this job."

For identifying as a first step I have built a vocab which have the set of keywords for identifying education based sentences. I was partly successful until I have problems with sentences like this.

sentence3 = "BE or BTech or any degree equivalent to it"

In my vocab 'BE' is also a keyword as 'BE' represents bachelors of engineering degree (in case of country India). As the parsing of algorithm is done in lower case, the issue is coming with 'BE' will be 'be' in sentence. Because of that every sentence that contains 'be' is recognized as educational sentence.

To built a strong ML model I dont have much data. If I want to use vocab based recognition, for this I have to understand the words beside them in the sentence.

Are there any built models so as to import and identify such scenarios or labelled dataset available for it ?
Are there any methods for accomplishing such task ?


texts: When is a time series considered to be "complex"? (not complex-valued)

texts:
what is a measure or definition for complex time series. When is a time series considered to be normal and when it is considered to be a complex?

A lot of researcher saying they predicting/analayzing complex time series. However, I am asking me what is the property that makes a time series a complex one.


texts: IndexError: index 804 is out of bounds for axis 0 with size 800

texts:
i installed a self driving car project from superdatascience site , when i open the map using terminal after a while the map window close up or it closes directly after i maximize the map window and it gives me this error :

    [INFO   ] [Base        ] Leaving application in progress...
 Traceback (most recent call last):
   File "map.py", line 235, in <module>
     CarApp().run()
   File "/usr/lib/python2.7/dist-packages/kivy/app.py", line 826, in run
     runTouchApp()
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 502, in runTouchApp
     EventLoop.window.mainloop()
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/window_sdl2.py", line 727, in mainloop
     self._mainloop()
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/window_sdl2.py", line 460, in _mainloop
     EventLoop.idle()
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 340, in idle
     self.dispatch_input()
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 325, in dispatch_input
     post_dispatch_input(*pop(0))
   File "/usr/lib/python2.7/dist-packages/kivy/base.py", line 231, in post_dispatch_input
     listener.dispatch('on_motion', etype, me)
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/__init__.py", line 1360, in on_motion
     self.dispatch('on_touch_down', me)
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "/usr/lib/python2.7/dist-packages/kivy/core/window/__init__.py", line 1376, in on_touch_down
     if w.dispatch('on_touch_down', touch):
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "/usr/lib/python2.7/dist-packages/kivy/uix/widget.py", line 460, in on_touch_down
     if child.dispatch('on_touch_down', touch):
   File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
   File "map.py", line 181, in on_touch_down
     sand[int(touch.x),int(touch.y)] = 1
 IndexError: index 620 is out of bounds for axis 1 with size 600

the code i have in the brain ( ai.py file ) is

    class Dqn():

    def select_action(self, state):
            probs = F.softmax(self.model(Variable(state, volatile = True))*100) # T=100
        action = probs.multinomial(num_samples=1)
            return action.data[0,0]

the code i have in the map ( map.py file ) is

class MyPaintWidget(Widget):

    def on_touch_down(self, touch):
        global length, n_points, last_x, last_y
        with self.canvas:
            Color(0.8,0.7,0)
            d = 10.
            touch.ud['line'] = Line(points = (touch.x, touch.y), width = 10)
            last_x = int(touch.x)
            last_y = int(touch.y)
            n_points = 0
            length = 0
            sand[int(touch.x),int(touch.y)] = 1

what should i do or change ??


texts: I have data with customer personal information and customer transaction. I cannot figure out how to use the data for training my model?

texts:
Customer information attributes:
ID
Age
Gender
State etc

Customer transaction
ID
Store ID
No of items bought
State etc

Store info
Store ID
State
Daily revenue
Store size etc

I want to predict if customer will buy at a particular store or not/
So can I have the train data with suppose 5 different stores for every customer where the customer shops and then predict in other store?


texts: how to shuffle the data for model.fit with custom data generator?

texts:
So trainfiles is a list that contains the files' directory and name e.g. ['../train/1.npy' ,  '../train/2.npy']
and then I create a dataset as shown in the middle of the code
then I apply it to model fit function

just adding shuffle=True in the argument for model fit function doesn't do anything it seems.
How do I go about shuffling the trainfiles correctly? i.e. when does the trainGenerator function gets executed? i.e. does it gets executed 516 times  when generating 1 batch?

can I just add random.shuffle(trainfiles) right above the for-loop in trainGenerator?


texts: Unable to reduce loss when doing sentiment analysis with Keras

texts:
I created a model in Keras that predicts 4 sentiments/emotions based on text input. Size of my data:
label_1 : 100.000
label_2 : 100.000
label_3 : 100.000
label_4 : 50.000

Validation data: 45.000

I have set the class weights to: class_weight = {'label_1':1, 'label_2':1, 'label_3':1, 'label_1':2}
I have used scikit-learn for vectorising with CountVectorizer. For preprocessing, I have converted all to lowercase, removed emails, urls/links and removed stopwords.
This is my model:

model = Sequential()
model.add(Dense(50, input_dim = features.shape[1], activation = 'relu')) # input layer requires input_dim param
model.add(Dense(100, activation = 'relu'))
model.add(Dense(50, activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(4, activation='softmax'))

opt = SGD(lr = 0.001, momentum = 0.01)
model.compile(loss="categorical_crossentropy", optimizer = opt, metrics=['accuracy'])

es = EarlyStopping(monitor='val_loss', min_delta = 0.03, patience = 120, verbose=1, mode='auto')
history = model.fit(features, results, validation_split = 0.25, shuffle = True, class_weight = class_weight, epochs = 600, batch_size=512, verbose=2, callbacks=[es])

score = model.evaluate(x_test, y_test, batch_size=512)
print()
print(history.history.keys())
print()
print(score)
print('Test loss:', score[0],  'Test accuracy:', score[1])

Test loss: 0.215 Test accuracy: 0.900

These are the accuracy and loss graphs:

This is my prediction:
validation_features = transformerVectoriser.transform(validation_features)
prediction = model.predict_classes(validation_features , batch_size=512) # making prediction

These are confusion_matrix and accuracy_report:
[[11678   256  1181    23]
 [  477 12023   432    13]
 [ 1538   322 10947    18]
 [   16     9    17  6050]]

precision    recall  f1-score   support

0       0.85      0.89      0.87     13138
1       0.95      0.93      0.94     12945
2       0.87      0.85      0.86     12825
3       0.99      0.99      0.99      6092

accuracy                    0.90     45000
macro avg. 0.92   0.92      0.92     45000
weighted avg  0.91  0.90    0.90     45000

My question is, how can I reduce loss? I have tried to change number of layers, nodes and dropout layers, to change optimisers, learning rate and momentum, to change number of epochs and batch size, and to change max number of words (I have tried with 4.000, 5.000, 6.000, 8.000, 10.000, 12.000)
It does not matter what I change, my accuracy is always around 87%-90% (and I think thats good), but my loss is always around 0.21-0.24 (and I do not like that). On some data science related blogs I have read that loss should be less than 0.1. Is that true? What is acceptable loss value for multiclass classification?
Do you have any advices?


texts: Is there a python version of Mitchell's face pose recognition code?

texts:
The original code for chapter 4 in Tom Mitchell's book for recognizing face pose from images is here:  http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/faces.html

But it's in C, not that there is anything wrong with that and I guess that's a really good project to try to port it to Python but I am looking for something I can git clone quickly and use for a personal project.


texts: Where can we find the application of bayes's theorem in Bayesian optimiation with gaussian processing

texts:
I am trying to learn bayesian optimisation by following this tutorial.
However, until now I don't get the relation between bayes's theorem to the gaussian process formalism.

Any ideas?


texts:
In the paper "A NOVEL FOCAL TVERSKY LOSS FUNCTION WITH IMPROVED ATTENTIONU-NETFOR LESION SEGMENTATION" the author use deep supervision by outputing multiple outputmask which have different scale.
I do not understand how it can work with regards to the loss function. y_pred and y_true doesnt share the same dimension exept for the final output.
model = Model(inputs=[img_input], outputs=[out6, out7, out8, out9])

the input seems to only be the one with real resolution

I checked the code (https://github.com/nabsabraham/focal-tversky-unet/blob/master/newmodels.py) and I haven't see anything special that would make it works. The loss function doesn't explicitly handle it neither.


texts:
I am trying out POS tagging using RNN but not able to figure out what's wrong in my implementation because of which the gradient check is failing. Please help. I am pasting the relevant part below:

# weights
Wxh = randn(inputLayerSize, hiddenLayerSize)*0.01; # input to hidden
Whh = randn(hiddenLayerSize, hiddenLayerSize)*0.01; # hidden to hidden
Bh = zeros(1, hiddenLayerSize); # hidden bias
Why = randn(hiddenLayerSize, outputLayerSize)*0.01; # hidden to output
By = zeros(1, outputLayerSize); # output bias

function forwardRNN(x, y, h, p, hPrev)
  global Wxh, Whh, Why, Bh, By;
  cost = 0;
  for time in 1:length(x)
    if time == 1
      h[time] = tanh(x[time]'*Wxh + hPrev*Whh .+ Bh);
    else
      h[time] = tanh(x[time]'*Wxh + h[time-1]*Whh .+ Bh);
    end
    score = h[time]*Why .+ By;
    p_softmax = exp(score) / sum(exp(score));
    p[time] = vec(p_softmax);
    cost += -sum(log(y[time]'*p[time]));
  end
  return cost;
end

function backwardRNN(x, y, h, p, hPrev)
  global Wxh, Whh, Why, Bh, By;
  dWxh = zeros(size(Wxh));
  dWhh = zeros(size(Whh));
  dBh = zeros(size(Bh));
  dWhy = zeros(size(Why));
  dBy = zeros(size(By));
  dh = zeros(size(Bh));
  dhnext = zeros(size(h[1]));
  for time in length(x):-1:1
    dy = p[time] - y[time];
    dWhy = dWhy + (dy * h[time])';
    dBy = dBy + dy';
    dh = (Why*dy)' + dhnext;
    dhRaw = (1 - (h[time].*h[time])) .* dh;
    dWxh = dWxh + (x[time] * dhRaw);
    dBh = dBh + dhRaw;
    if time == 1
      dWhh = dWhh + (hPrev' * dhRaw);
    else
      dWhh = dWhh + (h[time-1]' * dhRaw);
    end
    dhnext = dhRaw*Whh;
  end
  return dWxh, dWhh, dBh, dWhy, dBy;
end

# gradient checking
function gradCheck(inputs, targets, h, p, hPrev)
  paramNameList = ["Wxh", "Whh", "Bh", "Why", "By"];
  global Wxh, Whh, Why, Bh, By;
  paramList = [x for x=(Wxh, Whh, Bh, Why, By)];
  num_checks = 2;
  delta = 1e-5;
  cost = forwardRNN(inputs, targets, h, p, hPrev);
  dWxh, dWhh, dBh, dWhy, dBy = backwardRNN(inputs, targets, h, p, hPrev);
  dParamList = [x for x=(dWxh, dWhh, dBh, dWhy, dBy)];
  for (param,dparam,name) in zip(paramList, dParamList, paramNameList)
    s0 = size(dparam);
    s1 = size(param);
    if s0 != s1
      println("Error dims dont match: ", s0," and ",s1);
    end
    println(name)
    for i in 1:num_checks
      ri = rand(1:length(param));
      old_val = param[ri];
      param[ri] = old_val + delta;
      cg0 = forwardRNN(inputs, targets, h, p, hPrev);
      param[ri] = old_val - delta;
      cg1 = forwardRNN(inputs, targets, h, p, hPrev);
      param[ri] = old_val
      grad_analytic = dparam[ri];
      grad_numerical = (cg0 - cg1) / ( 2 * delta );
      rel_error = abs(grad_analytic - grad_numerical) / abs(grad_numerical + grad_analytic);
      println(grad_numerical,", ", grad_analytic, " => ",rel_error);
      if rel_error > 1e-5
        error("Gradient check failed.");
      end
      println("Gradient check passed.")
    end
  end
end

The code is in Julia programming language and is inspired from Karpathy's min-char-rnn.py


texts:
I would like to use non-atomic data, as a feature for a prediction.
Suppose I have a Table with these features:

- Column 1: Categorical - House
- Column 2: Numerical - 23.22
- Column 3: A Vector - [ 12, 22, 32 ]
- Column 4: A Tree - [ [ 2323, 2323 ],[2323, 2323] , [ Boolean, Categorical ] ]
- Column 5: A List [ 122, Boolean ]

I would like to predict/classify, for instance, Column 2.

I am making something to automatically respond to questions, any type of question, like "Where was Foo Born?" ...

I first make a query to a search engine, then I get some text data as a result, then I do all the parsing stuff (tagging, stemming, parsing, splitting ... )

My first approach was to make a table, each row with a line of text and a lot of features, like "First Word", "Tag of First Word", "Chunks", etc...

But with this approach I am missing the relationships between the sentences.

I would like to know if there is an algorithm that looks inside the tree structures (or vectors) and makes the relations and extract whatever is relevant for predicting/classifying. I'd prefer to know about a library that does that than an algorithm that I have to implement.


texts: Generate new images based on a dataset of images

texts:
Is it possible take a training set of one million 120 x 120 pixel tiles, feed these tiles into a machine learning algorithm. And then make it synthesize images that look like the original training set with a certain, adjustable level of entropy?

What ML techniques should I consider?


texts:
How can I use a neural network for this example? I have already tried perceptron , but I would like to see something else, something that can be supplemented in the future. The example is cropped.
how can you use the data that the neural network should get: people, money, food
I need to make the neural network itself by trial and error clicking on these actions, found the ideal option for getting the maximum number of people(either money or points for each action that comes to mind) how can this be implemented?
import random

class game:
    def __init__(self):
        self.people = 100
        self.gold = 0
        self.food = 1000

    def One_Action(self):

        self.people += round(self.people/100) + random.randint(0, self.gold // 2)
        self.gold -= self.people // 2 + self.food // 2

        if self.gold <= 0:
            game.Lose(self)
        print(self.people)

    def Two_Action(self):
        self.gold += self.people // 2 + self.food
        self.food -= (self.people / 100) * 50

        if self.food <= 0:
            game.Lose(self)

    def Three_Action(self):
        self.food += self.people // 3 * random.randint(2,5)
        self.people -= self.people / 100 * random.randint(1,3)
        self.gold -= self.people / 100 * random.randint(1,3)

    def Lose(self):
        print(f"Max People: {self.people}")

```


texts: Can we talk about vanishing activations?

texts:
When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to the weights of units on deeper layers (according to the chain rule) as well as the results of activations of units on more shallow layers. The deeper the layer, the more activations influence the update and the less gradients do.
On Wikipedia, the vanishing gradients or exploding gradients (which I personally call amplifying gradients) problem is said to affect "early layers". This makes sense, respectively for small or large gradients on later layers.
However, using rectified linear units (ReLU) is often cited as a solution to this, since the gradient is a crisp constant 1 for positive activations. Still, as can be seen on this visualization, a network which uses ReLU is still prone to amplifying gradients.
This time around, however, I would say vanishing gradients are caused by vanishing activations, in turn caused by initializing to values which are too small, and it is late layers, not early layers, which are affected. This answer seems to describe the same idea, referring to them as exploding values.
Am I wrong? Furthermore, in the linked visualization, too small initialization still leads to convergence, but too large initialization doesn't seem to at all... Why is that?


texts: What to do if one out of 2 one-hot encoding variables have a very high p-value?

texts:
I ran an OLS model on a dataset with 2 categorical variables. One of them was gender. The other one had 3 different categories. I used one-hot encoding for it during pre-processing before running my model.

Variables in the image: Embarked_C and Embarked_Q.
The results showed a p-value for Embarked_Q as 0.785. In this case, should I remove both Embarked_Q and C or just Q?


texts: Prediction problem across a wide space of Clash Royale card games

texts:
I have assembled a database of Clash Royale games in an attempt to understand the outcomes of various match-ups.  The game is composed of an 8 card deck drawn from 102 cards.  As you can see from the Cnr, it's a very wide space with many possibilities.  Similarly, this deck will be opposed by a wide and diverse space.
I have been using essentially a 206 element wide table.  One of the variables relates to the relative strength of the players; one hundred and two variables are indicators for the "home deck"; one hundred and two variables are indicators for the "away deck"; and the final element is an outcome variable that is the binary response variable of interest.
I have utilized an xgboost model to predict the outcome of the opposing decks, but I understand that it is not that useful for extrapolation where card combinations have not actually been played in the data.  It seems to me that a logistic approach would be better, but the challenge i am faced with is that the interactions are highly critical, but also explode into too many features to manage.
As additional information, I have used segmentation to "group" the decks into commonly utilized "metas" with good success.  Per my note above, however, this does not help in the identification of "new metas" that might be worth exploring.
I am interested in suggested approaches to solving this problem.  And if you care to see some of my Clash Royale data, I host a website for my clan at The6ixclan.ca


texts: How to classify images Neural Network didn't trained to Understand

texts:
Let's say I trained a Convolution neural network to Identify Cats , Dogs and wolves . But suddenly I feed it pictures of rabbits and  Lions. so  how can I classify those as pictures as "Other"

I tried to do this by Adding "Sigmoid" activation functions and getting probabilities for each selection. I thought probabilities for each selection might be different. I thought If I feed a Rabbit picture to NN it will out 20% cat ,44% dog and 34% wolf. That way  I can clearly figure NN is confused. But Unfortunately it gives me results like 70% cat , 10% Dog , 20% wolf.

Can You suggest me a Way to fix this problem ?


texts: Which loss functions does h2o.gbm use by default?

texts: the GBM implementation of the h2o package only allows the user to specify a loss function via the  distribution argument, which defaults to multinomial for categorical response variables and gaussian for numerical response variables. According to the documentation, the loss functions are implied by the distributions. But I need to know which loss functions are used, and I can't find that anywhere in the documentation. I'm guessing it's the MSE for gaussian and cross-entropy for multinomial - does anybody here know if I'm right?


texts: How to measure the distance (in generalized sense) between geographical regions?

texts:
I need to construct a distance matrix for a few U.S. counties that are adjacent to one or another, and choosing the definition of distance is very tricky. The shortest path (i.e the minimum number of times one has to cross county borders to walk from A to B) and the Vincenty distance between the most populous cities are not working well already. What are other options?

Also, I read somewhere and some time ago that distance between two geographical regions can be measured by the strength of "interaction". Volume of telephone calls, trajectories of trucks, and frequencies of flights were mentioned. Yet I can't any details about such modeling. Can somebody refer me to a paper or something? Are there open datasets of those souces available?

Anybody help me out here?


texts: Regression Task - Spark, PyTorch, TensorFlow or scikit

texts:
I know it's a broad question, sorry for that, but I'm still testing the waters with machine learning.

I have a typical regression task (predict target numbers with the help of features x,y,z) and a dataset with about 1000 entries. I want to use this task to compare several machine learning libraries.

My current selection is Spark, PyTorch, TensorFlow, scikit. Is this selection okay? Do you have further recommendations? My only requirements are that they are open source.

Thanks for your help!


texts:
Problem
I am using Fully Convolution Network (FCN), to classify whether an image is a random noise or not. However my FCN always predict with the means of targets. the output is not stale, but only changing a little (outputs always between  0.49 and 0.51). And after 7000 or more images the loss either suddenly vanish (FCN suddenly overfit, the outputs either 0 or 1, not in between) or explode (FCN suddenly only predict with 1)

How I Train
each batch consist of 8 images, each batch contains either all real or all fake. I only use 200 images for testing purpose, but the problem persist if I use all my data and increase/decrease batch size.

What Works
My FCN able to reach +95% accuracy to classify mnist-like problem where given images, the network try to classify it to 10 classes.

What's different in botch case
in my mnist-like problem, the images are controlled, same background, same brightness, same angle, meanwhile in current problem the images aren't controlled but still the same object. The input plane is different, in my previous problem 1 (gray), in current problem 3 (rgb). The output plane is different, previous problem 10, current 1.

What I did

I tried to throw away some layers, the vanishing/exploding loss still there but happen only after 2000 images instead of 7000
I tried to use noisy label, no vanishing/exploding loss after 14000 images. But the FCN predicts only using the means of the targets
I tried to use tanh instead of logistic, the FCN predicts between -0.003 and 0.003 instead of 0.49 and 0.51
I tried using LeakyReLU and Average Pooling, same result
I tried using SGD and Adadelta instead of adam, same result
I tried using MSE instead Binary Cross Entropy, same result
Mix real and fake images in one batch, same result


texts: Stanford NER is not properly extracting percentages

texts:
I'm trying to extract percentages using Stanford NER. But it is not extracting percentage properly.

import os
from nltk.tag import StanfordNERTagger
os.environ['CLASSPATH'] = 'stanford-ner-2015-12-09/stanford-ner.jar'
os.environ['STANFORD_MODELS'] = 'stanford-ner-2015-12-09/classifiers'

inp_str = 'total revenue received was one hundred and twenty five percent 125% for last financial year'
split_inp_str = inp_str.split()
st = StanfordNERTagger('english.muc.7class.distsim.crf.ser.gz')
print(st.tag(split_inp_str))

This gives following output

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'O'), ('hundred', 'O'), ('and', 'O'), ('twenty', 'O'), ('five', 'PERCENT'), ('percent', 'PERCENT'), ('125%', 'O'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Expected ouput:

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'PERCENT'), ('hundred', 'PERCENT'), ('and', 'PERCENT'), ('twenty', 'PERCENT'), ('five', 'PERCENT'), ('percent', 'PERCENT'),('125%', 'PERCENT'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Why is it not extracting 125% or one hundred and twenty five percent?


texts: StanfordTokenizer will be deprecated in version 3.2.5 Warning

texts:
I was testing the StanfordNERTagger using the NLTK wrapper and this warning appeared:

DeprecationWarning: The StanfordTokenizer will be deprecated in version
3.2.5. Please use nltk.tag.corenlp.CoreNLPPOSTagger or
nltk.tag.corenlp.CoreNLPNERTagger instead.
super(StanfordNERTagger, self).__init__(*args, **kwargs)

My code looks like this:

from nltk import word_tokenize, pos_tag, ne_chunk
from nltk.tag import StanfordNERTagger

sentence = "Today George went to school and met his friend Peter."

# stanford's NER tagger 3 entity classification
st = StanfordNERTagger('/home/hercules/Desktop/PhD/Tools/stanford-ner-
     2017-06-09/classifiers/english.all.3class.distsim.crf.ser.gz',
     '/home/hercules/Desktop/PhD/Tools/stanford-ner-2017-06-09/stanford-
     ner.jar',
     encoding='utf-8')

tokenized_text = word_tokenize(sentence)
classified_text = st.tag(tokenized_text)

print("Stanford NER tagger:")
print(classified_text)

I tried to use CoreNLPNERTagger but I could not find any examples or documentation.
I only found this link:
where it gives something like an example in the comments of the
class CoreNLPNERTagger(CoreNLPTagger)                                      (I found it by searching the keyword "CoreNLPNERTagge")

I tried to follow that example with no use.
I think I should start (if that is the correct term) the coreNLP server first but if is that the case I don't know how.

If anyone got any idea or advice I would be grateful.


texts: Alternatives to imputation of missing values?

texts:
So I'm quickly learning that dealing with missing values for feature(s) in some of your observations is a part of every day life in data.  I get the gist of imputation, when/how it's appropriate and when it's not, and I'll read up on it in the near future.  But, what about this:

Suppose you have predictors $X_1, \dots, X_p$, and you want to model, say, a binary response $Y$ via, say, logistic regression.  Instead of just imputing values to missing predictor values, couldn't you have a separate model built for every possible subset of the predictors, and apply that model for prediction when those predictors happen to be present?  Each of those models would naturally be trained on just the data for which those predictors (and also others) are present.  This seems to me a more reasonable approach than just making up values, but I have no theoretical justification for this.

I do realize that this involves building $2^p$ different models, with $2^p$ different model matrices, etc., but for a moderate $p$ and $n$ it could be feasible, and especially if only a few of your features tend to be missing more often than others.

Is this ever done?  And if so, is there a standard way to implement this in R?  In the case of logistic regression, you can specify to R's glm function how you want it to handle NA values, but your only options seem to be either tossing out observations altogether, or some kind of imputation scheme.

Thoughts?


texts: how to codify date-time feature in a dense vector?

texts:
if we have datetime as a feature for our model in time-series analysis (e.g. trading analysis ...). As you know any datetime-based features have two characteristics:

cyclic
linear

For instance if we consider these five sub-features (mothofyear, dayofweek, dayofmonth, hourofday, minute) extracted from the original datetime feature, then each one of them can be represented by sine, cosine and linear values(offset representation) ...
So depend on how many sub-feature we think are important, we would have 3times of the number of sub-features in the final set of our features.
Now let say the number of our datetime independent features are not that large, for instance we have a maximum of 15 another features.
So if we want to use our datetime aggregated features beside them, since the number of datetime-driven features are so large relative to others then, they easily dominate all other features which definitely have higher importance in comparison to the datetimes itself. Because this way datetime is over-represented.
I'm looking for a way to balance it and hopefully I want to find a way to codify whole datetime-based features as only one feature, I have been thinking concatenating all of them into a one dense vector or hopefully finding a way to codify them into an embedding similar to word2vec (but time2vec, I also found a paper and code about time2vec, but their work is not clear enough).
So I want to know what do you think about it, do have any idea, suggestion, tricks, ... to tackle this issue?


texts:
Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which one is better?:

  1) Treat a as a hyperparameter. Do grid search for a: consider range for a and evaluate the loss for a few points in the range. Pick the best a in the range along with the weights leading to the minimum loss.

  2) Treat a as a parameter. Since loss of the network is a function of parameter a, use gradient descent to update weights as well as parameter a at each iteration.

To me, the second option is better since it can lead to the optimal point but the grid cannot lead to the optimal point, it can lead to the points around the optimal point depending on your selection. But, is the second method right?

Is there a better method?

Lastly, in grid search, how do we pick the range?


texts: Do we calibrate prediction-threshold for a neural network based on prior distribution of each class?

texts:
I have a dataset with 4 classes, say their distribution in the training-set is
$P_{prior}(C1) = 60\% $
$P_{prior}(C2) = 25\% $
$P_{prior}(C3) = 10\% $
$P_{prior}(C4) = 5\% $
I have trained a CNN over the data set (no resampling i.e the network is trained over the true distribution), and I now want to predict unseen data. Usually we assign the data to the class with the highest score/probability, but say I get an outcome from the network on the unseen data, $X$ as:
$P(C1) = 50\%$,
$P(C2) = 10\%$
$P(C3) = 10\%$
$P(C4)=30\%$
we here notice that the probability of C4 is 6 timers greater than it being C4 at random, and C1 has   lower probability than than being C1 at random. Thus I would suggest setting the label as C4 even though $P(C1)>P(C4)$.
My question is; is that apporach wrong, do we always just pick the highest score or does it  make sense to scale each outcome by their prior-distribution i.e setting
$\tilde{P}(C1)=P(C1)/P_{prior}(C1) = 0,83$
$\tilde{P}(C2)=P(C2)/P_{prior}(C2) = 0,4$
$\tilde{P}(C3)=P(C3)/P_{prior}(C3) = 1 $
$\tilde{P}(C4)=P(C4)/P_{prior}(C4) = 6 $
and now assign the label here with the greatest score i.e $C4$ (in case of a tie, pick the greater $P(Ci)$)


texts: What is the difference between a Categorical Column and a Dense Column?

texts:
In Tensorflow, there are 9 different feature columns, arranged into three groups: categorical, dense and hybrid.

From reading the guide, I understand categorical columns are used to represent discrete input data with a numerical value. It gives the example of a categorical column called categorical identity column:

ID   Represented using one-hot encoding
0    [1, 0, 0, 0]
1    [0, 1, 0, 0]
2    [0, 0, 1, 0]
3    [0, 0, 0, 1]

But you also have a dense column called indicator column, which 'wraps'(?) a categorical column to produce something that looks almost identical:

Category (from category column)   Represented as...
0                                 [1, 0, 0, 0]
1                                 [0, 1, 0, 0]
2                                 [0, 0, 1, 0]
3                                 [0, 0, 0, 1]

So both 'categorical' and 'dense' columns seems to be able to represent discrete data, so that's not what distinguishes one from another.

My question is: In principle, what are the difference between a 'categorical column' and a 'dense column'?

I have read this answer that explains the difference between indicator columns and categorical identity columns, but I am looking for a more generic answer distinguishing categorical and dense columns.


texts: How do you determing the correct dimension of Mel Spectrogram Feature Extraction for NN

texts:
I trying to implement a Mel Spectrogram feature extraction:
n_mels = 128

# Extracting MelFrequency Spectrum for every file
def extract_features(file_name):
try:
    audio, sample_rate = librosa.load(file_name, res_type='kaiser_fast')
    mely = librosa.feature.melspectrogram(y=audio, sr=sample_rate, n_mels=n_mels)

except Exception as e:
    print("Error encountered while parsing file: ", file)
    return None

return mely.T

It appears that I am implementing this feature extraction incorrectly as when I check the x_test array it is (353,) and the x_train array is (1408,).  The data is not correctly being parsed and an error is cast.
ERROR BEGIN
    v = format % tuple(row) + newline
TypeError: only size-1 arrays can be converted to Python scalars

When I modify the extract_features code to:
def extract_features(file_name):
    try:
        audio, sample_rate = librosa.load(file_name, res_type='kaiser_fast')
        mely = librosa.feature.melspectrogram(y=audio, sr=sample_rate, n_mels=n_mels)
        melyscaled = np.mean(mely.T, axis=0)

    except Exception as e:
        print("Error encountered while parsing file: ", file)
        return None

    return melyscaled

The program works.
How to get the correct dimension from the definition code without doing any scaling?  What does the np.mean do to the feature extracted?
Also, how do you determine the correct value for n_mels?


texts: Training a neural network with TWO possible correct outputs for one input

texts:
I have a system as a black box that has two correct outputs for a single input sample.
now I want to train a neural network to generate at least one of the correct outputs for that input sample.
what should I do ♂


texts: Calibration using predict_proba vs class_weight

texts:
I am making a Random Forest Classifier to determine whether a sentence is "positive" (1), "negative"(-1) or "neutral"(0).
However, I prefer having false negative than false positive, that is, I prefer saying that a sentence is neutral even if it's not than to say that a sentence is positive when it's neutral, a fortiori if it's negative.
So I use predict_proba, with something like:
def my_pred(rfc, X, weight=0.5):
  res = rfc.predict_proba(X)
  if res[0]>weight: return -1
  elif res[2]>weight: return 1
  return 0

But I wonder if I can make such things (give more importance to the neutral class) using class_weight? Would it be better?


texts: Confusion between doing .copy() and using [[ ]]

texts:
I am reading "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron.
Sometimes he does
xyz = DataFrame[["col1"]]

Which returns a dataframe, but other times:
xyz = DataFrame["col1"].copy()

Which reuturns a series
I do understand that both are ways of copying and not using a 'view', but when should we use each?
(ps: I only found information about views refering to arrayas in numpy... Are the arrays in numpy the same DataFrames in Pandas?)