texts: Categorical data with order and blanks, is frequent dataset or k-modes a better option?

texts:
I have a dataset that's purely categorical: for each item it's ranked across a set of attributes, whether it's easy, moderate or difficult.

But there are blanks if the item doesn't have the attribute. Currently I intend to encode blank as 0 if that's appropriate, and difficulty level as 1, 2, 3.

In this case, should I use Frequent Itemset Mining (Apriory) or k-modes if want to pinpoint which items are the most similar & dissimilar, and how could I identity what kind of attribute combinations result in such clustering? and how do I assess if the clustering result is good or need further tuning?


texts:
Given a dataset that has a binary (0/1) dependent variable and a large collection of continuous and categorical independent variables, is there a process and ideally a R package that can find combinations/subsets/segments of the IVs that are highly correlated with the DV?

Simple example:
DV: college education (0/1), and IVs: age (20 to 120), income (0 to 1 million), race (white, black, hispanic etc), gender (0/1), state, etc.

Then finding correlations combining IVs and subsets of IVs (e.g. women between 30 and 50, with incomes over 100k are highly positively correlated with the DV), and then being able to compare the combinations (e.g. to find out women between 30 and 40, with incomes over 100k have a higher correlation than women between 40 and 50, with incomes over 100k)


texts:
When using the python / sklearn API of xgboost are the probabilities obtained via the predict_proba method "real probabilities" or do I have to use logit:rawand manually calculate the sigmoid function?

I wanted to experiment with different cutoff points. Currently using binary:lgisticvia the sklearn:XGBClassifier the probabilities returned from the prob_a method rather resemble 2 classes and not a continuous function where changing the cut-off point impacts the final scoring.

Is this the right way to obtain probabilities for experimenting with the cutoff value?


texts: I'm looking for alternatives for med7 library for other common languages. Training a custom NER model for different languages seems like not the right option to proceed here. It'd be great if you could suggest some options or places to look for.


texts: Newton's method optimization for Deep Learning

texts:
I'm reading this paper "Deep learning via Hessian-free optimization" by J. Martens, I am having difficulty figure out the following statement:

  In the standard Newton's method, $q_{\theta}(p)$ is optimized by computing the $N\times N$ matrix $B$ and then solving the system $Bp = −\nabla f(\theta)$.

(section 3 of the paper)

Is there any theorem, or statement anywhere regarding why the above system needs to be solved to optimize the local approximation? I came across another paper that has a reference to J. Martens and has used the same statement.


texts:
I am trying to cluster related areas of knowledge based in publications.For example a researcher has 3 keywords in a paper, in another paper he has 5 keywords, but 3 keywords are the same in the both papers. Then these 3 keywords are similar and It could be an area of knowledge. A sample of my file is:

'Domain Ontology,Semantic Web'
'Linked Data,Domain Ontology,Use Case'
'Domain Ontology,Linked Data,Semantic Annotation'
'GIS,Open GIS,Integrated Geo Systems'
'Open GIS,GIS'

My file contains 48963 rows and 19000 keywords. I have tried grouping words (like in the sample), using StringToWordVector (STWV), but I don't have good results. So I tried different K for my cluster since 3 until 13000. When K is greater my log likelihood decrease. In many models my words are grouped perfectly, but the percentages are not really good for some clusters. For example for K=3000, the 2999 have between 0,1% and 1% of the data, but the 3000 have the 21%. I am using WEKA. Anyone know some work related, some advice or any help is helpful...

Cheers.


texts: Validation Loss Decreases Then Increases to The Same Values

texts:
I am training a transformer-based model using Pytorch. The training loss decreases until it hits a floor, which is expected. However, the validation loss decreases to a minimum then starts increasing.
Furthermore, when changing the hyperparameters and features used during training (shown in the chart below), we can see that despite having different minimum validation loss, the validation loss eventually increases to a similar value given enough epochs.

Why do all the validation losses end up with similar values after sufficient epochs has passed?

Can switching optimizers in the middle of training help? Such as starting off with Adam then switching to SGD with momentum? Or adding warmups phases? Or learning rates schedules (and does Adam require LR schedules?)?


texts: How to train a neural network for high recall?

texts:
I would like to train a neural network for named entity recognition to tag an unlabeled dataset of texts. The generated labels will then be checked via a crowdsourcing platform. The goal is to annotate the dataset. Therefore, the neural net should find all possible entities in the text, i.e. have high recall rather than precision.

What would be the best way to train a neural network for high recall, i.e. assigning lower cost to false positives than to false negatives? Could the loss function be changed from negative log likelihood to something else to encourage high recall?


texts: Looking for an RL solution for sequence generation

texts:
I am looking for a proper reinforcement learning solution for the following problem:

Suppose I have a pool of candidate functions f \in Pool(it's like f1, f2, ... fn), and I am trying to synthesize several sequences of functions, each sequence contains arbitrary amount of f.

Given a sequence s, when deciding whether to extend s by appending a new function f and forming a new sequence s' = sf, I have the following reward scheme:

when F(s') = true, reward += 1
when T(s') = true, reward += 3
every time the sequence grows by one, I try to penalize by reward -= 0.5

In other words, I don't want each sequence s to be too long, but also will consider cases when special property F and T are satisfied.

I view this as a learning problem, and envision it can be solved by reinforcement learning techniques, such as DQN or so. But I am a newbie to this field and find it difficult for me to formalize this problem. Could anyone can shed some lights on this and what kind of reinforcement learning algorithms/scheme I can have a try? Thanks a lot. Any suggestion or advice will be appreciated.


texts:
I know the purpose of dropout is to avoid overfitting by deactivating some neurons.

However, I am interested in how it's done, ie: Math behind it, or intuition about why it works.


texts: I have a task of representing a  users feature matrix , i have features like gender , age etc but  I also have a multivalue feature called as "movies watched" which is essentially another table of movie names watched by that user with a numeric duration, the order of movies does not matter here. Also, movies watched can be from 20 movies to 300 movies. So what is the best way of representing this "movies watched" as a feature vector?


texts: I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the similarity scores that the function yields, similarly to how you would test regression results using the MSE?


texts: Looking for binary class datasets with high class imbalance, that also have intra-class imbalance in the minority class

texts:
Newbie question alert...
For a college project I want to compare a few variants of SMOTE in terms of how much they improve classification of the minority class, over using random oversampling.
I have a specific interest in the idea that the minority class may contain small disjuncts that may themselves exhibit imbalance within the class.
I am already looking at the credit card fraud dataset on Kaggle (https://www.kaggle.com/mlg-ulb/creditcardfraud)
Can anyone please point me towards other datasets that have the following kinds of properties:

a reasonably large number of examples (ideally at least a few thousand)
have only two class labels
are highly imbalanced, i.e. the minority class is severely under-represented
ideally the minority examples would have some intra-class imbalance too

Or even better, is there any kind of good search tool out there for finding datasets based on these kinds of characteristics?


texts: Ideal framing of data from LSTM + adding static features

texts:
I'm dealing with a problem that I'll try to simplify here :
You have a land where you can plant seeds and water them. On each day, you have the total area being watered and you have area that has just started being watered that day. The distinction is important : You can think of a production curve for seeds that has a decreasing derivative : Watering them on first days lead to a lot of production and afterwards you still get production out of them but not as much : "Pressing fresh oranges lands more juice"
I want to use LSTM to model this and we assume the following : We can predict production between day 21 and 25 based on the inputs between day 1 and day 15.
Here's how one training sample would look like :

If we go over this example : You can see that we start with a "base" production of 300 because we're watering 2000 sqf that still has some juice to it although not recent. Then we add 500sqf that bring it up to 400 (firtst days of watering that should be most productive) before it decreases to 350 and 320 which is kind of the new base of watering 2200 sqf (you can see that it's 2200 not 2500 because some other area is no longer being watered). Then we add 1000 sqf and production goes up again to 450.
My questions are the following :

If the data is framed this way, would an LSTM be able (at least in theory), to capture that a positive value in "new_area_watered" will be associated to a decreasing overtime uptick in production or should I engineer features like "new_area_added_lastXdays" ? The second option is what I've been doing when trying to frame this as a standard supervised regression problem but I'm thinking that the whole point of a LSTM structure is to get the interaction between timesteps.

Let's think of another example : We want to predict between days 22 and 26. We use input from day 2 to day 16. Let's assume all days between 2 and 16 have 0 new area watered but day 1 actually has >0 positive area watered. The input as framed above would miss out on that. I'm thinking that one way to handle that is to add a static feature that sort of accounts for the "recent" opening balance, like "new_area_added_16to20days_ago". Would that be a good way to tackle it ? If so, what's the best way to combine time-series input with static input in a LSTM architecture ?

Thanks all, looking forward to discussing this with everyone here. Please let me know if something is unclear and I'll try to provide more details or example.


texts:
I am running the following code with data traing with targetBuy as response
variable

# load the library
library(caret)
library(klaR)

# define training control
train_control <- trainControl(method = "repeatedcv", number = 10, repeats = 3)
model <- train(TargetBuy ~ ., data = traing, trControl = train_control, method = "nb")
print(model)

Getting the following error

  Error in train(TargetBuy ~ ., data = traing, trControl =
  train_control,  :     unused arguments (data = traing, trControl =
  train_control, method = "nb")


texts: Forecasting sales during time of epidemic

texts:
As we are going through a tough time because of the Coronavirus epidemic, is it possible to somehow include this affect of this in predicting sales as a time-series for next few weeks?

I am new to time series forecasting but any idea will help here.


texts: How to elegantly caclulate probability distribution parameters for a particular random variable given some observed data?

texts:
I am using bnlearn package and R to learn Bayesian Network structure and also fit it using Maximum Likelihood estimation(MLE). bn.fit uses MLE to learn[as i understand] a generalized probability distribution for all data. However, it is required to obtain probability distribution for a particular variables given observed (evidence) data from other variables.
There one straighforward answer is to sample required variable for given evidence variables and fit probability distribution from that data. I am new to bnlearn package and Bayes Nets and thinking of maybe there is more natural way of obtatining those probability distribution parameters?


texts: Extracting events with attributes from unstructured text

texts:
I am scraping websites of organisations (mostly retailers) and I want to use NLP to extract information from the websites’ unstructured text. The first thing I want to do is to identify covid-related events in the text, for example “The shop will be closed from the 3rd of March” or “Unfortunately we have to close permanently.” The lexicon is rather limited, involving perhaps a few dozen (or hundreds at most) phrases/expressions.
I am very familiar with regular expressions, and I think it is possible to use a rule-based approach to extract some events and their attributes (e.g., dates), particularly with a small lexicon. However, the limitations of the rules are obvious (it is easy to miss expressions with small variations), and I would like to use also some ML approaches. I am familiar with ML approaches like sentiment analysis and topic modelling, but they seem to be designed for classification problems, rather than on this kind of extraction of specific attributes and data points from text. I also know NER that would work well to get dates and place names, for example, but not for events (e.g., closure of a shop x at date y).
Are there smarter ways to do this kind of NLP, going beyond the manual definition of several RegEx? Perhaps a lexical pattern learning from annotated examples?


texts: X_train, y_train from ImageDataGenerator (Keras)

texts:
Can I have X_train, y_train, X_test, y_test from data_generator?
Here is my code:

data_generator = ImageDataGenerator(
    rescale = 1. / 255,
    shear_range = 0.2,
    zoom_range = 0.2,
    horizontal_flip = True,
    vertical_flip = True,
    rotation_range = 180,
    width_shift_range = 0.2,
    height_shift_range = 0.2,
    validation_split = 0.2)

train_generator = data_generator.flow_from_directory(
    train_data_dir,
    target_size =(img_width, img_height),
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset='training')

validation_generator = data_generator.flow_from_directory(
    train_data_dir,
    target_size =(img_width, img_height),
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset='validation')


texts: How to make prediction using tensorflow models?

texts:
As a newbie to tensorflow, I am using this tutorial from google for binary classification using a simple dense neural network.

The slightly annoying thing about this (and a few other) tutorials is they completely gloss over the part of how to actually make a prediction from a dataframe of features, and directly move to model evaluation using some method of the trained model which hides the actual prediction procedure. So basically, I finished the model training, but even after that, I see no way mentioned on how to actually use the model to predict classes of unknown samples.

To put concretely, I have a trained model, a pandas dataframe called test, and a list of columnames which correspond to feature names. Based on the variables declared in the tutorial, I tried

feature_names=['age', 'sex', 'cp', 'trestbps', 'chol', 'fbs', 'restecg', 'thalach', 'exang', 'oldpeak', 'slope', 'ca', 'thal']

model.predict_proba(x=test[feature_names])

But it is throwing a type error. Basically, I need a function which will give me the classes, or ideally, the softmax probabilities of the classes from the feature frames without the label, because that is how we use any model.


texts: Is the interval variable considered as a type of numerical variable or ordinal variable?

texts:
I have a fundamental question about interval variable and I have searched about it in different tutorials but still not sure.
"An interval scale is one where there is order and the difference between two values is meaningful." from graphpad.com
"The interval variable is a measurement variable that is used to define values measured along a scale, with each point placed at an equal distance from one another. It is one of the 2 types of numerical variables and is an extension of the ordinal variable." from formpl.us
My question is about the last sentence above, which states that interval variable is numerical and an extension of ordinal. If we have an atribute 'age' as follows:

Then based on the given definitions above, "age" must be an interval attribute. However, doesn't it seem to be more ordinal than numerical?


texts:
I have a feature array of around 4000 elements, extracted from one source. On this array I've extracted 7 more feature from other source and now I basically have a 4007 feature array from each data point. I am trying to classify these data points based on this feature array, basically doing a 1NN with manhattan distance.

However since I am really bad at maths I'm not sure how to weigh this so my 7 elements can actually help in regards to this. It feels like those 4000 elements are way more important and on distance calculation the other 7 are insignificant.

I've also given this input to a Neural Network and i'm wondering if I need to preprocess something there aswell or can I just give it the 4007 elements to the input neurons?

I've done feature-scaling but I guess that is irrelevant to the problem I'm asking.


texts:
The below graph is a scatterplot of daily stock price. My aim is to predict future stock price of the company.

From the scatterplot it seems that it is a multiplicative model, so I tried to "decompose" it in R. However it says that "time series has no or less than 2 periods". I also obtained a periodogram, which has only one peak at frequency close to 0.

However, my teacher told me that this time series cannot have a trend therefore to eliminate the seasonality I have to consider its period as 7 and then eliminate it choosing an appropriate model.
Can anyone tell me what could be an appropriate model along with a proper justification? Also is it true that the series cannot have a trend?


texts: Include time as a variable in regression model

texts:
I am currently working on a regression problem which requires me to predict the costs of a fixed asset. I have used several variables to do so and derived a predicted cost. However, my superior has wanted me to incorporate time as a variable in the regression model which I am at a loss on how to do so. My data set looks something like this which does not resemble a time series dataset.

Name    Capacity    OEM      Country   Date of valuation  MONTH   YEAR   Cost
A1       220        JAPAN    JAPAN     1/1/2012            1      2012   300,000,000
A2       220        JAPAN    JAPAN     1/1/2012            1      2012   300000000
B1       400        CHINA    CHINA     1/3/2013            3      2013   475000000
B2       400        CHINA    CHINA     1/3/2013            3      2013   475000000
B3       400        CHINA    CHINA     1/3/2013            3      2013   475000000
B4       400        CHINA    CHINA     1/3/2013            3      2013   475000000
C1       750        INDIA    USA       1/5/2016            5      2016   268000000
C2       750        INDIA    USA       1/5/2016            5      2016   268000000

The variables that I have used are capacity, OEM and country. Any help on how to incorporate time to my regression problem is welcomed.


texts: Non-Compact Domain/Range Machine Learning

texts:
What are some machine learning methods that can be applied to non-compact input domain and output range?

Neural Networks is clearly not one of them as they well approximate in an interval.

I have a problem at hand where I need to identify a function with domain the entire real line. The good thing is I know the asymptotic behavior, which is a finite point.

Unfortunately common ML methods diverge immediately outside of training bounds and cannot be hoped to have any asymptotic stability. Polynomials were one idea that came to my mind, but they also diverge asymptotically.


texts: GridSearchCV vs RandomSearchCV and How it works?

texts:
GridSearchCV vs RandomSearchCV

Can somebody explain in-detailed differences between GridSearchCV and RandomSearchCV?
And how the algorithms work under the hood?

As per my understanding from the documentation:

RandomSearchCV

This uses a random set of hyperparameters. Useful when there are many hyperparameters, so the search space is large. It can be used if you have a prior belief on what the hyperparameters should be.

GridSearchCV

Creates a grid over the search space and evaluates the model for all of the possible hyperparameters in the space. Good in the sense that it is simple and exhaustive. On the minus side, it may be prohibitively expensive in computation time if the search space is large (e.g. very many hyper parameters).


texts: Why do cost functions use the square error?

texts:
I'm just getting started with some machine learning, and until now I have been dealing with linear regression over one variable.

I have learnt that there is a hypothesis, which is:

$h_\theta(x)=\theta_0+\theta_1x$

To find out good values for the parameters $\theta_0$ and $\theta_1$ we want to minimize the difference between the calculated result and the actual result of our test data. So we subtract

$h_\theta(x^{(i)})-y^{(i)}$

for all $i$ from $1$ to $m$. Hence we calculate the sum over this difference and then calculate the average by multiplying the sum by $\frac{1}{m}$. So far, so good. This would result in:

$\frac{1}{m}\sum_{i=1}^mh_\theta(x^{(i)})-y^{(i)}$

But this is not what has been suggested. Instead the course suggests to take the square value of the difference, and to multiply by $\frac{1}{2m}$. So the formula is:

$\frac{1}{2m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2$

Why is that? Why do we use the square function here, and why do we multiply by $\frac{1}{2m}$ instead of $\frac{1}{m}$?


texts: Kernel Trick and Inner Product Preservation

texts:
I understand that the point of using the kernel trick is to project the problem onto a higher dimensional space, where the problem is linearly separable. In this explanation, https://www.quora.com/What-is-the-kernel-trick, it states that the inner product $\langle x,y \rangle$ will be equal to $\langle \phi(x),\phi(y)\rangle$. Understanding this equality seems to be key to understanding how this trick works.

My question is, how do know that our function $\phi$ will preserve the inner product and what are the conditions for this to happen?

I have tried google searching this and despite many references to the kernel trick, I do not believe that this has been answered anywhere.


texts:
My goal is to get a smartphone names from Twitter. So this is what I followed:

1- I extracted 100K tweets using the keyword “smartphone”.

2- I Applied LDA after applying ngram tokenization and cleaning.So, I got noisy results such as:(giveway, international, apple, iphone_6, samsung_s5,news…)

3- I filtered the results using a smartphone list (iphone_6, samsung_s5,iphone_4s,…) extracted from DBpedia in order to remove the noise.
Is what I did supervised or unsupervised Machine learning?


texts: How does a FC layer work in a typical CNN

texts: I am new to CNNs and NNs. I am reading this blog: CNN and I am confused about this part: What confuses me is the operation that will be performed on an input vector/matrix. Will we be using a typical ANN equation: "O = W.T * input"?. And then a sigmoid on top of it?


texts:
I recently came across a paper on using (rather simple version of) LSTM for sentiment classification, and it describes its network settings as:

  We randomize the parameters with uniform distribution U(-0.003,
  0.003), set the clipping threshold of softmax layer as 200 and set learning rate as 0.01.

I am trying to reproduce their results with my code written in Tensorflow. I have to say I find it a bit confusing by the meaning of "set the clipping threshold of softmax layer as 200". 200???

Can someone explain this to me please so I know how to implement this using Tensorflow? Thanks so much!!!

Update, ok so I found the their code and below is how they use this clipping threshold of softmax layer:

for(int k = 0; k < softmax.outputG.length; k++)
    softmax.outputG[k] = 0.0;
softmax.outputG[goldPol] = 1.0 / softmax.output[goldPol];

// if ||g|| >= threshold, then g <- g * threshold / ||g||
if(Math.abs(softmax.outputG[goldPol]) > clippingThreshold)
{
    if(softmax.outputG[goldPol] > 0)
        softmax.outputG[goldPol] =  clippingThreshold;
    else
        softmax.outputG[goldPol] =  -1.0 * clippingThreshold;
}

I still can't say I fully understand the use of this threshold..


texts: Rescale prediction to correct dispersion due to correlation between response and residuals in Random Forrest Regression

texts:
I am using Random Forest Regression and I observe a strong positive correlation between the residuals ($\hat{u}$) and the response variable ($y$) which lead to a dispersion : predictions are underestimated, resp. over estimated, for large, resp. small, values of response variable. Note that there is no correlation between predictions - estimated response variable - ($\hat{y}$) and residuals ($\hat{u}$) which is expected for Random Forest.

This issue is due to the fact that Random Forest can be seen as a local average smoother (page 3 in this article) and is well documented as in this post or in this one.

An answer, here, suggests that one way to correct for this dispersion is to "wrap the model in some kind of classical regression which would rescale the response to its desired distribution."

Is it possible to compute some predictions $y_c = f(\hat{y})$ that are corrected for this dispersion ? And how ?


texts: Want to Add More Sample in Majority Class

texts:
I want to add more sample in majority class (not in minority class using oversampling techniques), how to do this?

let suppose I have a car data set having 100 total samples, with class distribution A= 40 sample, B= 25 sample, C= 20 sample, D= 15 Sample, now my question is I want to add sample in A class, so that A= 200, ............. and left the remaining class unchanged


texts: How to do multivariate survival analysis on dataset having only categorical variables

texts:
I have a dataset where I have around 50 independent variables used to run survival analysis on the target variables. But out of these 50 variables, 46 variables are categorical variables i.e. having values in range 0-5(Its a survey dataset where user has to mark the experience between 0 to 5 values, where 5 means the highest). Now, the survey has 10 questions which are to filled from time to time(5 times) in order to assess the change in the user experience. So I have the dataset containing the values for same 10 questions(different values) repeated 5 times to make it 50 columns.

My questions are:

I want to do survival analysis on this dataset. Should I keep the dataset as it is(as the variables have different values from time to time) or should I process the dataset as the questions are repeated 5 times. If yes, can someone please advice any method to do so.
I have tried cox regression and survival trees to learn a model. But the performance of these models is not so good. Also, I am performing feature selection but there is not much difference as almost all the feature have same characteristics.

Below is the image of few columns of the dataset. For privacy reasons, could not share the complete dataset. Kindly understand that almost all the variables of the dataset have such data type except for age and gender.

Any help would be appreciated. Thanks in advance!


texts: Hypothesis Testing and Independent variables Importance

texts:
I am learning Data Science and I have a confusion in one topic. I would like to describe my approach.

I have understood the problem statement.
I have used snowflake schema to join tables that have the attributes required for the job.
I have formulated the dataset.

Now, I am facing a question that how would I know which independent variables are actually significant for the problem. I got a hint that hypothesis testing procedures are used for determining the variable importance. Can someone please tell me that why do we use p-value, z-test, ANOVA test if we can easily find the correlation through the correlation-matrix in python?


texts:
I am trying to solve a decision making problem. In it, information evolves and increases with time for each event observed, and the history of the event may be useful.

The problem is as follows: in an electric network at time $t_0$ there is signal that automatically triggers the possible need of a technical intervention in a network to correct the problem. The system at $t_0$ has a large set of general indicators describing the state of the network based on which I'd like to determine whether to proceed with an intervention at $t_{0}$.

It's possible to carry specific measurements at the point where the problem was detected (this takes some time $\Delta t$). At this stage we can again ask whether the intervention is necessary or not? We can iterate this procedure a few times. Then, my question is, which is the machine learning method best suited for this classification problem? Can it be formulated as a Survival problem?


texts: Scikitlearn grid search random forest using oob as metric?

texts:
Have looked at data on oob but would like to use it as a metric in a grid search on a Random Forest classifier (multiclass) but doesn't seem to be a recognised scorer for the scoring parameter. I do have OoB set to True in the classifier.
Currently using scoring ='accuracy' but would like to change to oob score.
Ideas or comments welcome


texts: Machine learning without explicit training

texts:
Is there any set of machine learning algorithms that do not require training and directly gives answers if a set of labeled and unlabeled data gave at once?

Is skipping explicit training of model can be skipped either in supervised or unsupervised machine learning algorithms? [I do not talk of using pre-trained model]


texts: Information Extraction/Semantic Search for long, unstructured documents

texts:
I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store them in a structured database.

The ultimate goal is to extract and store information in a way that we can query those and any new incoming documents for fast and reliable information. For instance, I want to query a combination of entities from the knowledge base and then return the n-most relevant paragraphs/sentences from the documents. Since some entities like “World Bank” are extracted dozens of times for some documents, I need a way to query the entity in context. Otherwise I just end up with a database that contains the names of specific entities without any way to map them back.

NER usually seems like a good solution for this, however, the documents all have very unique structures which also change from document to document. For instance, a lot of relevant information is stored in tables, but also in long paragraphs.

As far as I understand, NER uses the surrounding words to identify entities, hence loading in the whole documents as raw text and manually tagging terms in the tables will probably not serve as good training data.

For now I built a function that extracts the raw text from pdf, doc and docx documents and extracted entities with spaCys NER model, however, I need to define my own entities and the domain is to scientifc for spaCy to deliever good results.  I also obtained a prodigy license for annotation.

Also, it seems I have to find a way to distinguish between "relevant" text and table entries and "junk" - aka footnotes, annexes, titles, subtitles, etc.

Any input is highly appreciated!

Thanks!


texts: Do we need the Preprocessing step on both Test and Train data sets?

texts:
I have seen many people handle the missing or inconsistent data in both their test and train data sets. Sometimes they handle only the train data sets and sometimes they merge the train and test data sets and handle the missing parameter.

So, what is the best approach and What's the difference of these two? Does any approach effect the predictive model or it's just optional or good practice to handle missing data in both data sets.


texts: Transforming RGB to inputs for neural network

texts:
I have the RGB values from image and i want to make the neural network to recognize and see the colors like humans, is there an algorithm to train neural network with color images?
How can i make the neural network to recognize for example a blue object in more colors(orange, yellow,red,black...and so on)?


texts: How can I improve the accuracy of my confusion matrix to 100%?

texts: Is there a possibility of attaining the above? Can someone share with me how to go about doing it if it is?


texts:
My dataset has 140k rows with 5 attributes and 1 Attrition as target variable (value can either be 0 (Customer churn) or 1 (Customer Does not churn)). I divided my dataset in 80% training and 20% testing. My dataset is heavily imbalanced. 84% of my dataset has 0 as target variable and only 16% has 1 as target variable.

The feature importance of my training dataset is as follows:

ColumnA = 28%, ColumnB = 27%, AnnualFee- 17%, ColumnD - 17% and ColumnE - 11%

I initially wanted to do a very simple check of my model. After creating a Random Forest Classifier I tested the model on a dataset with just 5 rows. I kept all variables constant except Column AnnualFee. Below is a snapshot of my test dataset:

 Column A   Column B    AnnualFee   ColumnD ColumnE
 4500       3.9          5%         2.1      7
 4500       3.9          10%        2.1      7
 4500       3.9          15%        2.1      7
 4500       3.9          20%        2.1      7
 4500       3.9          25%        2.1      7

I expected that as annual fee increases the probability of customer churn also increases. But my rf.predict_proba(X_test) seems to be all over the place. I am not sure why this is happening:

I tried two different codes but the anomaly seems to be happening on both the codes:

Code 1:

rf = RandomForestClassifier(n_estimators = 400,random_state = 0,
min_samples_split=2,min_samples_leaf=5,
                      class_weight = {0:.0001,1:.9999})
rf.fit(X_train, Y_train )

Code 2: Not My Code - Got it Online

from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import GridSearchCV
clf_4 = RandomForestClassifier(class_weight = {0:1,1:5})
estimators_range = np.array([2,3,4,5,6,7,8,9,10,15,20,25])
depth_range = np.array([11,21,35,51,75,101,151,201,251,301,401,451,501])
kfold = 5
skf = StratifiedKFold(n_splits = kfold,random_state = 42)

model_grid = [{'max_depth': depth_range, 'n_estimators': estimators_range}]
grid = GridSearchCV(clf_4, model_grid, cv = StratifiedKFold(n_splits = 5,
random_state = 42),n_jobs = 8, scoring = 'roc_auc')
grid.fit(X_train,Y_train)

I would really appreciate any help on this!


texts:
I'm trying to build my first application scorecard with Logistic Regression and getting low ROC AUC score (about 0.7).
Dataset is about 25'000 rows with 10% of event-rate.

My steps:

Filling missing and rare values.
WOE binning.
Calculating Information Value and deleting low IV (<0.1) variables.
Remaining variables IV values  0.107, 0.112, 0.116, 0.164, 0.191, 0.265, 0.288, 0.321, 0.386, 0.388,
0.457)
Removing highly-correlated variables with 0.9 cut-off. (One var with lower IV was deleted)
Training model:

param_grid = {'penalty': ['l2'], 'C': 10.0 ** arange(-5, 6)}
estimator = LogisticRegression()
search = GridSearchCV(estimator, param_grid)
s_train = search.fit(X_train,y_train)
s_train.best_score_

Getting Accuracy  = 0.87

estimator = s_train.best_estimator_
estimator.coef_
pairs = list(zip(X.columns, estimator.coef_[0]))
pairs = sorted(pairs, key = lambda p: abs(p[1]))
pairs = pd.DataFrame(pairs)
pairs = pairs.rename(columns={0: 'VAR', 1 : "Coef_LR"})
coefs = pairs.sort_values(by = 'VAR')
intercept = estimator.intercept_
search.best_score_,coefs, intercept

Getting coefs (0.00215, 0.00257, 0.00281, 0.00318, 0.00399, 0.00553, 0.006, 0.00757, 0.0084, 0.01051)
and -0.03119 Intercept

ROC AUC and Gini:

y_pred_proba = estimator.predict_proba(X_test)[:,1]
fpr, tpr, _ = roc_curve(y_test,  y_pred_proba)
auc = roc_auc_score(y_test, y_pred_proba)
gini =  (auc-0.5)/0.5
print(('AUC = {}').format(round(auc,3)))
print(('Gini = {}').format(round(gini,3)))

The results is always about 'ROC AUC'=  0.7. I'm trying to get at least 0.8.

What should I pay attention to?


texts: How should I create a single score with two values as input?

texts:
I have two series of values, a and b as inputs and I want to create a score, c, which reflects both of them equally.  The distribution of a and b are below

In both cases, the x-axis is just an index.

How should I go about creating an equation c = f(a,b) such that a and b are (on average) represented equally in c?

Edit: c = (a+b)/2 or c = ab will not work because c will be too heavily weighted by a or b.

I need a function, f, where c = f(a,b) and c' = f(a + stdev(a),b) = f(a, b + stdev(b))


texts: How to Maximize recall for Minority class?

texts:
I have a dataset with 4.7k records and 60 features. 1558 records of indication label 1 and 3554 records indicating label 0. Am solving a binary classification problem. I used SMOTE on training data to balance the data

I applied various feature selection algorithms

1) RFECV - returns optimal 30-40 features based on model that I use

2) SelectFromModel - returns top n features based on what I keyed in. I tried n=15 and n=10

3) Kbest - returns top n features based on what I keyed in. I tried n=15 and n=10

My questions

1) The output from Feature selection algorithm 1,2,3 provides feature which are little bi different. I mean from algo 2 and algo 3, around 10 features are same but remaining 5 are different. What does this imply? Is it normal to see such differences? Or am I making any mistakes here?

2) I tried resampling the minority class using SMOTE and applied Gridsearchcv but still I don't see a good recall value for minority class(which is oversampled now). Please find the screenshot below for metrics

3) Does Gradientboosting, xtreamegradientboosting,catboost algorithms doesn't require us to select features? Are they intelligent enough to pick features and help us increase the accuracy? Should I pass the data (all features) as is

Can you help me as to how can I increase the metric values above 90 for both classes?

Currently am trying only with 15-20 features because certain features from feature selection output were correlated, so I removed them and using only 15-20 features.

Can you offer some suggestions please? CATboost is running for a long time.


texts: Which one of these tasks will benefit the most from SPARK?

texts:
My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. The tasks are:

The first job does analysis of a large quantity of text to look for ERROR messages (grep).
The second job does machine learning & calculate models prediction on some data with an iterative way.

My question is: Which one of the two jobs will benefit from SPARK the most?

SPARK relies on memory so I think that it is more suited to machine learning. The quantity of DATA isn't that large compared to the logs JOB. But I'm not sure. Can someone here help me if I neglected some piece of information?


texts: How to change parameters in LSTM for multivariate binary classification time series?

texts:
https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
I am trying to follow this tutorial's code with a slightly different dataset, where the predictor is a 0 or 1: presence or absence of something. My data is continuous time series data, and I have other continuous variables that are inputs.
For the most part, sans column numbers and names, my code is almost identical to the one in that tutorial. However, my predicted values show up like this on the plot:

And upon changing the "Dense" line from this:
model.add(Dense(1))

into:
model.add(Dense(1, activation='sigmoid'))

and the compile line from:
model.compile(loss='mae', optimizer='adam')

to:
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I thought this was all I needed to change to make the predictions binary, but my predictions plot still keeps giving decimals and nonbinaries, as the plot above shows. What else do I need to account for in a classification problem in LSTM?


texts: Time Series Classification & Seasonal Data

texts:
I am looking to predict values based on seasonal data. Bonuses are paid quarterly/ annually/ monthly and amount usually goes up after couple of time periods. Data is given below. I have converted the Bonus event as the numerical value (Yes = 1, No = 0). I have tried using Excel's forecast functions but it was not useful.
Is there a package with the help of which I can predict Next Month & Amount of Bonus? where the recent data points have higher weightage than the older ones.
My dataset has about 10 years worth of data and about 10,000 personnel. So, it is not possible to predict both Month and Amount manually. I am trying to predict the next Bonus Month and Amount.

Date
Bonus
Amount

Jan-15
0
000

Feb-15
0
000

Mar-15
1
100

Apr-15
0
000

May-15
0
000

Jun-15
1
100

Jul-15
0
000

Aug-15
0
000

Sep-15
1
145

Oct-15
0
000

Nov-15
0
000

Dec-15
1
145

Jan-16
0
000

Feb-16
0
000

Mar-16
1
145

Apr-16
0
000

May-16
1
150

Jun-16
0
000

Jul-16
0
000

Aug-16
1
150

Sep-16
0
000

Oct-16
0
000

Nov-16
1
150

Dec-16
0
000


texts: Gradient boosting vs logistic regression, for boolean features

texts:
I have a binary classification task where all of my features are boolean (0 or 1).  I have been considering two possible supervised learning algorithms:

Logistic regression
Gradient boosting with decision stumps (e.g., xgboost) and cross-entropy loss

If I understand how they work, it seems like these two might be equivalent.  Are they in fact equivalent?  Are there any reasons to choose one over the other?

In particular, here's why I'm thinking they are equivalent.  A single gradient boosting decision stump is very simple: it is equivalent to adding a constant $a_i$ if feature $i$ is 1, or adding the constant $b_i$ if feature $i$ is 0.  This can be equivalently expressed as $(a_i-b_i)x_i + b_i$, where  $x_i$ is the value of feature $i$.  Each stump branches on a single feature, so contributes a term of the form $(a_i-b_i)x_i + b_i$ to the total sum.  Thus the total sum of the gradient boosted stumps can be expressed in the form

$$S = \sum_{i=1}^n (a_i-b_i) x_i + b_i,$$

or equivalently, in the form

$$S = c_0 + \sum_{i=1}^n c_i x_i.$$

That's exactly the form of a final logit for a logistic regression model.  That would suggest to me that fitting a gradient boosting model using the cross-entropy loss (which is equivalent to the logistic loss for binary classification) should be equivalent to fitting a logistic regression model, at least in the case where the number of stumps in gradient boosting is sufficient large.


texts: How to perform Cleaning of a very large set of addresses

texts:
I have a dataset where I have a large number of addresses. The problem lies in the fact that many addresses although they are same but haven't been noted down in the same manner. So I want to find these addresses which are similar and clean my data from the duplicate data.

Is there any standard approach or algorithms that can help me? How should I go about this problem?


texts: Python svm classification, result vs amount of features not as expected


texts: Representing multi-channel input signals with a single signal

texts:
I am working on an EEG signal classification problem. My dataset consists of EEG signals stored as 19X30000 NumPy arrays. Each row represents a single channel. For now, I am converting each of the individual rows into a spectrogram and concatenating all these 19 spectrograms to produce a single image representing the EEG recording that will be fed to a CNN later.

I wanted to know if there is any other way of doing it. Is there any way by which I can represent the 19X30000 input array with a 1X30000 array and use spectrogram of that? I once read about convoluting first across time and then across the channel to reduce it to a single array but I cant remember it exactly. Can someone help me with this?


texts: Estimate class proportions of a feature, central limit theorem

texts:
haven't been feeling smart lately and this is probably the most trivial question ever but I really need to know. I'm trying to point estimate some population parameters. I sampled from 1000 randomly generated bootstrapped samples of 130000 observations and saved the frequency & percentile of each class of a categorical feature.
If I were take an average of the percentile or frequency, would that be a good way to estimate a proportion for the classes of a feature?  I know this issue seems easy.... but please help me.


texts: Opensource tools for help in mining stream of leader board scores

texts:
Consider a stream containing tuples (user, new_score) representing users' scores in an online game. The stream could have 100-1,000 new elements per second. The game has 200K to 300K unique players.

I would like to have some standing queries like:

Which players posted more than x scores in a sliding window of one hour
Which players gained x% score in a sliding window of one hour

My question is which open source tools can I employ to jumpstart this project? I am considering Esper at the moment.

Note: I have just completed reading "Mining Data Streams" (chapter 4 of Mining of Massive Datasets) and I am quite new to mining data streams.


texts: Unable to understand the meaning of following lines of the research paper for image segmentation

texts:
I am implementing a paper on image segmentation. It is based on the slight modification of the u-net architecture.
The paper is based on encoder and decoder steps
Following are the lines of the paper which I am unable to understand.

In the decoder path, features from the encoder path at the same
resolution are fused through concatenation.

and then the architecture of the network is given-

Layer .  Output size .  Filter Size    Stride Dropout
Inputs .  _ .              _ .           _ .    _
Conv1
Conv2
Max Pool1
Conv3
Conv4
Max Pool2
Conv5
Conv6
Max Pool3
Conv7
Conv8
Max Pool4
Conv9
Conv10
Upsampling1
Conv11
Conv12
Upsampling2
Conv11
Conv12
Upsampling3
Conv11
Conv12
Upsampling4
Conv11
Conv12
Output

I am not adding the other columns values for conciseness.
Anyone who want to read the whole paper, it is this. The required lines are in the Table 1.


texts:
In a machine learning procedure, suppose we've chosen k=10 for the "K-Fold Cross Validation".
After we do the k steps of "K-Fold Cross Validation", how do we choose the final model for the classifier ? (The one will we use in order to predict new data)


texts: How does Byte Pair Encoding work on the byte sequence?

texts:
I am reading a paper on OpenAI GPT-2, and in the paper the authors are mentioning that they have performed Byte Pair Encoding (BPE) on the byte sequence themselves, and I am not sure what they meant by that.

How is BPE on the byte sequence different from the regular BPE?

Could you give me some example?

Thank you,


texts: Skewed distributions in predictive models

texts: What are the issues of dealing with highly skewed variable in a supervised problem? What are the machine learning algorithms that suffer more from skewness in the data and what are the solutions to this problem?


texts: "10-year-challenge" data for age algorithms?

texts:
Both on FB and IG, I see people posting themselves before 10y and now. I have no idea how this challenge started.

Could it be a way to collect a colossal amount of data, that could be used to train age-related algorithms (e.g. recognition, progression, ..)?

Forget about data privacy etc., think it technically..for example cleaning the data shouldn't be trivial... Moreover, every person posts it's own version, for example some write onto the images.

On the other hand, FB at least have these photographs of yours more or less, it just asks you to label them now. :)


texts: i'm using GridSearchCV to find parameter C for SVC() classifier present in sklearn.svm . I'm not getting the optimal result desired

texts:
this is a screenshot of my code. i used abc.best_estimator_ (my GridSearchCV model) to find out best results. As you can see grid has values of C=1 and C=100 along with other values. abc.best_estimator_ says C=1 is the best value. For cross checking i tried using different values of c and here i'm getting a better score for C=100. I was getting similar results while finding gamma also, but later on i commented out gamma so as to focus on C only.
Any idea why is this happening? Am i doing something wrong?


texts: Tensorflow, how many layers does this model have?

texts:
I'm very new to machine learning and I'm not sure about Tensorflow implementation of neural network model. How many layers does the model below have?
model = Sequential()
    model.add(Dense(200, activation="tanh"))
    model.add(Dropout(0.3))
    model.add(Dense(1, activation='sigmoid'))

I think the answer is 2, because dropout applies to the input layer, but the input shape is (None, 20) and not 200, so does that Dense layer actually follow a implicit input layer?


texts: How to group every data point with HDBSCAN to some group to have no noise?

texts:
TASK

I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it

GOAL

The goal is when users come on our site and I can show similar products to what they viewing.

QUESTION

How to get all data point to be part of a group, so the goal is to not to have any noise?

CODE
clusterer = hdbscan.HDBSCAN(min_cluster_size=10,#smallest collection of data points you consider a cluster
                            min_samples=1 #LARGER this value -> more points will be declared as NOISE
                           ).fit(data)

color_palette = sns.color_palette('Paired', 2000)
cluster_colors = [color_palette[x] if x >= 0
                  else (0.5, 0.5, 0.5)
                  for x in clusterer.labels_]
cluster_member_colors = [sns.desaturate(x, p) for x, p in
                         zip(cluster_colors, clusterer.probabilities_)]
plt.scatter(*projection.T, s=20, linewidth=0, c=cluster_member_colors, alpha=0.25)

labels = clusterer.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print('Estimated number of clusters: %d' % n_clusters_)


texts: Is a Neural Network with 20 times the number of input neurons (on hidden layers) guaranteed to overfit? When is this not so?

texts:
I'm aware of the problem of over-fitting.  One way to describe it is your Neural Network learning your training data to a high accuracy and performing poorly (generalizing) on new data.

Was wondering if there are situations where having 20 times the number of input neurons on the first hidden layer wouldn't necessarily produce overfitting always.

Here is a screen shot of my work flow.  As you can see I'm using test data.  The way I've done it is split the data myself using a "Past/Future" column with a P or an F.  The future is the last 5% of the sequential time data.

I don't use randomization since I don't think it makes much sense to unsort the sequential time datapoints.

Thanks.


texts: What statistical method should i use to find Correlation between number of days and AmountEarned

texts:
I am new to Data Science and I have a python data frame with Number of days, CountofJobs, and AmountEarned what statistical method should I use to find a correlation between Days and AmountEarned.
NumberofDays     CountofJobs    AmountEarned
20                3                 50000
22                18                10000
35                10                80000


texts:
I have texts collected on different topics. I would like to study the possible correlation between these.
I have started looking at the word frequency and it seems that in one dataset the word with highest frequency is cat; in another dataset is mice;
in another one is house.

Do you know about some technique or approach (similarity, classification,...) to show the possible correlation between datasets (e.g. between cat and mice)? Should it need only the expert judgement?


texts: Get elements from lists in pandas dataframe

texts:
I have the following column of a data frame:

I get it by doing dataFrame['opcodes].

and I would like to consider only the first 20 and the last 20 elements of each list. Is there a way to do this?

I have tried to do the following:

dataFrame['opcodes_modified'] =  dataFrame['opcodes'].apply(lambda x: x[:20].append(x[-20:])

but i get the following error message:

SyntaxError: unexpected EOF while parsing

can somebody help me? Thank's in advance.


texts:
In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order?
For example:

The dog walked in the park.

Can be masked as

The [mask] walked in [mask] park.

In what order (if any) are these tokens predicted?
If you have any further reading on the topic, I'd appreciate it as well.


texts: Experiment click to lead prediction with Azure ML

texts:
I am experimenting now with the Azure ML Studio and I am trying to predict leads based on the clicks I have.

I am exporting a data set of 60.000 Clicks and 8.000 Leads from these clicks.

My data set:

With the "Edit Metadata" I have transformed all the features, except the hasLead in Categorical Feature.

I am testing as many combinations as I can with the data, but my "Recall" is very bad. Do you have some suggestions about what kind of missing data maybe can increase the Recall?

Here the evaluation of the model:


texts: From a mix of real-world and back-calculated data how to remove the part that was back-calculated?

texts:
I have a geostatistical dataset and I've been building linear regression model, but when I plotted the data I've noticed that part of the data shows an absolute straight line trend, i.e. it is most likely not a real-world data, but it was back-calculated.

All the data is in pandas dataframe and I guess I need to find subset of the dataset that shows a perfect linear relationship, but I'm not quite sure how to approach it.


texts: SelectFromModel vs RFE - huge difference in model performance

texts:
Note: I have already looked at Difference between RFE and SelectFromModel in Scikit-Learn post and my query is differnt from that post

Expectation:
SelectFromModel and RFE have similar/comparable performance in the model built using their recommendations.
Doubt: Is there any known usecase where RFE will fare better? As a student of data science (just starting learning) its a weird observation for me
Code:
# RecursiveFeatureElimination_ExtraTreesClassifier

from sklearn.feature_selection import RFE
from sklearn.ensemble import ExtraTreesClassifier

rfe_selector = RFE(estimator=ExtraTreesClassifier(), n_features_to_select=20, step=10)
rfe_selector.fit(x_raw, y_raw)
[x[0] for x in pandas.Series(rfe_selector.support_, index=x_raw.columns.values).items() if x[1]]

# returns
['loan_amnt','funded_amnt','funded_amnt_inv','term','int_rate','installment','grade','sub_grade','dti','initial_list_status','out_prncp','out_prncp_inv','total_pymnt','total_pymnt_inv','total_rec_prncp','total_rec_int','recoveries','collection_recovery_fee','last_pymnt_amnt','next_pymnt_d']

# SelectFromModel_ExtraTreesClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(ExtraTreesClassifier(n_estimators=100), max_features=20)
selector.fit(x_raw, y_raw)

[x[0] for x in pandas.Series(selector.get_support(), index=x_raw.columns.values).items() if x[1]]

# prints
['loan_amnt','funded_amnt','funded_amnt_inv','term','installment','out_prncp','out_prncp_inv','total_pymnt','total_pymnt_inv','total_rec_prncp','total_rec_int','recoveries','collection_recovery_fee','last_pymnt_d','last_pymnt_amnt','next_pymnt_d']

Code for Model train and test
# internal code to select what variables I want
x_train, y_train, x_test, y_test = get_train_test(var_set_type=4)

model = ExtraTreesClassifier()
model.fit(x_train, y_train)

# then just print the confusion matrix

ExtraTreesClassifier Model from SelectFromModel variables

ExtraTreesClassifier Model from RFE variables

My confusion matrix is powered by this Open Source project: DTrimarchi10
/
confusion_matrix


texts: When to use GAN over conventional sampling methods?

texts:
Let's say I have a dataset from a diabetes hospital which has 30000 Type 2 diabetes and 300 Type 1 diabetes patients.
So this dataset has millions and millions of other data points like lab measurements, drugs prescribed and diagnosis data.
Now my objective is to build a model which can classify Type 2 and Type 1 diabetes patients.
As you can see that the dataset is highly imbalanced and I don't have enough T1DM patients to understand their patterns/behavior/feature which can help me differentiate them from T2DM.
So, my question now is when should I use sampling approaches like oversampling and when should I use GANs?
Should I select features of my interest and then apply oversampling or should I apply GAN?
update (addition to the above scenario)
Let's consider another scenario. I have a dataset which has only 300 T1DM patients (there are no T2DM patients). Now, I would like to just increase my dataset size. Let's also think that I don't have any ML task in my mind (meaing classification/ Regression etc) but I know that 300 samples are very less and can never be used for any meaningful analysis. So, now I would like to increase the dataset size and the use it for analysis. Here, GAN is the only solution for the synthetic data? Since no model is involved, I can't apply oversampling etc. can help me with this?


texts:
I am implementing the codes for https://www.kaggle.com/eray1yildiz/using-lstms-with-attention-for-emotion-recognition
which is an emotion analysis from the text. I am having some errors when I am Encoding my samples with corresponding integer values. The following codes are:

X = [[word2id[word] for word in sentence] for sentence in input_sentences]

Error:
Traceback (most recent call last):
  File "<pyshell#32>", line 1, in <module>
    X = [[word2id[word] for word in sentence] for sentence in input_sentences]
  File "<pyshell#32>", line 1, in <listcomp>
    X = [[word2id[word] for word in sentence] for sentence in input_sentences]
  File "<pyshell#32>", line 1, in <listcomp>
    X = [[word2id[word] for word in sentence] for sentence in input_sentences]
KeyError: 'i'

can someone please help me. thanks


texts: Scalable open source machine learning library written in python

texts: I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being built in python (available in github or so) so that I can contribute to that.Is anyone aware of such effort.


texts: Best approach to find optimal solution to linear equation by group in R

texts:
I am currently modeling a pricing and discount system in R.

My data frame looks as follows:

df = structure(
  list(
    Customers = structure(
      c(1L, 1L, 1L, 2L, 2L, 2L),
      .Label = c("A", "B"),
      class = "factor"
    ),
    Products = structure(
      c(1L,
        2L, 3L, 1L, 2L, 3L),
      .Label = c("P1", "P2", "P3"),
      class = "factor"
    ),
    Old_Price = c(5, 2, 10, 7, 4, 8),
    New_Price = c(6, 3, 9,
                  6, 3, 9)
  ),
  class = "data.frame",
  row.names = c(NA,-6L)
)

There are several customers who buy different products with an "Old Price" and a "New Price". I now want to identify one discount parameter (a real from -1.0 to 1.0) for each customer that minimizes the difference of Old Price and New Price.

Because I do not know a lot about optimizations, etc. my current approach would be to do the following, which seems horribly inefficient and might not lead to the optimal solution anyway:

df %>%
    mutate(Individual_Discount = (Old_Price-New_Price)/New_Price) %>% # Identify optimal discount individually
    group_by(Customers) %>%
    mutate(Optimal_Discount = mean(Individual_Discount)) # Average individual discount to get approximate discount for customer

What's the best approach to solve a case like this and how can I implement it in R?

Update:

To clarify the problem more plainly. There is a data frame that looks like this:

Customers | Product | Old Price | New Price | Delta  | Discount | Discounted New Price
CustA     | ProdA   | 10.00     | 12.00     |  2.00  | -0.167   | 10.00
CustA     | ProdB   | 30.00     | 25.00     | -5.00  |  0.2     | 30.00
CustB     | ProdA   | 15.00     | 12.00     | -3.00  |  0.25    | 15.00
CustB     | ProdB   | 20.00     | 25.00     |  5.00  | -0.2     | 20.00

The discount represents the optimal discount to reduce the difference between the old and new price to zero (So a new price 2 would be calculated as New Price + New Price * Discount). However each customer can only get one discount, so which discount should I choose per customer to minimize the remaining deltas (the delta between the discounted new price and the old price)?

Update2: Mathematical relationships

Delta = New_Price - Old_Price

Discount = Delta / -New_Price

Discounted_New_Price = New_Price+New_Price*Discount

Update3:

I have fitted a linear model based on the comments but a "linear discount" based on the gradient of the grouped linear model yields worse results than my "mean hack":

df %>%
  group_by(Customers) %>%
  do({ co <- coef(lm(Old_Price ~ New_Price, .))
       mutate(., linear_discount = co[2])
  }) %>%
  ungroup %>%
  mutate(linear_discount = 1/linear_discount-1) %>%
  mutate(linear_price = New_Price+New_Price*linear_dis

The results are

Customers | Product | Old Price | New Price | Linear Discount  | Linear Price | Discounted New Price
CustA     | Prod1   | 05.00     | 06.00     |  -0.25           | 4.50
CustA     | Prod2   | 02.00     | 03.00     |  -0.25           | 2.25
CustA     | Prod3   | 10.00     | 09.00     |  -0.25           | 6.75
CustB     | Prod1   | 07.00     | 06.00     |   0.50           | 9.00
...


texts: What are the alternatives to Python + Spark (pyspark)?

texts:
I like Python, and I like Spark, but they don't fit very well together.  In particular,

it is very hard to use python functions in spark (have to create JVM binding for function in python)
it is hard to debug pyspark, with py4j in the middle

So I wonder if there are any alternatives to pyspark that supports python natively instead of via an adapter layer?
Reference


texts:
im new to this community and it always helped me with my concerns, i looked for an answer but didnt find a clear one yet

im working on study for insurance default, the data i received is already giving me the default, the issue here is im trying to predict the default and its represented as 0 and the not default is represented as 1

do i need to convert the values to default as 1 and not default as 0, what is the best practice here?

bearing in mind that the data is unbalanced toward not default (1 currently) also i have a lot features that are minus correlated with the default in the current setup

appreciate the help


texts: Grouping data where each cluster has a similar sum of a variable?

texts:
I have a data set of postcode areas and their populations. I would like to cluster the data set in a way in which each cluster has roughly the same population of 80,000. Phrasing mathematically I'd like to minimize the sum of the norms of (populations of each cluster - 80,000).
How would I go about this? I have researched unsupervised clustering but all those methods effectively aim to minimize within and between cluster similarity.


texts: Why do we have to remove most common words for text analysis?

texts:
I am trying to do sentiment analysis the task is to classify racist tweets from other tweets. And I have read many articles and many have mentioned to remove the most common 10 words from the column because their presence will not of any use in classification of our text data.

So these are my top 10 most common words on my dataset.

[('love', 4271),
 ('day', 3572),
 ('amp', 2709),
 ('happy', 2651),
 ('u', 1840),
 ('time', 1771),
 ('im', 1770),
 ('life', 1756),
 ('like', 1700),
 ('today', 1591)]

If I remove these will my classification model be more accurate?

Similarly they are also recommending to remove the top 10 rare words from the column.

I want to know why? Any help


texts: Extracting popular keyword terms or topics from blog posts based on post usage

texts:
I have dataset of posts from blog and for each post I have the number of views.  I want to extract the topics (or phrases) that made the posts with more views.

I am planning divide all posts in two sets based on number of views (one set with low number of views and the other with higher numbers), then extract topics using LDA from each set and compare how they differ.

I am wondering if this is right approach and if there are other approaches that can be better or similar?


texts: Explanation of generative adversarial net in Generative adversarial networks by IJ Goodfellow and et at. (2014)

texts:
In a paper titled Generative adversarial networks (link in comment - sorry I don't have enough reputation to include more than 2 links), an explanation of generative adversarial net is provided as following:

I have 2 questions about the explanation of figure 1:

"G contracts in regions of high density and expands in regions of low density of p_g". I'm interpreting this to mean that the maximum of the green curve indicates region of contractions, while the region of curve flattening indicates region of increasing expansion. Why are the concepts of contractions and expansions visually represented in such a manner ?
The blue dashed line is supposed to indicate the discriminative distribution's discrimination between samples from the data generating distribution and the generative distribution. It's unclear to me how the blue dashed line represents the discrimination (why does a straight horizontal line represents the discriminator's inability to differentiate between samples taken from real data and taken from the generative distribution?)

Thank you!


texts: is this a classification or clustering problem?

texts:
Suppose, I am building a hotel recommendation system, that learns user profile based on his/her interaction with the system.

I have two classes, "Like" and "Dislike". For example, a user likes 5 "5-star" hotels, likes 3 "1-star" hotels and dislikes 10 "1-star" hotels. Now, a Naive Bayes Classifier will put a new 5 star hotel in the "Like" class and a new 1 star hotel in "Dislike" class. Now, what I am trying to achieve is a bit different, I don't want to classify, instead I want to extract features/attributes more closely related with the "Like" class. For example, I need "5-star" attribute from the "Like" class.

Now, obviously, I have many more attributes, than the hotel star rating. What I want to know is that, can Naive Bayes be modified to extract features from a class. Or are there any specialized algorithms that are more suited for this task?

I would have data in this form

user_id | hotel_id | hotel_rating | hotel_attr1 | .... | hotel_attr(n)| preference

1       1        5 star        some value   some value    like

1       2        1 star        some value   some value    dislike

Another thing, If I want to extract features such as "5 star" and "1 star" from a class, Is the data formatted above correct? or instead of hotel_rating as the feature, I need 5_star and 1_star as the features, instead of them being values of the hotel_rating feature?


texts: I have been taking the FastAI course. And if anyone else has taken this course they know that they use their own python library called fastai that is a wrapper for PyTorch functions. This is effective for having time to explain theory, but I am wondering how these same methods would be implemented directly from PyTorch. Has anyone seen any blog posts or anything where someone has rewritten some fastai lessons in base PyTorch commands? I found one here: https://medium.com/@tomgrek/wtf-is-going-on-with-fast-ai-db59741b5da2 but that is all I could find, and I'm not even sure which lesson this converts.  Thanks!


texts: What do we visualize in showing a VAE latent space?

texts:
I am trying to wrap my head around VAE's and have trouble understanding what is being visualized when people make scatter plots of the latent space. I think I understand the bottleneck concept; we go from $N$ input dimensions to $H$ hidden dimensions to a $Z$ dimensional Gaussian with $Z$ mean values, and $Z$ variance values. For example here (which is based off the official PyTorch VAE example), $N=784, H=400$ and $Z=20$.

When people make 2D scatter plots what do they actually plot? In the above example the bottleneck layer is 20 dimensional, which means there are 40 features (counting both $\mu$ and $\sigma$). Do people do PCA or tSNE or something on this? Even if $Z=2$ there is still four features so I don't understand how the scatter plot showing clustering, say in MNIST, is being made.


texts: Creating categorical variable, without knowing true categories (through binning time series data)?

texts:
I have a temperature dataset (data every 15 mins) to build a supervised classification/prediction algorithm, but only know one of the true classes (when data is nearly flatlining around 35deg)

However, given the academic literature on the subject and the data visualization it looks like there are three "true" classes: regulated, sub-ideal regulation, and no regulation. Here is a graph with vertical lines delineating 3 potential categories.
What is the best practice to build this classification? First attempt binned data every 6hrs and built categories based off stdev, but that seemed a bit arbitrary and did not align well with the three hypothesized categories.


texts: I have a simple question about convolution layers in CNNs. Consider that we have 32 features map with size $100\times100$. So, can we set 16 convolution layer with size $9\times9\times16$ after features map? There isn't any limit on depth of the convolution layer? Then if we can, so every $9\times9\times16$ layer produces just one layer? (How does depth=16 work on depth=32)?


texts: Can I treat concept drift detection as a binary clarification problem?

texts: Assume I have the ground truth labels for both non-drited and drifted samples, can I treat concept drift detection as a binary classication problem (one class non-drifted one class drifted)? If not why? One issue that I can see is drifting can occur in different ways so potentially it can become a multi-class classification problem (with 1 non-drifted class and many drifted classes) but it should still be possible to group all drifted classes as one class?


texts:
I expected sklearn's mutual_info_classif to give a value of 1 for the mutual information of a series of values with itself but instead I'm seeing results ranging between about 1.0 and 1.5.  What am I doing wrong?
This video on mutual information (from 4:56 to 6:53) says that when one variable perfectly predicts another then the mutual information score should be log_2(2) = 1.  However I do not get that result:
import pandas as pd
from sklearn.metrics import confusion_matrix

y = [1,1,1,1,1,0,0,0,0,0]
print("Confusion matrix:")
print(confusion_matrix(y,y))

print("Mutual information:")
result = mutual_info_classif(pd.DataFrame(y), y)
print(result)

which gives:
Confusion matrix:
[[5 0]
 [0 5]]
Mutual information:
[1.28730159]

When the two variables are independent, I do however see the expected value of zero:
x = [1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1]
y = [1,1,1,1,1,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0]

print("Confusion matrix:")
print(confusion_matrix(x,y))

print("Mutual information:")
result = mutual_info_classif(pd.DataFrame(x), y)
print(result)

which gives:
Confusion matrix:
[[5 5]
 [5 5]]
Mutual information:
[0]

Why am I not seeing a value of 1 for the first case?


texts: AzureML - How to add hidden layers in neural network

texts:
Another title of this question can be : AzureML - missing "custom definition script" in "Hidden layer specification"
In AzureML modules - Two class neural network or multiclass neural network, I want to add more layers. As per documentation, by default there is one hidden layer, and we have option to specify the nodes in that hidden layer.
As per blogs, we can add hidden layers by specifying Net# script in "custom definition script" in "Hidden layer specification". Though in AzureML there is only one option - Fully Connected Case.
Am I missing something, or this option is removed by Microsoft. And ultimately how to specify number of hidden layers?


texts: Predict with some probability the day of the month paycheck received through daily transactions

texts:
I am using R to do machine learning. I have daily shopping expenditure data of individuals over a couple of months and my goal is to be able to identify through their shopping pattern the day of the month they receive their paychecks. That is, to measure with some probability the day of the month they received their paycheck. Paycheck can be received on the 1st of the month or weekly or bi-weekly. Hence, my goal is to look at their daily shopping expenditure  pattern to predict with some probability the day of of the month they receive their paycheck with some probability.

I guess this would be an unsupervised learning problem since my goal is to predict the day the salary was received and I don't have it recorded in the data.

I had the following steps in mind. But stuck on what to do next and if it is the most efficient way to go about it?

Step 1: detect local peaks of daily shopping pattern for each individual over a month.

Step 2: from the local peaks, detect the global maxima, i.e the highest peak among the local.

Step 3: record the associated day of the month the global peak was observed.

Please let me know if this seems logical. And if there are resources out there that I can use for help.

Thank you.


texts:
If I understand correctly KL-divergence is relative entropy of two distributions. To calculate KL divergence of two distributions, you would need two vectors of random variables.

What I do not understand it, how you can calculate KL divergence in VAE (latent space vector and N(0,1) as it is stated in many tutorials.

Latent space vector is not a vector of random variables. It is a vector of product of input, weights,bias and activation functions. All these do not make your vector a random variable vector. My question is, how to properly create latent space vector as random variable vector, so you could eventually calculate KL divergence.


texts: My first machine learning experiment , model not converging , tips?

texts:
I wanted to recreate the model mentioned in this paper:https://arxiv.org/pdf/1610.09204v1.pdf . I am using keras with tensorflow backend, and a gtx 1050ti.

I am an ML beginner, and thought this would be a good way to get a hands on feel for things. However, My model is not converging(loss is same as first epoch).
This is what I read from that paper:

  The first convolutional layer re- ceives an input of 56px by 56px
  images with RGB channels. It uses 32 filters of size 5×5×3, stride 1
  and then sampled with max pooling of size 2 × 2, stride 1. The second
  convolutional layer has 64 filters of size 5×5×32, stride 1 and a max
  pooling of size 2 × 2, stride 1. The results of the second max pooling
  provide the first fully-connected layer with a vector of length 12,544
  (14 × 14 × 64) which are used by 512 neurons. The final
  fully-connected output layer uses a 20-wide softmax [21] which
  represents the probability of each respective 20 class labels. This
  architecture is similar to the LeNet model [3], but with using
  rectified linear unit (ReLU) [22] activation functions instead of
  sigmoid activation functions. We also use dropout [23], a technique to
  prevent overfitting, with a keep probability of 0.5 for the
  fully-connected layers.

and my code is:

model = Sequential()
model.add(Convolution2D(32, 5, 5, border_mode='same', input_shape=(70,52, 3))) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Convolution2D(64, 5, 5, border_mode='same')) model.add(Activation("relu")) model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten()) model.add(Dense(output_dim=512)) model.add(Activation("relu"))
model.add(Dense(output_dim=2))
model.add(Activation("softmax")) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, nb_epoch=70, batch_size=500,verbose=1)

the full code can be found here : https://gist.github.com/harveyslash/5c98f9fdab0d53a2a48f477a52d8588d
I have scrapped the data from goodreads
Help appreciated !

EDIT

I forgot to actually ask what i wanted. Since its my first experiment, i would like to ask what are some things that I should do to make my model converge.


texts: Does it make sense to label a dataset manually?

texts:
I would like to implement a classifier on a dataset which does not have a label. I've written a script which labels each row of the source file by some specific values like "IF H > 45 && T <= 89 THEN label = True".

Does it make sense to do something like this? I mean if I'm able to label the dataset by the script I've wrote, why do I need the classifier? There are any arguments why it makes sense to still use a classifier for it?


texts: how to read emotion label of MSP-IMPROV dataset samples?

texts:
I need to read MSP-IMPROV dataset audio file's to conduct research. I read the help file that says 15th character of the file name, shows the emotion label of each utterance.
For example:
MSP-IMPROV-S01A-M01-P-FM01  intended Angry emotion.
But when I run the code that follows this rule, I can't reach the other emotions that are labeled O as others and WA as without agreement.
Moreover, I must extract

Anger     (792)
Happiness (2644)
Sadness   (885)
Neutral   (3477)

But I find out the number of each emotion is

Anger     (2511)
Happiness (2267)
Sadness   (2035)
Neutral   (1625)

Can anyone help me to find out where I made a mistake?
Thanks in advance.
*for more information about this database, you can find it in this link and this page


texts: Same predictors in test set but I want different outputs

texts:
I have a (training) dataset about what TV spectators are watching and for how long. The goal (at new set - the test set) is to predict for how long the TV spectators will watch a specific channel and show.

Specifically, I have the following predictors:

TV Channel (e.g. BBC, CNN etc)
Content (e.g. news, entertainment, business etc)
Starting time (e.g. 11:00, 14:00 etc)

and the following target:

End time (e.g. 12:00, 15:00 etc)

Obviously I am going to apply One Hot Encoding with the TV Channel and Content predictors and handle Starting time in a way (see more here: Encoding features like month and hour as categorial or numeric?).

However, in my training set I may have multiple observations with the same predictors' values (e.g. 'BBC', 'news', '20:00') but with different output. This is obviouly done because different users are starting to watch the same thing at the same time but they stop at different times.

Is this going to be a problem since also my test set includes observations like these?

Specifically, I do not want to receive the same output (end-time) for these observations but I want to receive different outputs which (ideally) follow the distribution of the respective observations in the training set. How can I achieve this?

Shall I simply add a new categorical variable for each user?


texts: How to incorporate new features in an existing machine learning model?

texts:
Suppose we have trained a regression model $M$ on a fixed set of $n$ features, $F_1,F_2,…,F_n$ on a particular dataset $G$. Now assume that after model training, additional features ($F_{n+1},…$) become available for a subset $H\subset G$.

What would be the best way incorporate these features to improve predictions on the subset $H$?

I can think of a few possible solutions:

Train a new model $N$ on the dataset $G$ where the new features are null in $G \setminus H$. This could be useful when $|H| \ll |G|$.
Train a new model $N$ on the dataset $H$, disregarding the old model and the (useful) training data $G \setminus H$.
Train a submodel $M'$, using $M$ as a starting point, on the new features ($F_{n+1},…$). This has the advantage of not throwing away the useful training done beforehand.

To me the last option seems like the best solution. Unfortunately, I cannot find any literature doing this sort of thing. Does that depend on the type of model? Or is it better to train a new model?


texts: Model an infrequent linear feature in TensorFlow

texts:
I am trying to predict energy usage of homes. I have a feature (home square footage) that is highly useful when available, but is often not available (in which case it is 0).

I want TensorFlow to ignore the feature if it is 0, but otherwise incorporate it in a linear model (which supports a neural net).

Is there a type of feature column that can do something like this?

Currently I am doing:

square_footage_buckets = tf.feature_column.bucketized_column(
    square_footage, boundaries=[.01, 1000, 2500, 5000, 20000])

But this makes really bad predictions for skyscrapers - it treats them the same as your local home depot (and I don't have training data for skyscrapers, so I don't want to add another bucket)

Thanks!


texts: Which packages or functions can I use in R to plot 3D data like this?

texts:
There're many data points, each of which is associated with two coordinates and a numeral value, or three coordinates. And I wish it is coloured.
I checked packages "scatterplot3d" and "plot3D" but I couldn't find one like the example I give. It is like it has a fitting surface.

My data is basically like the following. In this way I think this kind of plot is gonna be perfectly suitble for this data:

    ki,kt,Top10AverageF1Score
    360,41,0.09371256716549396
    324,41,0.09539634212851525
    360,123,0.09473510831594467
    36,164,0.09773486852645874
    ...

But I also may have one more additional variable, which makes it like:

    NeighborhoodSize,ki,kt,Top10AverageF1Score
    10,360,41,0.09371256716549396
    15,324,41,0.09539634212851525
    15,360,123,0.09473510831594467
    20,36,164,0.09773486852645874
    ...

Do you also have any good idea for visualizing the second case? What kind of plot and which packages and functions, etc.


texts: Deciding about dimensionality reduction, classification and clustering?

texts:
Could you please help me to understand it because I'm not sure if I got it correctly.

Let's say I have a dataset, of persons, with 100 features, various characteristics like height, weight, age, etc. I want to classify if are normal or abnormal. By abnormal I mean if a 20 years old man is 170cm and 150kg to identify it as abnormal.

Should I use Self Organising Map to reduce the dimensionality (these 100 features) and then K-means to classify them into normal and abnormal? Is that a correct approach? Or I can use just K-NN to classify them into normal - abnormal without any dimensionality reduction?

How many features can I use with K-NN? All the examples I've found so far use just two.

And if later I'd like to find why this person has included into the abnormal class how can I find that this happened because of these two features, his weight according to his height?


texts: Where can I learn the complete mathematics involved in LDA?

texts:
I have come across Latent Dirichlet Allocation (LDA) on multiple occasions while reading about sentiment analysis and recommender systems.

Where can I find good reading material which explains the concept in depth, especially by taking an example?


texts:
I want to read an excel file using pandas read_excel method.

But it shows an out of range error for the date value .
How can I fix this?.

df = pd.read_excel('/home/destiny2/Documents/elejio_data/Raipur/4 Star Hotels.xls')

error

return epoch + datetime.timedelta(days, seconds, 0, milliseconds)

OverflowError: date value out of range


texts:
I know about the problems that imbalanced dataset will cause when we are working on classification problems. And I know the solution for that including undersampling and oversampling.
I have to work on a Ranking problem(Ranking hotels and evaluate based on NDCG50 score this link), and the dataset is extremely imbalanced. However, the example I saw on the internet use the dataset as it is and pass it to train_test_split without oversampling/undersampling.
I am kind of confused if that is true in the Ranking problems in which the imbalanced data does not matter and we do not need to fix this before passing the data to the model?
And if that is the case why?
Thanks


texts:
I am a recent graduate.Currently I work in a late stage startup (product company) as a Software Engineer where my responsibilities are to interact with customers, understand their environment and write code so that they can use our product. I work on C#, web services and relational databases. I also work rarely on Big Data space to migrate some small components of our product to big data platform. I've got an opportunity to work in a large firm as a big data engineer to work on apache projects like yarn and hadoop and solve customer issues. And also to contribute to open source community. In curreny company, there is no one to review my work and mentor me.
Will it be a wise decision to leave a company where I've an opportunity to see product from end to end and go to a company where I'll not the same visibility of product ?

Is my next opportunity a good one if I want to make my career in big data space ?

Please share your inputs to help me take my decision ?


texts:
It's rather a strange question about feature engineering for classification problems (churn). I've read a lot of articles and tutorials for such problems, especially on telco domain. In my case the is no subscription termination, inactivity or lack of traces are used to measure performance instead. I want to detect any changes from the past with aggregated (mean) features (something I don't see often in churn analysis for telco).
My question is simple, should I build the features of interest with Incremental or Decremental methods? I borrowed this term from time series domain, hope it fits. By incremental I mean most recent data points are retained just like this:
week1-4, week3-4, week4
and by decremental we get disjoint periods:
week1-4, week5-6, week7,
I don't know if I made my self clear, hope I did, cause here it comes the second part of my question, what is better for the modelling process to adapt, high dynamics (decremental - high forgetting factor - more weight to recent data) or low dynamics (incremental - low forgetting factor - past data are retained).
Of cource Random Forests will split in any order those features but which case is considered as more computationally heavy. Intuitively I get that Incrementaly will limit recent changes in favor of the past, while Decremental will permit abrupt changes. Are there any particular model structures to accommodate these cases? Any thoughts are welcome!


texts: How to use correlation matrix when the dataset contains multiple columns with text data?

texts: How to use it with Amazon fine food reviews dataset?


texts: Rather use many linear classifiers than one complex one for numerical data?

texts:
I need to classify machine data (sampled by the second) and I need a prediction if a signal is good, medium or bad every ten seconds.
The data that I got from the machine at the moment look like this:

using different linestyles for better visibility. The performance of the machine can be fairly easy indicated by looking at the minimal value and the time it takes to reach this value.
Extending the database
In order to train an algorithm, I extended the database by first doubling it with a random smear effect and then I extended these curves that they reach the minimum later. Hence my extended database is 6 times larger then my start.

In the picture above, the second line denotes the set with which I started. The first red line denotes where something happened (the switching point was changed, so the process took less time and different minimum values were reached) which is not a major problem. I just wanted to mention it for a better understanding of the data.
To achieve one prediction every ten seconds for 200 seconds, I trained 20 models, one taking the first ten seconds, one the first twenty and so on. This worked well, after 30 seconds the confusion matrix shows at lest 94% correct predictions for every class, using a linear SVM. The earlier approaches probably cannot score well due to the data quality.
However, I also tried to make several classifiers which only looked 10, 20, 30 and 50 second into the past, no matter at which position in time. (See here for the algorithm Generate matrix from sliding window on vector) The first entry of the data always corresponded to the position of the frame. The classifiers took a long time to train and and even the 50 seconds classifier performs much worse than the 0-30 seconds classifier. I used linear SVMs here, too, and have not yet tried nonlinear methods.
My question(s)

Obviously, many small classifiers (about 20) work better than fewer complex ones, and they have the further advantage of increasing the interretability of the result. However, I would like to hear more opinions about this. What are the chances of building a complex classifier with a similar performance and what would be the best way to get there? If a complex classifier would have the same performance, would it still be less desirable?

And, to a lesser extend:

Would the complex classifier benefit from getting the position from a one-hot-encoder instead of using a number? After all, the position in which the data is at the moment is a crucuial information, however it occupies only one number in the data.

Similar questions:
Trying to predict live, but without the benefit of a starting point:
How to classify movement data (time series) in real time
Segmentation of data, but with different constraints:
Is it possible to implement a classifier according to quarters? What about missing data?
Looking at different aspects of a dataset:
How to make machine learning specifically for an individual in a group when we have the data on the group?


texts: KL divergence for exponential family distribution

texts:
In reinforcement learning, normal distribution is commonly used for continuous actions. I'm checking Pytorch's implementation for KL divergence between two normal distributions. I know it's impossible to compute KL divergence between two continuous polices, but I can't understand what the code below means and why it is using gradients. Could you please give me the related materials?
https://github.com/pytorch/pytorch/blob/master/torch/distributions/kl.py
[code]
def _kl_expfamily_expfamily(p, q):
    if not type(p) == type(q):
        raise NotImplementedError("The cross KL-divergence between different exponential families cannot \
                            be computed using Bregman divergences")
    p_nparams = [np.detach().requires_grad_() for np in p._natural_params]
    q_nparams = q._natural_params
    lg_normal = p._log_normalizer(*p_nparams)
    gradients = torch.autograd.grad(lg_normal.sum(), p_nparams, create_graph=True)
    result = q._log_normalizer(*q_nparams) - lg_normal
    for pnp, qnp, g in zip(p_nparams, q_nparams, gradients):
        term = (qnp - pnp) * g
        result -= _sum_rightmost(term, len(q.event_shape))
    return result


texts:
I am currently reading Boosting the Performance of RBF Networks with Dynamic Decay Adjustment by Michael R. Berthold and Jay Diamond (online) to understand how Dynamic Decay Adjustment (DDA; a constructive trainining algorithm for RBF networks). Doing so, I stumbled over the word prototype a couple of times:

  Unfortunately PRCE networks do not adjust the standard deviation of their prototypes individually, using only one global value for this parameter.

  [...]

  This paper introduces the Dynamic Decay Adjustment (DDA) algorithm which utilizes the constructive nature of the PRCE algorithm together with independent adaptation of each prototype's decay factor.

  [...]

  PNNs are not suitable for large databases because they commit one new prototype for each training pattern they encounter, eeffectively becoming a referential memory scheme.

I've tried to find it in the only resource they referenced in this context (D.L. Reilly, L.N. Cooper, C. Elbaum: "A Neural Model for Category Learning"), but sadly I don't have access to that one.

I found an explanation on https://chrisjmccormick.wordpress.com/2013/08/15/radial-basis-function-network-rbfn-tutorial/:

  An RBFN performs classification by measuring the input’s similarity to examples from the training set. Each RBFN neuron stores a “prototype”, which is just one of the examples from the training set. When we want to classify a new input, each neuron computes the Euclidean distance between the input and its prototype. Roughly speaking, if the input more closely resembles the class A prototypes than the class B prototypes, it is classified as class A.

So a prototype is just the parameters (center and radius, assuming Gaussians are used) of an RBFs neuron?

Rephrasing the first quoted sentence, does it mean that the RBF networks usually only learn the center and the radius is fixed?

My question is if I understood it correct. Please add a reference (which is not a random blog article) which makes it more clear.


texts:
In SVMs the polynomial kernel is defined as:

(scale * crossprod(x, y) + offset)^degree

How do the scale and offset parameters affect the model and what range should they be in? (intuitively please)

Are the scale and offset for numeric stability only (that's what it looks like to me), or do they influence prediction accuracy as well?

Can good values for scale and offset be calculated/estimated when the data is known or is a grid search required? The caret package always sets the offset to 1, but it does a grid search for scale. (Why) is an offset of 1 a good value?

Thanks

PS.: Wikipedia didn't really help my understanding:

  For degree-d polynomials, the polynomial kernel is defined as

  where x and y are vectors in the input space, i.e. vectors of features
  computed from training or test samples,  is a constant trading
  off the influence of higher-order versus lower-order terms in the
  polynomial. When , the kernel is called homogeneous.(A further
  generalized polykernel divides  by a user-specified scalar
  parameter .)

Neither did ?polydot's explanation in R's help system:

  scale: The scaling parameter of the polynomial and tangent kernel is a
  convenient way of normalizing patterns (<-!?) without the need to modify the
  data itself

  offset: The offset used in a polynomial or hyperbolic tangent kernel (<- lol thanks)


texts:
I have a data set like this. Here the first column is date, the second column is Temperature, third one is humidity, fourth and fifth column are two other boolean data. I have data of 6 years like this.

2010-01-01,25.6,59,0,1
2010-01-02,25.6,60,0,1
2010-01-03,24.2,45,1,1
2010-01-04,26.3,20,0,1
2010-01-05,26.2,17,0,1
2010-01-06,24.3,65,0,0
2010-01-07,23.1,50,0,1
2010-01-08,26.3,25,1,0
2010-01-09,26.6,23,0,1
2010-01-10,24.3,60,0,1

And the label (The variable I want to predict) for this data set is: (boolean)

0,0,0,1,1,0,0,0,1,0

Now I want to implement Machine Learning to predict from this data set for a future time frame. I have almost no knowledge on machine learning. I want to use python to do it. Which library or methodology will be best and easiest for this? And can I have a simple sample code?


texts:
I am working on a problem where I have to classify products into multiple classes (more than one) based on product descriptions. For instance:
"Tresemme shampoo and conditioner - sulfate-free" = Personal Hygiene
"Lavender-scented handwash with moisturizer" = Personal Hygiene
"Doritos Ranch flavor 18 oz mega party pack" = Snacks
"Painting and Craft kit for adults above 18" = Art and Craft
However, my training dataset is highly imbalanced. A few classes have only 10 records while there is one that has 3,000 records. 50,000 records overall.
Can anyone suggest any good techniques to deal with the imbalance in text data?
Thanks,
GD


texts: Over fitting and association with regularization

texts:
Heard and read lot about regularization helps in reducing over fitting.
But I'm not sure how exactly regularization works in reducing over fitting issue and whats the maths behind it?
Appreciate if can somebody provide a concrete example?


texts: XGBoost vs ARIMA for Time Series analysis

texts:
Doing time series analysis, I have doubts on choosing the right model. I want to predict the next 30 mins window, from the input dataset which contains the no. of error count for that particular 1 min interval.

Should I use XGBoost or ARIMA regressions?

Most of article or tutorial I found online use ARIMA for time series, while XGBoost is used more in Kaggle competitions.