texts: How would keras model learn at every epoch?

texts:
I know concept of Epochs, batch size and iteration.

let's say,

Total_data = 6400

Batch_size = 64

Iteration = 100

In this, basically we are taking in 64 data points to computer memory and calculate and at each iteration we get weight updated. so after 100 iterations, we will fulfill one epoch.

My question is, after one epoch, we are using again the same 6400 data. How is it different from the first epoch in terms of learning? does the model select different 64 data point than first epoch in second epoch and try to learn? how does it internally really work?

I wish I can get some clear answers.

Thank you people in advance.


texts: Why Continous Variable Buckets Overfitting model

texts:
I have a continuous (high cardinal discrete) variable 'numInteractionPoints' in my dataset during training model - I binned this feature in order to avoid overffing , first top bar chart is from training and second bar chart is from testing for this binned feature. Both datasets has same distribution as per following bar graph

this is how i created bin for this feature
i converted variable 'numInteractionPoints' into bins using simple bin method based on visual analysis
bins = [0, 6, 12, np.inf]
names = ['0-6', '6-12','12+']

train_data['numInteractionPointsRange'] = pd.cut(train_data['numInteractionPoints'], bins, labels=names)
train_data['numInteractionPointsRange']=train_data['numInteractionPointsRange'].astype('object')

in both cases when i include raw variable numInteractionPoints without bin and with above defined bucket the overall F1 score of model decrease on test data from 0.47 to 0.33 for a class 1. On validation dataset it is giving 0.47 while on test it gives 0.33 with or without bins

Any idea please what i am missing? This behavior is in both algo Xgboost and catboost, target class is highly imbalanced and i am controlling it Xgboost parameter scale_pos_weight. Why model overfitting due to defined buckets , as you can see above bar chart distribution of both dataset are same


texts: How to drop points from a data series for presentation?

texts:
We monitor long running industrial engines and we have data series that we want to present on a line chart on a web page. For instance, we have sensors that monitor the oil temperature and pressure on the engine.

There are several other similar data series on the components of the equipment.

The objective is to have a human operator identify deviations in engines, for post-analysis. Our chart will display 24 or 48 hours of engine operation and the operator may identify peaks in temperature or pressure, or on the other measurements.

As such, it is a large amount of data to present on the chart on the web page, and we're starting to hit limitations in several places.

At 24 hours * 3600seconds/hour * 1 data point/second = 86400 data points on the chart.

This amount of points is slowing down the rendering of the web page, and is a lot of data to transfer.

We want to reduce the count of data points presented on the chart, without losing much context. So I ask:

How can I drop data points without losing much precision?
What techniques are usually applied in this scenario?

A first (naive) thought was to group them in 5-second windows and only return one data point to represent the 5-second window on the chart;

should I average the points in the window?
should I consider the maximum in the window?

Are there other techniques than grouping data points in window, to reduce the loss of meaning for the monitoring?


texts: What is the model architectural difference between transductive GCN and inductive GraphSAGE?

texts:
Difference of the model design.

It seems the difference is that GraphSAGE sample the data.

But what is the difference in model architecture.


texts: Which network structure to use for multivariate time series data with unknown input and output size at each time step?

texts:
I have data that has is multivariate and can be thought of as many sentences of variable length. I'm trying to predict what the next sentence will be and it depends on multiple preceding sentences. I was thinking of using something like an encoder decoder neural network but I am not sure how to structure the network to have unknown input/output size at each time step
An example encoded input for consecutive time steps could be:
[ 0 , 1 , 3 , 5 ,6] t=0
[ 0 , 5 , 4]        t=1
[4 , 5 , 5 , 6]     t=2
[0]                 t=3

With example output:
[1 , 4 , 6 ]        t=0
[2 , 3 ]            t=1
[4 , 5, 6 , 7]      t=2
[2 , 4 , 6 ,7 , 8]  t=3


texts: Predictive modeling when output affects future input

texts:
Assume I have a model  which predicts the outcome of number of icecream sold in a store.
The model is trained on data for the last 5 years while keep the last year as a validation set and has produced very good results.
We now put the model into production such that the CFO can create an estimate for the upcomming year's budget. The CFO now look at the prediction for May, say 2000 ice creams, and thinks "Ooh... I was hoping for some more sale in May. I'll go 4000" thus he orders some more advertising, introduces new flavours etc. and reaches the 4000 sold ice cream at the end of May as he was hoping for.
The first of June we talk to the CFO to evalutate the model after the first 6 months, and we see that our prediction in May is off by 100%!
This spike can be explained with the increased advertising etc., and all the other days the model have done really well, but if the CFO starts tweaking the advertising, flavours etc. each day to hit the budget, how will we ever be able to test, if our model is indeed good in production/real-world? And how will we be able to re-train the model, since the first 5 years sale is without any "human influence" whereas after a year, the sale has been influenced by advertising etc., thus the spike in May is not "natural" but is due to some exogenous variable we are not able to incorporate (e.g we don't know the CFO's budget).


texts:
I have read that we make the following assumption for linear regression:
1. Linearity (correct functional form)
2. Constant error variance (homoskedasticity)
3. Independent error terms (no autocorrelation)
4. Normality of error terms
5. No multicollinearity
6. Exogeneity (no omitted variable bias)

So are these assumptions specific to Linear Regression or applicable for all types of regression techniques like Support Vector Regression, Lasso and Ridge regression, Stepwise regression etc.


texts: How to use Pytorch's Transformer module "out of the box"

texts:
I am working on implementing my first transformer, and recently I've been working in Pytorch and I see that they offer a pre-packaged transformer model. Here are the docs. I have been reading through the original paper, "Attention is all you need", as well as other papers about transformers. I have been through a couple online tutorials. Anyways, I think I am ready to start coding, however I realize that there must still be something I don't understand. Below is a screenshot from the documents. In the basic example, I need to supply a source variable and a target variable in order to produce a network output. I notice in the example these are just random matrices. What do these 2 variable correspond to in the real world?
For context I am envisioning a typical unsupervised language task where the transformer basically learns to predict the next word in the sentence. If I understand correctly this simple task allows the transformer to perform much more complex task, like writing whole documents to answering math questions (GPT-3 Paper). For this task the input is some (embedded) text and the output is the (embedded) text shifted forward by one word (if I understand correctly).
So in this Pytorch example, what exactly should the source variable be and what example is the target? Traditional in supervised learning we would just create the network output as a model of some input data matrix. It seems here there is a big difference in that I need to supply 2 data matrices as input to create an output.
Thanks!


texts:
I am building a CNN from scratch, and I am trying to change the number of convolutional layers to see what happens.
I have noticed that decreasing the number of convolutional layers increases the number of parameters.
This fact really surprised me, since I was expecting to see a decrease in the number of parameters.
Why decreasing the number of convolutional layers inside a CNN increases the number of parameters?


texts: How to perform model selection for One-Class Classification?

texts:
Consistency based model selection does not perform well for many datasets for One-Class Classification (OCC). So I am looking for some other model selection criteria.

Since, only one class of data (i.e. the positive class) is available, classification error is no longer a valid criterion. It is very challenging task to decide on the optimal parameters in OCC. Here, model selection means the optimal parameter selection for the one-class classifier.

Please suggest any additional methods except consistency based model selection method.


texts: How to define a neural model as linear or non linear

texts:
TL;DR: in what sense is the model of a neuron seen in the image above nonlinear?

In chapter 1, section 1.3 MODELS OF A NEURON of Simon Haykin's Neural Networks book, the standard model of a single neuron is described and visualised in the picture above.

Haykin states that this model, which consists of a set of inputs x1..xm, their corresponding weights w1..wm, a linear combiner that sums the weighted inputs and the bias (b) and an activation function that takes that sum and produces the output, is nonlinear. So, my question is, isn't the output linearly dependent on the input? For example, if the neuron only takes one input, x1, then the linear combiner takes the form v = x1 + b and the activation function is φ(v). So, the only way that I can see this model being nonlinear is if the activation function is nonlinear. But there are clearly cases where the activation function is linear (like the piecewise-linear function described in that same section of the book). So how can this model be inherently nonlinear?

I realise that this isn't a major concern, but I'd like to understand every part of the book before moving on, and this has been bugging me since I saw it.

Thanks to everyone in advance for your answers.


texts:
I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's  sklearn.feature_extraction.text.TfidfVectorizer. The issue however, is that the actual strings contain very few words. Exactly 1.8 in average. Here's a sample:

 print(df.keyword)

0 fre lifeproof
1 car stereo
2 analog clock
3 refrigerator

So my question is,

is the TfidfVectorizer also suited for such case? Or will such sparse resulting matrix not be benefiting the model?
Is there a more suited approach for such case?


texts: Train classifier on balanced dataset and apply on imbalanced dataset?

texts:
I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works best with F-Score ~0.83.

Now, I have to apply the trained classifier on an unlabelled dataset DS2 with ~ 5 million entries (and same features). However, for DS2, the target distribution is expected to be highly unbalanced.

Is this a problem? Will the model reproduce the trained target distribution from DS1 when applied on DS2?

If yes, would another algorithm be more robust?


texts: Difference between validation and prediction

texts:
As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction.
To keep it simple, I will refer to train, val and test:

Training data: Train model, especially find hyperparameters through GridSearchCV or similar
Validation data: Validate these hyperparameters on "new" data?
Test data: Make prediction on unseen data

My status so far:

Split data: 60 % Training - 20 % Validation - 20 % Test
Find hyperparameters on training data
Fit again with best parameters on training data by using .fit(X_train, y_train, validation_data(X_val, y_val)).
Check model on unseen data through .predict() or .evaluate().

Is this correct? Though using GridSearchCV do I have to split train manually into training and validation?


texts: isnull().sum() returning incorrect figure

texts:
I am doing a prediction using RandomForestRegressor.
Below is the code for test data:
    X = test_set.drop(['C_Nxt10DMx'], axis=1) # drop labels for training set
    Y = test_set['C_Nxt10DMx'].copy()
    test_prepared     = full_pipeline.transform(X)
    final_predictions = forest_reg.predict(test_prepared)
    final_mse = mean_squared_error(Y, final_predictions)
    final_rmse = np.sqrt(final_mse)
    (forest_rmse*100)/Y.mean()

I am printing Y.isnull().sum() and it is returning a 0 value. However, when I trying to bind the value in a data frame, it is returning an incorrect figure.
df =  pd.DataFrame()
df['pred'] = final_predictions
df['Label'] = Y
print(Y.isnull().sum())
print(df['Label'].isnull().sum())

# 7646


texts: Hyperparameter search for LSTM-RNN using Keras (Python)

texts:
From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge."

So this is more a general question about tuning the hyperparameters of a LSTM-RNN on Keras. I would like to know about an approach to finding the best parameters for your RNN.

I began with the IMDB example on Keras' Github.

the main model looks like this:

(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features,
                                                      test_split=0.2)

max_features = 20000
maxlen = 100  # cut texts after this number of words (among top max_features most common words)
batch_size = 32

model = Sequential()
model.add(Embedding(max_features, 128, input_length=maxlen))
model.add(LSTM(128))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          class_mode="binary")

print("Train...")
model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=3,
      validation_data=(X_test, y_test), show_accuracy=True)
score, acc = model.evaluate(X_test, y_test,
                        batch_size=batch_size,
                        show_accuracy=True)

print('Test accuracy:', acc)
Test accuracy:81.54321846

81.5 is a fair score and more importantly it means that the model, even though not fully optimized, it works.

My data is Time Series and the task is binary prediction, the same as the example. And now my problem looks like this:

#Training Data
train = genfromtxt(os.getcwd() + "/Data/trainMatrix.csv", delimiter=',', skip_header=1)
validation = genfromtxt(os.getcwd() + "/Data/validationMatrix.csv", delimiter=',', skip_header=1)

#Targets
miniTrainTargets = [int(x) for x in genfromtxt(os.getcwd() + "/Data/trainTarget.csv", delimiter=',', skip_header=1)]
validationTargets = [int(x) for x in genfromtxt(os.getcwd() + "/Data/validationTarget.csv", delimiter=',', skip_header=1)]

#LSTM
model = Sequential()
model.add(Embedding(train.shape[0], 64, input_length=train.shape[1]))
model.add(LSTM(64))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          class_mode="binary")

model.fit(train, miniTrainTargets, batch_size=batch_size, nb_epoch=5,
      validation_data=(validation, validationTargets), show_accuracy=True)
valid_preds = model.predict_proba(validation, verbose=0)
roc = metrics.roc_auc_score(validationTargets, valid_preds)
print("ROC:", roc)
ROC:0.5006526

The model is basically the same as the IMDB one. Though the result means it's not learning anything. However, when I use a vanilla MLP-NN I don't have the same problem, the model learns and the score increases. I tried increasing the number of epochs and increasing-decreasing the number of LTSM units but the score won't increase.

So I would like to know a standard approach to tuning the network because in theory the algorithm should perform better than a multilayer perceptron network specially for this time series data.


texts: Why rejection of a true null hypothesis is called type I error?

texts:
I’m comparing two confusion matrices:

https://en.wikipedia.org/wiki/Confusion_matrix#Table_of_confusion
https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

The 2nd is rotated, the Decision is on Y-axis. But I assume both reflect the same concept.
I have two options to render the word “Reject”.
(1) When we look at Null hypothesis matrix, the Reject of a “True Null hypothesis” means a decision which doesn’t reflect reality (convicting an innocent), and this is indeed FP (type I).
(2) Following Confusion_matrix wiki, I interpret Reject as False. Therefore, making a False decision (H0 is false) over Actual True (H0 is true) brings me to claim this is FN (type II).


texts: Proper loss function for sequence prediction model with multi-step output

texts:
Consider a typical time series (sequence) prediction problem that use previous $k$ step historical features to predict the next step target. We use RNN model as an example.

As shown in the picture, {$x_0$,...,$x_k$} is the sequence of historical features used to predict the target $y_{k+1}$ at step $k+1$. Typical method uses only the final output $\hat{y}_{k+1}$ for training and the corresponding loss will be, for example, mse as $||y_{k+1}-\hat{y}_{k+1}||_2$. Thus, I have following few questions.

Q1: Though {$y_1,...,y_k$} is known at time step $k$, will it be better to use last multiple (even all) steps of outputs for prediction than using only the last one $\hat{y}_k$ ? For example, using mse for multiple steps outputs as

$$loss=\frac{1}{N+1}\cdot\sum_{n=0}^{N} ||\hat{y}_{k+1-n}-y_{k+1-n}||_2 $$

Q2: How to choose a proper loss function for multiple outputs ? It seems the multi-step mse is not a good choice when the rank between $y$ and $\hat{y}$ is  prefered than absolute deviation.


texts: Looking for classifcation model that optimises weighted accuracy. 'Modified decision tree classifier' or something else?

texts:
I have a large list of items that represent products sold by some company. For each of these items, I have their revenue and a number of (numerical) features. I have fitted a number of demand prediction models on each of these items, and computed accuracy scores for each model for each item. An example table with made up data is given below:

I would like to find a model or algorithm that, using item features (x1, x2, ...), can determine/predict the best model for individual items, such that the 'overal accuracy' is optimized. With 'overal accuracy' I mean the weighted sum of the accuracies of selected models using revenue.
The first thing that came to mind was using a decision tree classifier. However, this particular model tries to maximise how often the 'right model' is chosen, rather than looking at how good or bad the chosen model is. For example, for 'item 1', model 1 has the highest accuracy with 81%. While model 2 is thus not the 'best' model, it is still 'pretty good' with 80% accuracy. Model 3 on the other hand performs much worse with only 50%. Thus I would like to 'punish' misclassifying model 3 harsher than misclassifying model 2.
Additionally, I would like to take into account the revenue of the different items. For example, classifying item 1 correctly would be more important than classifying item 4 correctly (500 and 1 revenue respectively). I suppose this aspect is easier to handle, using weighted sampling.
Does any model/algorithm already exist that I could apply to solve this problem?


texts: Techniques for dealing with unevenly spaced time series data that have missing time-stamps?

texts:
What are the techniques to infer missing time-stamps in the unevenly spaced time series data that has missing time-stamps?

The frequency of data recording is about 1-3 times a day without any fixed time of measurement. In some cases the time stamp of the measurement is unavailable.

It is possible to predict at which time $t_0$ the new sample $x_0$ was recorded knowing the history of previous measurements $(t_n,x_n)$?

$$
(t_{-n},x_{-n}), ... ,(t_{-3},x_{-3}), (t_{-2},x_{-2}), (t_{-1},x_{-1}), (t_0,x_0)
$$

Or predict multiple missing time-stamps e.g. $t_{-2},t_{-6},t_{-7}$ from recorded sequence $t_n,x_n$?


texts: Does it make sense to use Transformer encoders on top of a pretrained Word2Vec embedding for a classification task?

texts:
As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch.
I was thinking of using an existing Word2Vec embedding and placing a stack of Transformer encoder layers on top of it, with a final Dense layer for the classification.
From my limited understanding of the Attention paper, this seems like it should work, but my model ends up performing very poorly (predicting all 0). Before I start looking for errors in my code, is there a conceptual reason for this architecture to fail?

Edited to add:
I am performing multilabel classification, where each label has two classes. The data I have is from legal contracts. Each sample is a paragraph, and the different labels correspond to the presence or absence of legal concepts. I have a Word2Vec model trained on legal text, and the embedding is not part of the model - I tokenize, normalize and vectorize as part of the generator.


texts: Text categorization: combining different kind of features

texts:
The problem I am tackling is categorizing short texts into multiple classes. My current approach is to use tf-idf weighted term frequencies and learn a simple linear classifier (logistic regression). This works reasonably well (around 90% macro F-1 on test set, nearly 100% on training set). A big problem are unseen words/n-grams.

I am trying to improve the classifier by adding other features, e.g. a fixed sized vector computed using distributional similarities (as computed by word2vec) or other categorical features of the examples. My idea was to just add the features to the sparse input features from the bag of words. However, this results in worse performance on the test and training set. The additional features by themselves give about 80% F-1 on the test set, so they aren't garbage. Scaling the features didn't help as well. My current thinking is that these kind of features don't mix well with the (sparse) bag of words features.

So the question is: assuming the additional features provide additional information, what is the best way to incorporate them? Could training separate classifiers and combining them in some kind of ensemble work (this would probably have the drawback that no interaction between the features of the different classifiers could be captured)? Are there other more complex models I should consider?


texts: Keras Softmax is Hardmaxing for some reason

texts:
I am new to Keras and am bit confused at the moment:

def get_compiled_model():
    model = tf.keras.Sequential([
    tf.keras.layers.Dense(1000, input_shape = (1000,), activation='relu', kernel_initializer = 'glorot_uniform'),
    tf.keras.layers.Dense(1000, activation='relu', kernel_initializer = 'glorot_uniform'),
    tf.keras.layers.Dense(41, activation='softmax', kernel_initializer = 'glorot_uniform')
])

    model.compile(optimizer='SGD', loss='categorical_crossentropy', metrics=['accuracy'])
    return model

I then call my model as follows:

model = get_compiled_model()
for i in range(10):
     model.fit(train_object, epochs=10)
     test_loss, test_acc = model.evaluate(test_object, verbose=2)

I keep getting 0 accuracy even after a lot of training. I think it is because the model is hardmaxing from the start:

for row in test_object.take(1):
    row
print(model.predict(row[0])[0])

array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0.], dtype=float32)

This behavior happens even at the beginning which is confusing since we would expect something with decimals rather than all 0's and 1's.

Any help would be appreciated. To rephrase the question I am confused to why the model is hardmaxing instead of softmazing.


texts: Using strong predictor in Model training?

texts:
I am trying to build a Disease predictor based on symptoms. I am using data scraped from Symcat website. After sampling the data we have symptoms to disease mapped for training purpose.Data looks like these: X --> S1,S2,S4... Y --> D1,D2,D5. The problem is some symptoms are strong predictors for a particular disease. Say S1 is a strong predictor for D1. So basically model should predict higher probability score for D1 compared to others due to presence of S1. How to imbibe the information that S1 is strong predictor for D1 so that it can be used in model training

Tried oversampling those strong predictor to disease combination. But dont think that is right as I have a multi label data set. results are not reflecting either
Dataset looks like this:
![Symptoms     D1  D2  D3
S1,S2,S4    0   1   1
S3,S4,S5    1   0   0
S1,S4,S6    1   0   1]1
Symptoms are features. We are using symptoms definition embeddings. Y is 1 hot vector.


texts: Scalable training/updating of many small LSTM models

texts:
My situation is that I have many thousands of devices which each have their own specific LSTM model for anomaly prediction. These devices behave wildly differently so I don't think there is any way to have a shared global model, unfortunately. Periodically I will update each device model with the new data from the device - so maybe once per day I will load an additional daily batch of readings and use the properties of stateful LSTM training to update the model.

Each of these models is very small, containing at most 10-20 thousand readings in their whole history, and fitting into memory on even a modest GPU. Training from scratch is relatively quick, and updating a single batch should be even more so, as typically there are only 48 new readings/day (30 minute intervals), but going as high as 1440 readings/day.

My question is - what is an optimal architecture for handling all of these models and updates? They are all independent and small, so I don't see a need for anything distributed, but there are so many of them that I am unsure how to proceed. I am thinking that perhaps using something like an AWS 'large' GPU cluster, with each GPU loading a serialized model from the DB, updating with a new batch of data, and writing back could work - however, I suspect that having to repeatedly compile the models will be a blocker and that being able to do only 4 at a time is woefully insufficient.

Alternatively, it may be possible to do the 'initial' training on some kind of powerful GPU rig, and then do the batch updates on CPU's, which would make it easier to run many instances in parallel. Is this an example of where Spark might be useful?

Any advice is appreciated - including 'your premise is absurd, why would you do this!' This is only one of many ways available to do the anomaly detection, so if the architecture implementation is unfeasible, I will abandon the LSTM's altogether and try something simpler.


texts: How to define confusion matrix for classification?

texts:
Below is the dataset where the response variable is play with two labels (yes, and no):

No. outlook temperature humidity    windy   play
1   sunny       hot     high        FALSE   no
2   sunny       hot     high        TRUE    no
3   overcast    hot     high        FALSE   yes
4   rainy       mild    high        FALSE   yes
5   rainy       cool    normal      FALSE   yes
6   rainy       cool    normal      TRUE    no
7   overcast    cool    normal      TRUE    yes
8   sunny       mild    high        FALSE   no
9   sunny       cool    normal      FALSE   yes
10  rainy       mild    normal      FALSE   yes
11  sunny       mild    normal      TRUE    yes
12  overcast    mild    high        TRUE    yes
13  overcast    hot     normal      FALSE   yes
14  rainy       mild    high        TRUE    no

Here are the decisions with their respective classifications:

1: (outlook,overcast) -> (play,yes)
[Support=0.29 , Confidence=1.00 , Correctly Classify= 3, 7, 12, 13]

2: (humidity,normal), (windy,FALSE) -> (play,yes)
[Support=0.29 , Confidence=1.00 , Correctly Classify= 5, 9, 10]

3: (outlook,sunny), (humidity,high) -> (play,no)
[Support=0.21 , Confidence=1.00 , Correctly Classify= 1, 2, 8]

4: (outlook,rainy), (windy,FALSE) -> (play,yes)
[Support=0.21 , Confidence=1.00 , Correctly Classify= 4]

5: (outlook,sunny), (humidity,normal) -> (play,yes)
[Support=0.14 , Confidence=1.00 , Correctly Classify= 11]

6: (outlook,rainy), (windy,TRUE) -> (play,no)
[Support=0.14 , Confidence=1.00 , Correctly Classify= 6, 14]


texts: Why is the idf important in tf-idf when it seems to just re-scale your features?

texts:
I am trying to understand why tf-idf is useful. As I understand the formula to work out the tf-idf is:

Can someone explain what is wrong with the reasoning below:
Imagine I have 100 documents that have a text corpus of 10,000 words. As I understand each document will have 10,000 features once transformed using tf-idf. For a single feature we get the tf part by counting the number of occurrences of that word in each document. We then multiply that feature by the idf.
TL;DR
My question is given that we multiply the entire feature by a constant (the value of log(N/dfi)), how does that help in making a model better. As I understand multiplying a feature by a constant doesn't help as the model can figure out this new scale.
Edit
After applying tf we might get a document-term matrix like the following:
word /
doc num. computer walk smell help warmth
1        1        0    2     1    0
2        0        2    3     1    1
3        0        1    0     0    3
4        1        2    0     1    0
5        0        1    2     2    1

Then after tf-idf we might get the following:
word /
doc num. computer    walk        smell      help        warmth
1        1*log(5/2)  0*log(5/4)  2*log(5/3) 1*log(5/4)  0*log(5/3)
2        0*log(5/2)  2*log(5/4)  3*log(5/3) 1*log(5/4)  1*log(5/3)
3        0*log(5/2)  1*log(5/4)  0*log(5/3) 0*log(5/4)  3*log(5/3)
4        1*log(5/2)  2*log(5/4)  0*log(5/3) 1*log(5/4)  0*log(5/3)
5        0*log(5/2)  1*log(5/4)  2*log(5/3) 2*log(5/4)  1*log(5/3)

tf-idf seems to just scale each feature here by the same value so why does that help certain models like decision trees that can handle different scaling to predict better?


texts:
I have a data set of questions belonging to ten different categories namely (definitions, factoids, abbreviations, fill in the blanks, verbs, numerals, dates, puzzle, etymology and category relation).

The categories are briefly described as follows:

Definition – A question that contains a definition of the answer.
Category Relation – The answer has a semantic relation to the question where the relation is specified in the category.
FITB – These are generic fill in the blank questions – some of them ask for the completion of a phrase.
Abbreviation – The answer is an expansion of an abbreviation in the question.
Puzzle – These require derivation or synthesis for the answer.
Etymology – The answer is an English word derived from a foreign word.
Verb – The answer is a verb.
Number – The answer is a numeral.
Date – The question asks for a date or a year.
Factoid – A question is a factoid if its answer can be found on Wikipedia.

I used the Stanford core NLP package called shiftreducer to find out the Part-Of-Speech (POS) values for each question in a category. I thought of using this POS pattern as a discriminant among the classes but it turned out to be generalized since:

All the classes follow a similar pattern
Nouns top the POS count followed by Determinants, Prepositions, Adjectives, Plural nouns and finally verbs.

What could be the other ways in which I could differentiate among the question categories? Or as my question was in its first place, "What kind of features do I select for efficient categorization?"


texts:
I have data with 95 numeric variables and 5 categorical variables. My Y has 2 values. I built a decision tree and my accuracy was 81.8%. Then I created 3 new variables as follows. It improved accuracy to 84.3%

Normalize numeric variables and for training data, find mean vector for Y=1 and Y=0
for each data     point, find euclidean distance from each mean vector - distance0 and distance1
third variable will be 0 if distance0 is <= distance1

I was wondering if there is any other new variables that i could create to improve the accuracy

I used a decision tree as it is fast to build and gives me indication whether a newly created variable is useful or not.

Please let me know if you have any thoughts


texts: Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed

texts:
I'm running an LM model using the LMest package available in R. The dataset contains NO missing values.

pct_miss(df_long)
[1] 0
n_miss(df_long)
[1] 0

The lmest function with no covariates works fine. However, when I added covariates in the latentFormula, I got the following error message.
"Error in lmest(responsesFormula = responseA + responseB + responseC + responseD ~  :
missing data in the covariates affecting the initial probabilities are not allowed"
My code follows:
LMmodel <- lmest(responsesFormula = responseA + responseB + responseC + responseD ~ NULL,

          latentFormula =  ~

              SEX + GRADE | SEX + GRADE,

          index = c("ID","Time"),

          data = df_long,

          k = 3,

          paramLatent = "multilogit",

          start = 0)

where the four response variables (responses A to D) are categorical, ranging from 0 to 8; SEX is binary (1 and 2 represent Male and Female, respectively); GRADE is categorical, ranging from grades 7 to 12.
Can anyone point out what went wrong and how to address this error?
Thanks so much.


texts: Convert non-binary value into binary value?

texts:
I'm kind of a newbie in ML

I have a dataset like:

282,4,1,1,0,0,1,1,41,0,1,1,1,0,0,0,0,0,0,0,0,1,0,8,4
282,5,1,1,0,1,1,2,41,0,1,1,1,0,0,0,0,0,0,3,1,2,0,12,4
161,5,2,1,0,0,1,1,41,0,1,0,0,0,0,0,0,0,0,3,1,4,0,3,5
282,4,2,1,0,32,1,3,41,0,1,0,0,0,0,0,0,0,0,0,0,1,0,4,1
0,3,1,1,0,32,1,1,32,1,1,1,1,0,0,0,0,0,0,0,0,4,0,8,4
0,4,2,5,0,0,1,1,32,0,1,1,1,0,0,0,0,0,0,0,0,1,0,2,5
....

With output is the last column, it has 6 outputs {A,B,C,D,E,F} reference to {0,1,2,3,4,5}. I'm using NN and SVM(C-SVC, NuSVC) for training this dataset, but the accuracy still low (>60%).

I want to try something else to improve the performance of current results so I plan to train 6 binary models like {A, not_A}, {B, not_B}...

Should I only change the output into binary and keep all features for training?

Or I need to change all features into binary too? If yes, can you give a short explanation for me?

Thanks for all your help!


texts: Advance Methods of Understanding Significance of Customer Behaviors

texts:
I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways are to measure the significance of the change.

I have 2 scenarios:

I changed one of my website's background from black to orange, this actually increased the site traffic by 17%.
I moved the check out button to a more obvious position on a different site, this specific site unfortunately experienced decreased traffic the day after the implementation by 8%.

I have three questions:

I know the somewhat traditional method of measuring significance, basically run a t-test (look up the z-score in a table and see the probability of this happening by chance, and if its <5% then the change is significant, etc). But is there a more advanced (data-science-y) way of measuring the significance of the change? If it's too much to explain, pointing me to a general direction to look is greatly appreciated!
Speaking of a more data-science-y approach, I've been thinking about triple-exponential smoothing. I can break the day down by hour and build a model of usage by hour, then I can compare the actual hourly usages to the predicted usages and compare the two. Is this a more sophisticated way to go about this or am I just being dumb?
Right now I'm testing out features on different sites in order to minimized interference of the new features I'm implementing. If I implement both these changes to the same site, what are some of the ways I can separate and measure the significance of these two features (kind of similar to partial correlations where I take control of certain variables).

Thanks for reading! Please let me know your thoughts. Any advice would be greatly appreciated.


texts:
Can Transformer Models be used for Training Chatbots?
Note  - I am talking about the transformer model google released on the paper 'Attention is all you need'


texts: How do I replace NaN values using group by pivot_table in pandas DataFrame?

texts:
I am working on a machine learning practice problem, from https://datahack.analyticsvidhya.com/contest/practice-problem-big-mart-sales-iii/#ProblemStatement
I want to replace the null values in the column 'Item_Weight' and for that I am using the mean values given by a pivot_table where I calculated the mean of 'Item_Weight' and grouping the mean by column 'Item_Identifier' of the dataset.
item_weight_mean = ds.pivot_table(values='Item_Weight',columns='Item_Identifier')
loc2 = ds['Item_Weight'].isnull()
ds.loc[loc2, 'Item_Weight'] = ds.loc[loc2, 'Item_Identifier'].apply(lambda x: item_weight_mean[x])

I am getting an error for the same code.
(key)
-> 2902             indexer = self.columns.get_loc(key)
   2903             if is_integer(indexer):
   2904                 indexer = [indexer]

D:\Important Applications\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2895                 return self._engine.get_loc(casted_key)
   2896             except KeyError as err:
-> 2897                 raise KeyError(key) from err
   2898
   2899         if tolerance is not None:

KeyError: 'FDN52'

How do I remove this error?


texts: Maximum likelihood Estimation of three-parameter log-logistic distribution in R

texts:
I am trying to estimate parameters from a three-parameter Log-logistic distribution in R. I have a code as follows:

nloglike <- function(p,y) {
 n <- length(y)
 alpha <- p[1];   beta <-p[2];gamma <-p[3];
if (alpha > 0 && y > gamma && beta >= 1)
{logl <- sum(log((beta/alpha)*((y-gamma)/alpha)^(beta - 1))-log(1+((y-
 gamma)/alpha)^beta))^2}
 else { logl <- 10000 }
return((logl))}

yy<-exp(data[,2])
sol<-optim(p=c(1,1,0),nloglike,method ="BFGS",y=yy,control=list(trace=0))$par
print(sol)

The code does not throw any errors, but the output is far different from what's expected. I would like to know what to consider when choosing the initial / starting values from the optim command. i.e. p=c(1,1,0)


texts: How to find the average and median values of the column 'Total Traded Quantity' for each of the types of 'Trend'

texts:
df_trend = df.groupby(["Trend"])
average_trades = df_trend["Total Traded Quantity"].mean()
print("The average traded quantity is: ", average_trades)

>>The average traded quantity is:  Trend
Slight or No Change    1.635627e+06
Name: Total Traded Quantity, dtype: float64

But I have 9 types of trends in Trend column, although I get only one type here in the output. How shall I proceed?


texts: Multiple Categorical values for a single feature how to convert them to binary using python

texts:
I have a data set of movies which has 28 columns. One of them is genres. For each row in this data set, the value for column genres is of the form "Action|Animation|Comedy|Family|Fantasy". I want to encode them using pandas.get_dummies() but since the columns have multiple values, how to deal with such conditions?
      Additinal information on below link(question moved from stackoverflow)
https://stackoverflow.com/q/40331558/4028904


texts: The automatic construction of new features from raw data

texts:
Some observations are far too voluminous in their raw state to be modeled by predictive modeling algorithms directly.

Common examples include image, audio, and textual data, but could just as easily include tabular data with millions of attributes.

Feature extraction is a process of automatically reducing the dimensionality of these types of observations into a much smaller set that can be modelled.

For tabular data, this might include projection methods like Principal Component Analysis and unsupervised clustering methods. For image data, this might include line or edge detection. Depending on the domain, image, video and audio observations lend themselves to many of the same types of DSP methods.

What about generating new features with higher predictive values from the raw data and concatenate them to the raw data?

For example I have data about student wealth, health, family status and I want somehow generate a new feature I can call Social status whic is generated from the raw data and with high predictive value? Is this possible? linear regression can be a good way I need to discover?


texts: How to construct validation set for time series for NN?

texts:
I would like to train my model with a validation set. As the data is a time series I have to use timeseriessplit:

import numpy as np
from sklearn.model_selection import TimeSeriesSplit
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=5)
print(tscv)
TimeSeriesSplit(max_train_size=None, n_splits=5)
for train_index, test_index in tscv.split(X):
...    print("TRAIN:", train_index, "TEST:", test_index)
...    X_train, X_test = X[train_index], X[test_index]
...    y_train, y_test = y[train_index], y[test_index]
TRAIN: [0] TEST: [1]
TRAIN: [0 1] TEST: [2]
TRAIN: [0 1 2] TEST: [3]
TRAIN: [0 1 2 3] TEST: [4]
TRAIN: [0 1 2 3 4] TEST: [5]

Using this method, I obtain a train and test set.
But how can I generate a validation set now?


texts: Can we fine-tune a model on the same dataset which it is pretrained on?

texts:
So I was reading this paper (about a use case of pretraining then self-training) which got me thinking - suppose I pre-train a model on a particular dataset, then fine-tune it again on the same dataset.
Theoretically, if we pre-train it as a masked LM and fine-tune, it might lead to overfitting - but I am not sure. Maybe it won't be able to generalize well, but would still give an accuracy boost?
Does anyone know some research or other credible sources to explain why or why not this should be done?
Also, if this is indeed possible/advised do we have to take some certain extra steps in Tensorflow,  Keras, or HuggingFace (basically any Deep-Learning Framework or library) for doing so?
EDIT:- A simple example, I am saying like if we have a specific supervised task, and pre-train on the same features as we would fine-tune on - would that increase accuracy by any bit?


texts: Cost/loss functions for multi-tasking regression neural networks

texts:
The mean square loss function is the standard for regression neural networks. However, if I have a neural network learning two tasks (two outputs) at once, is it more advisable to train on the sum of the relative errors for the different outputs or the sum of the mean square errors of both tasks?

Intuitively, the mean square loss function favours the task with larger expected values so it's "unfair" to the other task. I'm leaning towards using relative error but are there any caveats to using it as the loss function?

Update: Problem with using relative error is that it's a percentage and therefore the gradient updates will be very small. I think it may be more practical to use an error function proportional to the mean relative errors but scaled so that they are large enough.


texts: Contributions of each feature in classification?

texts:
I have some features and I am using Weka to classify my instances.

For example I have:

Number of adj   number of adverb    number of punctuation

in my feature set. However, I would like to know the contribution of each feature in the feature set. So what metrics or parameters are helpful to get the contribution of features?


texts: Why is a 2D->2D tSNE transformation not idempotent (up to random influences)?

texts:
My understanding of tSNE is that probability distributions are optimized in a way to maintain Euclidean (by default) distances (on average) when transforming from the input to the output space. If that is the case, when transforming from 2D to another 2D space, wouldn't the identity transformation be the ideal tSNE? I know that tSNE transformation is a somewhat random process, but I'd expect distances to be maintained generally.
By contrast, when applying a tSNE to the following data, the result has a very different distribution of distances. Note in particular the distribution of red dots:
Input

Output

This example has been produced using
from sklearn.manifold import TSNE
tsne = TSNE(random_state=1).fit_transform(data)

compare https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html


texts: How does Google's 'showing results for' work?

texts: If I search 'I love to eate my food' on Google then Google will 'show results for' I love to eat my food.... How does this algorithm work?


texts: how can i implement a loss function with percentile cutting?

texts:
I want a loss function which only care about the pecentile head and tail.
for example:
I have a rmse target, but i only care about rmse which y_pred > y_pred.percentile(90) and y_pred < y_pred.percetile(10), in the middle of y_pred, i dont want to consider them in my loss.
here is my code:
class myloss(keras.losses.Loss):
  def __init__(self, coef, name='myloss'):
    super().__init__(name=name)
    self.coef = coef

  def call(self, y, y_pred):
    ## TODO: i want to sum over y_pred > y_pred.percentile(90) and y_pred< y_pred.percentile(10)!!!
    # return tf.math.reduce_mean(tf.square(y * self.coef- y_pred), axis=1)

from util import *
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow as tf
if __name__ == '__main__':
  df = read_cache_data('cache')
  df = df.dropna()
  print(df.shape)
  fv_cols = df.columns[21:-3]
  res_col = 'res_10'
  #model = keras.Sequential([layers.Dense(50, input_shape=(len(fv_cols), ), activation='relu'), layers.Dense(30, activation='relu'), layers.Dense(1)])
  model = keras.Sequential([layers.Dense(1, input_shape=(len(fv_cols), ))])#, activation='relu')])
  model.compile(optimizer=keras.optimizers.Adam(), loss = myloss('loss'))#keras.losses.mean_squared_error)
  model.summary()
  model.fit(df[fv_cols].values, df[res_col], batch_size=288)


texts: Why is "next state" kept in RL experience replay?

texts:
Following this explanation on what is experience replay (and others), I noticed an experience element is defined as

$e_t = (s_t,a_t,r_t,s_{t+1})$

My question is, why do we need the next state in the experience?

To my understanding, our networks learn state to action and action to reward mappings, so I fail to see where the "next state" is used in experience replay?


texts: How are the positions of the output nodes determined in the Kohonen - Self Organizing Maps algorithm?

texts:
In the Cooperative stage of Kohonen's SOM, the neighborhood for a winning neuron(output node). In most cases, the neighborhood function happens to be the Gaussian Function.
For example,
$$h_j,_i = exp(-d_j,_i^2/2*\sigma^2)$$
where $h_j,_i$ is the window function ($i$ is the index of the winning neuron and $j$ is the index of the encompassing neuron), $d_j,_i$ is the lateral distance between the winning neuron $i$ and the excited neuron $j$.

Now the user who is going to implement the SOM has the input data and the randomly initialized weight vectors. How is $d_j,_i$ determined? Is it randomly initialized too? Also what is the relation between the weight vectors of the output nodes and their positions?
The figure below is just to give a visual representation of the organization of the input and the output nodes


texts: I don't understand RidgeCV's fit_intercept, and how to use it for my data

texts:
Alright, I have an assignment that makes me calculate weights for a function with different terms. At first, I thought I might just leave the weight for the term $1$ out, and instead use the intercept. I have decided to use RidgeCV, as I had a large amount of multicolinearity.

However, now I have appended my $x$ by a row of $1$'s, and did the following:

RidgeCV(fit_intercept = False).fit(x, y)

Alright, now I have an array of weights. For comparison, I have tried also

RidgeCV(fit_intercept = True).fit(x, y)

As expected, the weight for $1$ became 0. However, all other values changed too - and the intercept is different from the weight for $1$ from before. I have also a different .score() - the first one is higher.

Why is that the case? I thought all fit_intercept was doing is adding a row of $1$ to my $x$, which obviously can't be true. Also, should I try centering my data myself, or is that now unnecessary?


texts: Relationship between VC dimension and degrees of freedom

texts:
I'm studying machine learning and I feel there is a strong relationship between the concept of VC dimension and the more classical (statistical) concept of degrees of freedom.

Can anyone explain such a connection?


texts: Retraining Neural Network with Optimal Number of Layers/Neurons

texts:
So let's say you have determined the optimal number of layers/neurons for your neural network. When it comes time to re-train your network with new data values (the inputs are the same, just newer data for the inputs), would the optimal number of layers/neurons need to be determined again, or would the previous numbers be sufficient for re-training?

I do not have formal training in neural networks and I don't have a background in statistics, but from the answers in the post How to set the number of neurons and layers in neural networks I get the strong indication layers/neurons describe the complexity of the relationships between the inputs/outputs and I can surmise that as long as those relationships hold in the new data, determining the optimal layers/neurons is not needed before re-training, but I'd like some more experienced opinions or facts.


texts:
There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python):

HBase Thrift API
HBase Stargate (REST API)

Does anyone know which one should be used in which circumstances?
I.e. what are their main differences, and pros/cons?


texts: Outlier detection on categorical network log data

texts:
I am working with a completely categorical network log data that consists of source ip address, destination ip address, source port, destination port, protocol.

Data Preprocessing performed : Converted IP addresses to integers. Removed rows with NA's. Dis not perform scaling as it doesn't make any sense with categorical data and also since I'm computing Hamming distance.

Modelling and Implementation : I have implemented 'Hamming Distance' to compute the distance matrix for the set. I have tried converting the categorical variables to numerical by converting them to factors and perform Kmeans on the data, however the accuracy I'm getting is very low as expected. I have used KModes as well and I have got cluster labels to the data, I'm having challenge proceeding further from here.

Question : Once the distance matrix is computed, how can I go about detecting the outliers with such data type? I really appreciate any recommendations. Thanks in advance

Data Snippet :  2887562076 2344654028        6   41940      80
                2344654028 2887562076        6      80   41940
                2344654028 2887562076        6      80   41940


texts: Data visualization for pattern analysis (language-independent, but R preferred)

texts:
I want to plot the bytes from a disk image in order to understand a pattern in them. This is mainly an academic task, since I'm almost sure this pattern was created by a disk testing program, but I'd like to reverse-engineer it anyway.

I already know that the pattern is aligned, with a periodicity of 256 characters.

I can envision two ways of visualizing this information: either a 16x16 plane viewed through time (3 dimensions), where each pixel's color is the ASCII code for the character, or a 256 pixel line for each period (2 dimensions).

This is a snapshot of the pattern (you can see more than one), seen through xxd (32x16):

Either way, I am trying to find a way of visualizing this information. This probably isn't hard for anyone into signal analysis, but I can't seem to find a way using open-source software.

I'd like to avoid Matlab or Mathematica and I'd prefer an answer in R, since I have been learning it recently, but nonetheless, any language is welcome.

Update, 2014-07-25: given Emre's answer below, this is what the pattern looks like, given the first 30MB of the pattern, aligned at 512 instead of 256 (this alignment looks better):

Any further ideas are welcome!


texts: Intuition / Importance of intermediate supervision in deep learning

texts:
These days, I have seen many papers using intermediate supervision.

Single Network

When using a single neural network, multiple neurons output predictions, perhaps by processing data in different ways. Then, the loss function sums up the individual losses computed over each prediction.

For example, consider a part of the FlowNet architecture below:

In this network, all the Convolution# layers (Convolution1, Convolution2, ...) output predictions at different stages of the network. Then, the loss function is computed by applying a mean-squared-error over all these predictions separately and adding them all.

Multiple Networks

When using multiple networks, like in Stacked Hourglass Networks:

Each individual network outputs a prediction, and the overall loss function is computed by computing a mean-squared error over each network's prediction and summing them all up.

My question is: what is the intuition behind doing this? I thought that this will force the first network to learn to predict the task well, while the remaining networks just perform identity transformations. Why is this not observed in practice?

Also, I have only seen this applied in CNNs, but I could be wrong.


texts: Bags of visual words - counter intuitive result

texts:
I'm reading the frequently-cited paper Bags of Binary words for Fast Place Recognition in Image Sequences and have found something strange in the paper. The similarity measure is presented as:

$s(v_1, v_2) = 1 - \frac{1}{2}|\frac{v_1}{|v_1|}-\frac{v_2}{|v_2|}|$

where $v_1$ and $v_2$ are bag-of-word vectors. Now, suppose $image_1$ contains 100 occurrences of a certain word and $image_2$ contains just one occurrence. Because $\frac{v_1}{|v_1|} = 1$ and $\frac{v_2}{|v_2|} = 1$, wouldn't $s(v_1,v_2) = 1$ even when the two images are very different? That's a rather counter-intuitive result.


texts: Skip NA in Mean function within Pandas agg function

texts:
Here's my working code:
new_df = old_df.groupby('store', as_index = False).agg({'reviewScore': 'mean'})

I would like to add an argument to the mean function: skipna=True
I can't figure out the syntax to make it happen.
Do I need to learn how to use lambda functions?


texts: XGBOOST : model.predict_proba() and model.predict() conflicting behaviour

texts:
I have two classes : 1 and 2

The output of model.predict_proba() -> [0.333,0.6667]

The output of model.predict() -> 1

This is happening for around 200 test values out of the test data of 10 lac. For all other records, both the functions' output don't conflict.
Even if the probability of class 2 is higher, predict function gives final class as 1.

What could be the reason for the output of model.predict() and model.predict_proba() not syncing with each other for those 200 records?


texts:
Since the aim of a Discriminator is to output 1 for real data and 0 for fake data, hence, the aim is to increase the likelihood of true data vs. fake one. In addition, since maximizing the likelihood is equivalent to minimizing the log-likelihood, why are we updating the discriminator by ascending its stochastic gradient as mentioned in Algorithm 1 in https://arxiv.org/pdf/1406.2661.pdf. Shouldn't we update the discriminator by descending its stochastic gradient?

Any help is much appreciated!


texts:
I need to develop a report that will show automated queries in an audit log of queries on a system of the company. The logs have this fashion:

query_id id          query_time
1        1           2018-02-01 00:09:02
2        1           2018-02-01 00:24:55
3        1           2018-02-01 00:58:55
4        1           2018-02-01 01:01:49
5        1           2018-02-01 01:05:42
6        1           2018-02-01 01:18:56

Where query_id is an index of the query, the id is who has queried and query_time is the time of the query.

I tried to make the difference between the time of the queries and count how much was less than 60 seconds, but don't make the job. One other way is see the periodicity of the queries. I was thinking if there was a better way to do this. Maybe something using time series. There is some technique that could identify a query bot?


texts: Sklearn Random Feature Importances Identical for Predicting Different Response Variables

texts:
I have created four random forest models they have the same X data, but their y data are four different response variables. The sklearn random forest feature importance is identical for all four. All four models achieve their purpose and make different predictions, but their random forest feature importance is the same.
Has anyone experienced this before?
I created the models with a series of nested objects like illustrated below. I used the same code before without having identical random forest feature importances, there was however the difference that inside each object I ran a 3-fold CV to determine max_features, whereas here I just used the default which is all of them.
Current code:
class NoCVMethod:
    def __init__(self, X_train, y_train, X_test, y_test, y, Method):
        self.clf = Method
        self.clf.fit(X_train, y_train)
        self.predictions = self.clf.predict(X_test)
        self.rev_preds = rev_pred(y[-(13978+97):].values,self.predictions)
        self.residuals = y_test - self.rev_preds
        self.RMSE = np.mean((self.residuals)**2)**0.5

class Different_variables:
    def __init__(self, X_train, y_train, X_test, y_test, Method):
        self.TSS = NoCVMethod(X_train, y_train[y_train.columns.tolist()[0]], X_test, y_test[y_test.columns.tolist()[0]], y[y.columns.tolist()[0]], Method)
        self.NOx = NoCVMethod(X_train, y_train[y_train.columns.tolist()[1]], X_test, y_test[y_test.columns.tolist()[1]], y[y.columns.tolist()[1]], Method)
        self.NH4 = NoCVMethod(X_train, y_train[y_train.columns.tolist()[2]], X_test, y_test[y_test.columns.tolist()[2]], y[y.columns.tolist()[2]], Method)
        self.PO4 = NoCVMethod(X_train, y_train[y_train.columns.tolist()[3]], X_test, y_test[y_test.columns.tolist()[3]], y[y.columns.tolist()[3]], Method)

Old code:
class CVMethod:
    def __init__(self, X_train, y_train, X_test, y_test, y, param_dict, Method):
        self.pipeline = Pipeline([
            ('scale', StandardScaler()),
            ('clf', Method)
        ])
        self.param_grid = param_dict
        self.grid = GridSearchCV(self.pipeline, param_grid = self.param_grid, cv = 3, verbose = False, n_jobs = -1)
        self.grid.fit(X_train, y_train)
        self.predictions = self.grid.predict(X_test).ravel()
        self.rev_preds = rev_pred(y[-(13978+97):].values,self.predictions)
        self.residuals = y_test - self.rev_preds
        self.RMSE = np.mean((self.residuals)**2)**0.5

class CVDifferent_variables:
    def __init__(self, X_train, y_train, X_test, y_test, param_dict, Method):
        self.TSS = CVMethod(X_train, y_train[y_train.columns.tolist()[0]], X_test, y_test[y_test.columns.tolist()[0]], y[y.columns.tolist()[0]], param_dict, Method)
        self.NOx = CVMethod(X_train, y_train[y_train.columns.tolist()[1]], X_test, y_test[y_test.columns.tolist()[1]], y[y.columns.tolist()[1]], param_dict, Method)
        self.NH4 = CVMethod(X_train, y_train[y_train.columns.tolist()[2]], X_test, y_test[y_test.columns.tolist()[2]], y[y.columns.tolist()[2]], param_dict, Method)
        self.PO4 = CVMethod(X_train, y_train[y_train.columns.tolist()[3]], X_test, y_test[y_test.columns.tolist()[3]], y[y.columns.tolist()[3]], param_dict, Method)
``


texts: Change number format of confusion matrix

texts:
I have the following confusion matrix:

I would like to change the format of the numbers that, when they exceed the value 99, appear in scientific format.
I would like them to appear in a standard format. That is: 3.3e + 02 would be 330.

This is the function I have implemented:

  cm= confusion_matrix(y_test,predicted_classes)
  plt.figure(figsize=fig_size)
  plt.title('Confusion matrix of the classifier')
  sns.heatmap(cm,annot=True)
  plt.xlabel('Predicted')
  plt.xticks(rotation=45)
  plt.ylabel('True')
  plt.ioff()
  plt.show()


texts: Handling nominal category features in decision tree

texts:
I have been reading some stackoverflow questions on how to handle nominal features for decision tree (sklearn implementation).
One of the answer states that :
Using a OneHotEncoder is the only current valid way, allowing arbitrary splits not dependent on the label ordering, but is computationally expensive.
But if we use LabelEncoder and make our tree go deep enough it will eventually isolate each category and the output remains the same.
So whats is the advantage of OneHotEncoding then?


texts: very important features, but rarely input

texts:
The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would go about even normalising or whatnot these features:

     age        wgt       sex   gain        loss         hpw
0   0.379310    0.119825    1   2174           0         -0.078664
1   0.568966    0.131898    1   0              0         -2.327679
2   0.362069    0.407596    1   0              0         -0.078664

The gain can skyrocket to 100,000 and that is definitely of significance considering I'm predicting whether someone will make  over $50k, though they occur at such a small rate, close to 3-5% that I'm not sure how I could go about implementing them. I doubt imputing data would be a good idea considering just about everyone is on 0 for both results.

Also to note is I have 15 columns total (many more with get_dummies()).


texts: LSTM keras model architecture interpretation

texts:
I would appreciate if anyone could correct my interpretation of the LSTM architecture in keras-
for example in this simple case
model = Sequential()
model.add(LSTM(32, input_shape=(10, 2)))
model.add(Dense(1))

My interpretation is that, 2 features get mapped to 32 cells (sort of dense layer connection) and each of this cell unrolls in time 10 times. Next, since return_sequences=False by default, the output would be 32 values, one from each cell which then again get mapped to a single output neuron.
So something like where
F -> Feature matrix at an instant
I -> Identity matrix
for 10 times:

$F[1*2]_{matrix}$ X $I[2*32]_{matrix}$ ----> outputs $[1*32]_{input-to-32-cells}$
$in[1*32]$ X $[LSTM-equations]$ ----> outputs $[1*32]_{output-to-dense-on-last-iteration}$

if last_iterration:

$[1*32]$ X $[32*1]_{trainable-weights}$ ----> outputs a single scalar value

and the second case around which I am trying to understand is, when there are multiple LSTM layers-
something like
model = Sequential()
model.add(LSTM(32, input_shape=(10, 2)))
model.add(LSTM(64))
model.add(Dense(32))
model.add(Dense(1))

Focusing on what is happening between the two LSTM layers, I assumed that there would be a $I[32*64]$ identity matrix but what controls the timesteps of this second LSTM layer? I mean-
A) does the output from 64 cells is processed every time the layer below processes 10 timesteps or
B) does it wait a for a total of 100 timesteps or 10 samples (10*10) and then produce an output?
(I think A should be the right one, but please do comment on it)
Hope I was able to express clearly, if not please let me know.


texts: How does one feed graph optimization problems into Python's anneal function in SciPy?

texts: I am interested in graph problems like 2-color, max-clique, stable sets, etc but the documentation for scipy.optimize.anneal seems to be for ordinary functions. How would one apply this library towards graph formulations?


texts: How can I save my learning rate on each finished epoch using Callbacks?

texts:
I used LearningRateScheduler for my model training. I want to save learning rates on each epoch in CSV file (or other document files).
Is there any way to save those learning  rates using callbacks?


texts: Encoding Categorical Data Without Increasing the Dimension

texts:
I've been exploring methods for encoding categorical data. I was hoping to find a good method that does not increase the dimension of the dataset, similar to the one used on this dataset about drug use: Drug consumption (quantified) Data Set

Each piece of categorical data in this dataset was converted to some real number, but yet the dimension of the dataset was not increased. Instead of just randomly replacing values with numbers, there appears to be some thought out method behind this. Can anyone shed some light on this matter?


texts: Using the GA R package to optimize the weights of a MLP neural network

texts:
The neural network I am trying to evolve uses the tanh as an activation function in each neuron and has a topology of 1-5-1, so I need at least 5 weights. The solution of the GA is a real-number vector of length 5, which represents the weights of the network and each weight should take values between -5 to 5. I wrote an R function to use as a fitness function which returns the mean squared error (MSE) of the output data in comparison to the desired output. I want it to learn the cubic function. The input data I am using is

input<-seq(-1,1,0.02)

and the output data is its cubic function

des_out<-input^3

The evaluation function is the following

evalnn<-function(x){

mse<-0
for(i in 1:length(input)){

     nn.out <- tanh(x[1]*input[i]) + tanh(x[2]*input[i]) + tanh(x[3]*input[i]) + tanh(x[4]*input[i]) + tanh(x[5]*input[i])
     mse <- (des.out[i] - nn.out)^2 + mse

}
return(-(mse)/length(input))
}

I have set the return value to negative, because I want the smallest value to be thought as the best fit.

gann<-ga(type="real-valued", fitness=evalnn,min=c(-5,-5,-5,-5,-5),max=c(5,5,5,5,5),popSize=100,maxiter=150,pmutation=0.01,pcrossover=0.8)

What I always get back from the GA are weights that make up a linear function although I have been experimenting with the ga's parameters quite a lot, i.e. I have tried all crossover methods and mutations. https://cran.r-project.org/web/packages/GA/GA.pdf.

The cubic function plotted together with my output function.

I have used a linear function as training data before the cubic with this implentation and worked. It has trouble with the non-linear.

If anybody could figure out why do I get this, is there something I have missed.
Thank you


texts: Predicting Player Position from pervious positions and LSTMs

texts:
I am trying to use LSTMs to predict player positions in a field game. I try to overfit 8 slightly different time series. For this overfitting task I just use the positions of the players.
A data sample looks like this:

[Blue are the player trajectories corresponding to the green targets I want to predict, while yellow are the enemies and red is the ball]
When trying to overfit 8 future positions for a single player ( one single player is a single blue track with one green target), this is happening:

[Where red are the targets and blue are the predictions. So all predictions are basically a single point]
So the model is basically collapsing to the mean of the targets. Actually I dont know why this is happening.
My model looks like this:
model = Sequential()
model.add(LSTM(128 ,return_sequences=True))
model.add(LSTM(128 , return_sequences=False))
model.add(Dense(64,activation='relu'))
model.add(Dense(2, activation = 'linear') )
model.compile(loss='mae', optimizer='adam')

while my training data takes the form of input: [BatchSize, SequenceLength, FeatureLen] and output: [BatchSize, Target] which result in [8,20, 28] and [8,2] respectively. I also tried for the output the input shape of [8,1,2], yet this is not changing anything.
then I'll train the model with
loss = model.fit(inp, out,epochs =150, verbose=False)

Do you guys have an idea, why the LSTM is collapsing to the mean?


texts: Neural network got a lucky guess. Can it be trusted?

texts:
Say you come across a loss curve as shown below. At which loss should you trust the model? The initial lucky guess or after it has stabilized?

And more importantly, why?


texts: Is there any tool for data visualization and manipulation?

texts:
I have a time series data set that I need to manually label for supervised learning.
What I am doing now is using excel to the plot, and when I see the pattern that I want, I hover over the data on the plot, read its index, then mark the data accordingly on the data. I think it is not very efficient, for example, I can not zoom or scroll.
I want to ask is there any tool that I can visualize the data, and manipulate the data at the same time, please.


texts: Average Precision if Target Class is Not in Evaluation

texts:
Suppose I have 5 classes, denoted by 1, 2, 3, 4, and 5, and this is used in object detection.
When evaluating an object detection performance, suppose I have classes 1, 2, and 3 present, but classes 4 and 5 are not present in the targeted values.
Will each of classes 4 and 5 have average precision of 0 (due to its precision being zero as no true positives can be identified)? Or perhaps there are other considerations to take when this case occurs?


texts: Why does Keras' Model() have an inputs parameter?

texts:
In every single example of Keras' Functional API documentation, when building a model with Model(), the inputs parameter seems redundant: it can be deduced from the steps that lead to the construction of the outputs. Why does this inputs parameter seem necessary, then?

One guess is that otherwise there can be questions about the order in which the inputs should be taken (especially when looking at multiple outputs). The (order of the) inputs could however be defined through some convention, but describing this convention would be more cumbersome than asking the user to manually re-inject the inputs of the outputs?

That's only a guess: maybe there are some more fundamental reasons to ask the user to include redundant information even in cases like all those of the Functional API documentation?


texts: Poor performance of regression model for imbalanced data

texts:
I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). Therefor, the model seems to output just values less than 2 (if the range is [0,6]), for instance, and it absolutely ignores bigger target's values, which have a smaller performance in the dataset. How is it possible to improve the model's results in such a case?
For example, when it comes to a multiclass classification, we can weigh penalties for errors on a smaller class to enhance performance with imbalanced data. Are there any tricks in terms of regression? Which loss-functions could be useful? It seems, that MSE loss function is more preferable than RMSE. Is it more powerful loss functions for this problem?
There is a paper about such a problem of imbalanced regression (http://proceedings.mlr.press/v74/branco17a/branco17a.pdf) which might be helpful to someone. However, I'm more interested in special tricks for a neural network, not pre-processing approaches (I can't generate more data for example).


texts: What would be the mse (mean squared error) of my scaled dataset on the original scale?

texts:
I build an LSTM model on a standardized dataset using sklearn's MinMaxScaler. All values of the dataset are between 0 and 1. Features and target variables were standardized between 0 and 1.
I achieve an mse of around 0.02 .

Now this mse is valid for the standardized dataset. What would the mse be on the original scale?
Let's say the original scale is dollars. What would an mse of 0.02 be on the original scale in dollars?


texts: Trying to plot REC curve and calculate AUC in R but getting error

texts:
I am working on regression machine learning techniques and calculating AUC (plotting REC) by using below paper link: http://homepages.rpi.edu/~bennek/papers/rec.pdf
My Code;
REC <- function(y_val , y_predval){
initilizing the lists
Accuracy <- "[]"
initializing the values for Epsilon
Begin_Range <- 0
End_Range <- 1.5
Interval_Size <- 0.01
List of epsilons
Epsilon <- seq(Begin_Range , End_Range , Interval_Size)
Main Loops
for (i in range(length(Epsilon))){
count <- 0.0
for (j in range(length(y_val))){
if ((y_val[j] - y_predval[j]) / sqrt((y_val[j]) **2 + (y_predval[j])**2 ) < Epsilon[i]){
count <- count + 1
}
append(count/length(y_val))
}
### Calculating Area Under Curve using Simpson's rule
AUC <- sintegral(Accuracy , Epsilon ) / End_Range
### returning epsilon , accuracy , area under curve
return (Epsilon , Accuracy , AUC)

}
}
AUC <- REC(y_val, y_predval)
Error in np_array(y_val[j])^2 : non-numeric argument to binary operator


texts: Decision Tree Classifier to classify values based on values of other columns

texts:
I have data with multiple labels, for example

My X set is fromt second to third column, and I want to classify either first column or the last column, so I made my Y the last column.

The goal is so that if I would classify Vios it would return me Car or 0 in other words it can find its way to the first row.

Classification use case:

classify("poodle") #just pretend this is a working function

returns: Pets

How I did it in attempt to train my model:

from sklearn.feature_extraction.text import TfidfVectorizer
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 72)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf3 = RandomForestClassifier().fit(X_train_tfidf, y_train)

I'm using a guide from somewhere on the net that works a bit the same with it, but at the end im getting returned:

ValueError: Found input variables with inconsistent numbers of samples: [5, 4156]

I knew immediately i was doing it wrong. How do I train a model so that it achieves my goal? Any relevant guides or techniques I should be following intead of this? I dont even know the correct way to use vectors on this case.


texts: Keras lstm equations, matrices and diagram

texts:
I was reading this post, mainly the LSTM equation section for understand the structure diagram and relate it with the matrices and I can deducted this cell diagram:

I made this homemade diagram because in the Colah's article it's not the exactly same cell that in keras.
I need know if it ok, can someone confirm me that? This are the equations

W are weights matrices for input data and U are weights matrices for previous hidden state
b are biases
In the pink circles there are operations. The point it's multiply element by element and X is matrix multiplication


texts: Pytorch: Starting with a high loss value, but the loss converged at the end. I dont know if the model could start with a loss > 100. Help!

texts:
I have been trying to attempt plant disease detection using transfer learning methods. I chose ResNet50 first. I also performed a baseline model which is a CNN model. In resnet50, I used cross entropy loss and trained the model for 30 epochs. I did batch normalization too. Initially epoch 1 loss was 112.5250 and the training loss was 87.512. But, for the last epoch it was 2.1660 and Validation loss was 1.8905 with Validation accuracy as 0.995. The overall accuracy of the model was 98.8% and the model doesn't seem to overfit too.
Before training the model, I performed hyper parameter optimization where I optimized Learning rate and momentum using Bayesian optimization. I optimized weight decay using Cross validation. I performed batch normalization. Before that, I trained the model without any optimization. I just assumed the learning rate to be 0.001, same loss function and trained for 30 epochs. In this case, the model started with a loss value of 114.2 and validation loss was 130.46. It converged to 9.9038 with accuracy of 97%.
So, my question is, it is possible for the model to start with such a high loss value despite giving a good accuracy at the end? Does the magnitude of the loss have nothing to do with the correctness of the model nor the accuracy? If the model started with a huge loss, but gave a good accuracy, is my model bad?
First 5 epochs
EPOCH: 1

Epoch time taken: 28.61398434638977 seconds. Epoch 1 loss: 118.5250 Validation loss: 87.5214 Validation accuracy: 0.753

EPOCH: 2

Epoch time taken: 28.737553358078003 seconds. Epoch 2 loss: 68.8141 Validation loss: 50.3452 Validation accuracy: 0.858

EPOCH: 3

Epoch time taken: 28.45273518562317 seconds. Epoch 3 loss: 41.5181 Validation loss: 31.1520 Validation accuracy: 0.928

EPOCH: 4

Epoch time taken: 28.434630155563354 seconds. Epoch 4 loss: 26.8844 Validation loss: 21.1203 Validation accuracy: 0.948

EPOCH: 5

Epoch time taken: 28.762181282043457 seconds. Epoch 5 loss: 19.3646 Validation loss: 15.4843 Validation accuracy: 0.955

Last 5 epochs
EPOCH: 26

Epoch time taken: 28.35645818710327 seconds. Epoch 26 loss: 2.8168 Validation loss: 2.2771 Validation accuracy: 0.990

EPOCH: 27

Epoch time taken: 28.51978588104248 seconds. Epoch 27 loss: 3.3986 Validation loss: 2.2274 Validation accuracy: 0.992

EPOCH: 28

Epoch time taken: 28.51390767097473 seconds. Epoch 28 loss: 2.2707 Validation loss: 1.9976 Validation accuracy: 0.992

EPOCH: 29

Epoch time taken: 28.456573247909546 seconds. Epoch 29 loss: 2.5344 Validation loss: 1.9272 Validation accuracy: 0.990

EPOCH: 30

Epoch time taken: 28.486974239349365 seconds. Epoch 30 loss: 2.1660 Validation loss: 1.8905 Validation accuracy: 0.995
```


texts: Evaluate prediction from multiple classification model

texts:
Given that I have data containing images of oranges, apples and pineapples and I want to classify depending on a set of features.

Expected that I have completed the model and ready for prediction.

My questions are:

How can I output each score for every category the model predicted like this one?

Image 1:
     60% apple, 20% orange, 1% pineapple
     It implies that the image is an apple.

Image 2:
     40% apple, 60% orange, 10% pineapple
     It implies that the image is an orange.

Is there any libraries I can use with?

Does it depend on the model I am using? If yes, what are these models that implement this evaluation?


texts: What is the meaning of pct_trend_adjusted in FiveThirtyEight's polling data?

texts: I'm working on polling data from fivethirtyeight. What is the meaning of the pct_trend_adjusted column here? I understand that pct stands for percentage, but I don't get what the trend is that they are adjusting for, and how these figures are obtained.


texts: How to make overfitting (powerful) model?

texts:
According to my professor one of the first steps in modelling a NN is to use a powerful enough model.

  The first step is to create a model that is powerful enough to
  achieve very high accuracies (very low loss) on the training data, at
  least when no regularisation is used.

What are some of the things (obviously apart of regulazing and adjusting learning rate), that I can do to make my model "powerful" enough, in other words to let it overfit on the training data?

Am I looking in the right direction with the following things?

Add extra layers
Make layers thicker (more neurons)


texts: User-to-item and item-to-user recommendations

texts:
I'm currently creating a recommender system and there are different types of the systems.
Does anyone know something about the user-to-item and item-to-user recommendations? I have been searching for days, but I couldn't find anything about them. I have only found info about the user-to-user and item-to-item ones.


texts: segmentation of brain tumor in MRI images

texts:
I have a dataset of brain tumours images. and I have to build a model to classify the malignancy grade of these tumours.
The size of the tumours varies from small to large. The ROI are already selected by an expert (the boundary of tumours is drawn).

To work on such dataset, for sigmintation step,
do you recommend on cropping the tumours based on boundries, which will result in a variation of image dimensions as shown below. Or having the original image, but subtrackting the background of the tumor, which will result in fixed sized images of the tumor with black background, but has a large dimention considiering the tumor is only a part of the original image.

also, is it applicable working on cropped images that are resized to have same fixed sized images?


texts: Problem when cherry picking actions - Proximal Policy Optimization

texts:
I'm using the implementation of PPO2 in stable-baselines (a fork of OpenAI's baselines) for a RL-problem.

My observation space is 9x9x191 and my action space is 144. Given a state, only some actions are "legal". If an "illegal" action is taken, the environment will return the same state. Think of it as the game of Go, where you try to place a stone on an intersection that is already occupied.

When a legal action is taken, it can result in a state that is completely different from the state before. It can also result in legal actions that are completely different. So it's not as in Go, where the stones stay on the board.

I currently feed the model a vector containing the legal actions (a multi-hot-vector, 1 for legal, 0 for illegal). When the model samples the actions, I first run the action means through a sigmoid so they all become positive but still keeps relative size order. Then I multiply the means with the legal action vector which sets the illegal actions to zero. The model then samples from only the legal actions with argmax. I do not change the underlying probabilities of the actions, so when calculating the neglogs of an action, they will be the true neglogs.

This works well for some time, but after a while I get the following error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Found Inf or NaN global norm. : Tensor had NaN values

The error is thrown when I calculate the tf.clip_by_global_norm:

grads, _grad_norm = tf.clip_by_global_norm(grads, self.max_grad_norm)

I'm guessing that the grads become infinitely large.

A general trend is that new and old neglog action probabilities starts to increase (which means that the action probabilities for the new and old policy starts to decrease). The Kullback Leiber distance also increases rapidly between them. This will lead to the ratio (r(θ) in the paper) to increase, which also will lead to a very volatile policy loss.

If I increase the number of steps before update, it can run for longer, but eventially it will crash.

I have tried to punish illegal actions instead of removing them, but the convergence is very slow.

My questions are:

What might be the reason for the grads going bananas?
Can it affect the model in an unexpected way if I cherry pick the legal actions?
Is there any other, better way that limit the action space to just legal actions? I haven't found any papers that analyze it. Do anyone know how AlphaZero handles illegal actions?


texts:
In R-CNN paper, they give the definition of the target values for bounding box regression
Given that $(P, G)$ is a (prediction box, ground-truth box) pair of the form $(x, y, w, h)$ where $x, y$ is the center coordinate of the box, $w, h$ are width and height respectively.

$t_x = (G_x - P_x) / P_w \hspace{2.0cm} t_y = (G_y - P_y) / P_h$
$t_w = \log(G_w / P_w) \hspace{2.0cm} t_h = \log(G_h / P_h)$

And the goal is to find $\textbf{w}_*$, where $*$ can be $x, y, w$ or $h$, so that

$\textbf{w}_* = \arg \min_{\hat{\textbf{w}}_*} \sum_i (t^i_* - \hat{\textbf{w}}_*^T \phi(P^i))^2 + \lambda \|\hat{\textbf{w}}_*\|^2$ where $\phi(P^i)$ is the feature map given by the last pooling layer of the feature extractor after passing predicted bounding box $P^i$

I don't understand why they come up with this approach of bounding box regression ? Can anyone tell me about this ?
P.S: since this regression approach is used not only in R-CNN but also in later models, I really want to get a clear understanding of this


texts:
I have a txt file (here) and i have tried to plot it with pandas, xarray of python
I tried with
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv("name.txt",
                 delim_whitespace=True,
                 usecols=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],
                 names=["no.", "January ", "February ", "March", "April ", "May", "June", "July", "August", "September", "October", "November", "December"])
df.head()
df.info()
df.shape
print(len(df));
print(len(df.columns))

df_full = df.replace(-99.000000,np.NaN)
df_full.describe()

Then, I would like to plot a time-series with x is time (daily, monthly or yearly) and y-axis is value. Do you have any suggestions?


texts: How Box cox and other transformations convert data into Normal Distributions?

texts: How Box cox and other transformations convert data into Normal Distributions ?


texts: keras: extract feature names from a saved model

texts:
I have a large number of features. I train several keras models by varying the number of features to see the impact of these features on model accuracy. In the end, when I load these models, I no longer know on which features the model was trained on so when I do the predictions, I get the error like:

Error when checking input: expected dense_4_input to have shape (29,) but got array with shape (47,)

Is there a way to know what are the exact feature names on which the model was trained?
I use keras.models.load_model to load the model and keras.models.sequence.save(path) to save the model.


texts: Linear Discriminant Analysis (LDA) , Removing colinear features in cross validation set up is correct or not?

texts: My data is a matrix with 725 features and 667 observations which have binary data(either 0 or 1). my target variable is a univariate which has two classes (either 0 or 1). I removed the features which had zero as column sums, so features reduced to 652. since I have a binary data I did not do the preprocessing (Mean = 0 and s.d=1). But, My data is highly sparse; there is 99% sparsity. If I want to use LDA model in a 10 c.v setup. Then I am required to remove the colinear features (this means that all the features in a training set whose column sum is zero should be removed) to predict small set of test data in every fold. If this is the case then in every fold I will have to remove a different set of features which are zero in that training set for the test set. Is it an acceptable practice to complete my y.predict after 10 c.v (each fold used different features to predict the y) and do a rowmean for all the folds in the end and then show the accuracy of a classifier? I cant have the correct feature selection from this strategy but I can get a classifier prediction performance.


texts: What is the effect of the use of online-learning algorithms on non steaming data?

texts: am wondering what the effects of using a passive-aggressive classifier instead of something like a SVM classifier on a non-streaming data would be. In other words, what are the general assumptions behind the online learning algorithms that distinct them from general machine learning algorithms?


texts: What is a simple image classification ANN architecture and a simple data set?

texts:
I'm looking for a neural network architecture, which gives pretty good performance on an image classification challenge and is still simple to implement in a framework and also reasonably fast to train. Do you have suggestions?

I'm new the neural network implementation, and in the end, I'd like to do some "research-style" playing with it.

Moreover, a pointer to an image classification data set which is not too tough for a simple network would be great. However, not something like MNIST which is so easy that almost any classifier does a good job.

PS: Which neural network framework do you recommend if I'd like to modify the way the network is trained?


texts: Why seq2seq models are superior to simple LSTMs?

texts:
It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to machine translation, to text generation).

Why? What are the underlying mathematical reasons for which an LSTM encoder-decoder architecture would outperform more canonical RNNs? Is it in the generation of dense latent representations? Is it about the comparatively higher number of parameters? Any hint is appreciated.


texts: Make classification and prediction at the same time

texts:
I am working on the detection and prediction of epileptic seizures and I was thinking about something : would it be possible to apply classification and prediction at the same time. I mean, having seen some data points of EEG, it would be possible to predict the (few) next values of the time series while saying "it's part of a seizure" (or at the opposite, "nothing happens, remain quiet") at the same time ?

If yes, would it be possible to do both using the same neural net ?

Thanks for any help


texts: What is "Missing" in output of plot_tree API of XGBoost

texts:
What this "Missing" term means here at each node after split in Image? and also what is at leaf, is this means prediction value? I converted Output variable to 1 and 0.
I tried searching on google, but didn't get the answer. Please help


texts: Feature extraction - wavelet transformation + autoregression

texts:
I am working on a feature extraction problem (ECG signal). Within my literature review I stumbled across the following text:

"The wavelet transform is used to extract the coefficients of the transform as
the features of each ECG segment. Simultaneously, autoregressive
modelling(AR) is also applied to obtain the temporal structures
of ECG waveforms." here

My question now is, how do the wavelet transformation and the AR differ, i.e. how does the AR 'obtain the temporal structure'?


texts: Time series change rate calculations for displaying trend line chart


texts: Anybody know what this type of visualisation is called?

texts:
I think this is a pretty cool way to visualise changes in values but I can’t find any name for this type of visualisation.
I
Source: https://www.economist.com/graphic-detail/2020/07/28/americans-are-getting-more-nervous-about-what-they-say-in-public


texts: Small amount of training data set for naive Bayes classifier for binary classification

texts: I'm implementing prediction system for young cricketers in ODI format using Naive Bayes classifier. The output of the system is to predict whether the young player is rising star or not. I have collected data from statsguru API of espncricinfo but I'm getting only about 300 records of players from ODI. Is it very small for training dataset?


texts: Parameter Tuning by Cross Validation for Random Forest

texts:
I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data.

Is there any problem with this methodology?

For the max_depth parameter I get a really high value of 500 and that seems too high. Any advice?

The code is:

from __future__ import print_function
import sklearn
import sklearn.ensemble
import sklearn.metrics
from sklearn.datasets import fetch_20newsgroups
from sklearn.grid_search import GridSearchCV

categories = ['sci.med', 'soc.religion.christian']
newsgroups_train = fetch_20newsgroups(subset='train', categories=categories,
remove=('headers', 'footers', 'quotes'))
newsgroups_test = fetch_20newsgroups(subset='test', categories=categories,
remove=('headers', 'footers', 'quotes'))
class_names = ['medicine', 'christian']

vectorizer =
sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)
train_vectors = vectorizer.fit_transform(newsgroups_train.data)
test_vectors = vectorizer.transform(newsgroups_test.data)

rf = sklearn.ensemble.RandomForestClassifier(max_features='sqrt')

param_grid = {
           "n_estimators" : [10, 100, 1000],
           "max_depth" : [5, 100, 500],
           "min_samples_leaf" : [1, 20, 40]}

CV_rf = GridSearchCV(estimator=rf, param_grid=param_grid)
CV_rf.fit(train_vectors, newsgroups_train.target)
print(CV_rf.best_params_)


texts: Error::Type of predictors in new data do not match that of the training data

texts:
I am building a classification model using randomForest. When trying to predict I get the below error

  Type of predictors in new data do not match that of the training data

I made sure that the testing and training data has same levels.

I also included

levels(test_var) <- levels(train_var)

to make sure that the levels are matching.

But still I end up getting this error, is there anything else that I should look for?

**EDITED ON 3rd Aug 2015**

  Here is the structure of the training dataset

  Structure of the test dataset

  Sapply training data

  Sapply testing data

  In order to make sure that the levels are matching between the training and test datasets, I wrote this loop to see if there are any differences exist between the datasets

for(i in 1:28)

 {
   if(is.factor(testing[,i]))
    {

    print(names((testing[i])))
    difference_in_test = setdiff(levels(testing[,i]), levels(training_data[,i]))
      print(difference_in_test)

    }
 }

  Results of the above For loop which shows that there are no differences exist between the levels.

[1] "VAR1"
character(0)
[1] "VAR2"
character(0)
[1] "VAR3"
character(0)
[1] "VAR4"
character(0)
[1] "VAR5"
character(0)
[1] "VAR7"

  I still continue to get the error as mentioned below:

  Edit on 4th Aug

I started getting the above error only after including a filter in my dataset like this:

training_data <- subset(training_data, gender !="F")


texts: Are there any examples of neural networks that take two samples as input with a label of {same class, different class}?

texts: There must be examples of this, though I haven't been able to find any. Maybe I don't know what to search for.


texts:
I'm new to data science. I have a question on anomaly detection techniques.

There are several anomaly detection techniques such as statistical, density based, depth based, clustering, etc..

Given a dataset, what are the criteria or how should I choose which one of the techniques above (not the algorithms inside the techniques). In general why choose a particular technique.


texts: Convert Lat /Lon of User input to Lat/Lon of Open Data

texts:
I have data from a public data set in gridded form 2.5 degree x 2.5 degree(lat,lon).
Latitude goes from 90 N to -90 S. Longitude goes from 0 to 357.5. It is stored every 2.5 degrees and there are no intermediate values.

My user will want to download data from this data set. However they are not restricted from entering only the values that the public data set possesses. My goal then is to convert the data they enter to the "nearest" latitude and longitude of the public data set. For example while data may exist at 5 N and 60 E(and the next pont is 7.5 N and 62.5E) they may enter 6 N and 61.3 E.

How do I map the user input to the nearest latitude and longitude ? Note - user input will be in the form of a rectangle - lat_min,lat_max, lon_min and lon_max.

I took a shot at this and I hope I can find a better algorithm through DS SE.

lat_min = floor(lat_min/2.5) * 2.5

lat_max = ceiling(lat_max/2.5) * 2.5

lon_min = floor(lon_min/2.5) * 2.5

lon_max = ceiling(lon_max/2.5) * 2.5


texts: How to get rid of some parts in a string variable?

texts:
The original string is like this "965 - Vehicle parking area", how do I get rid of the "965 - " part in R? (There are two spaces before and after the dash.)
Thank you!


texts: Feature fusion multiple modalities example

texts:
I am trying to replicate this paper. In their architecture, there is an attention-fusion subnet explained as follows:

In the multimodal activity recognition problem, not all modalities are
equally contributed to the classification task. In order to prioritize
the important modalities, we introduce a self-attention network, which
takes the sensors’ feature vectors [vt1, · · · , vtk, · · · , vtK] as
input and outputs an attention weight for each modality. And the
attention weights represent the importance of different sensors in the
HAR task. Then those feature vectors of all sensors are fused by using
their attention scores as weights to form a uniform feature
representation vector ct. Here, we compute the hidden representation
of vtk through a one-layer MLP to get µtk, then we measure the weight
of the k-th sensor as the similarity of µtk with a sensor-level
context vector w1 and get a normalized weight αtk through a softmax
function. ct is the uniform representation of all K sensors which is
computed by the sum of all sensors’ feature vectors weighted by their
attention weights. {W1, b1, w1} are parameters of the attention subnet
which are randomly initialized and jointly learned during the training
process.

What I do not understand is how the features are finally created and fused using the attention scores. So for every modality, there is an attention layer that computes the attention scores of every feature for that modality. Then how are these scores used to fuse all sensors? How is the uniform representation created?
I do not understand how to implement this layer. Any clarification would be appreciated.