texts: Why is 10 considered the default value for k-fold cross-validation?

texts:
I understand very well what k-fold cross-validation is. In my studies, and at work, I've always heard something along the lines of:

  We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.

I intuitively can wrap my head around this. However, I cannot seem to find any research that went into declaring k=10 as the default. How would one go on to prove that 10 is the best value?

I understand the effects of using a smaller k versus a larger k in terms of bias. But why is it 10? And no 5 or 20? How could one end up on the number 10?


texts: ktrain predictor.predict_proba is too slow - Bert classifier with ktrain

texts:
Training:
(x_train,  y_train), (x_test, y_test), preproc = text.texts_from_array(x_train=X_train, y_train=y_train,
                                                                       x_test=X_test, y_test=y_test,
                                                                       class_names=class_names,
                                                                       preprocess_mode='bert',
                                                                       maxlen=350,
                                                                       max_features=35000)
model = text.text_classifier('bert', train_data=(x_train, y_train), preproc=preproc)
learner = ktrain.get_learner(model, train_data=(x_train, y_train),
                             val_data=(x_test, y_test),
                             batch_size=6)
learner.fit_onecycle(2e-5, 3)
predictor = ktrain.get_predictor(learner.model, preproc)
predictor.get_classes()
predictor.save("bert_model2")

Prediction of probability for each class:
predictor = ktrain.load_predictor('bert_model2')
predictor.get_classes()
val=predictor.predict_proba(l1)

l1 is a list of 100000 sentences. I want to predict the probability of each class for each sentence.
But predict_proba is taking too long approx 5 hours. I am on a GPU runtime in google colaboratory.
Is there a way to speed up this process as I want to do the same for approx 44 lakh sentences?


texts: Predicting hardware failures with limited data

texts:
I am exploring using machine learning to predict if a particular hardware component would fail within a timeframe, say 3 months. The ultimate goal is to minimize physical human inspection so that maintenance crew would always performance maintenance/servicing just in time before hardware failure.

However, servicing are carried out at a fixed interval regardless of the condition of the hardware and it has practically never failed. The dataset we have is of this nature:

equipment: pump1
location: outdoor (or indoor)
provider: providerA
operation: -
   temperature (as a function of time)
   moisture (as a function of time)
   team (as a discreet function of time, ie teamA, teamB, teamC)

Basically, we know some features of the equipment (here pump1), and we know some operating parameters (here temperature, moisture) as a function over time, and we also know which team did the servicing over time. I understand that survival analysis would be most suitable for a predictive maintenance problem but each operating parameters vary over time of study. Nothing is constant.

While regression does not take into account time. As such how do we actually model it to predict its failure?

Note: The above parameters are adapted from https://www.kaggle.com/ludobenistant/predictive-maintenance


texts: To One-Hot-Encode or not to One-Hot-Encode?

texts:
I have been struggling to find proof for that but I couldnt

Every time I prepare dataset I face the same issue

when a column is a classification such as CountryCode or TaskType in this dataset

TaskType  CountryCode  Target
1         61           Red
1         962          Yellow
2         1            Yellow
6         61           Yellow
4         81           Red
2         1            Blue
1         61           Red
2         962          Green
4         61           Blue

if I applied the dataset as to different models such as linear regression, SVM, KNN, etc.

will these model consider CountryCode and TaskType as numeric fields and treat them as continuous data?

Shall I One Hot Encode these features before using them?

what is the best way to handle this scenario?


texts: Figure out relative importance of entity attributes

texts:
I'm trying to understand how various aspects of a movie contribute to its gross revenue. I want to rank a movie's attributes in that sense - the attributes that most strongly determine the revenue are ranked higher.
Let $A_1,\ldots,A_n$ be a list of attributes of a movie and let the possible values of $A_i$ be $a_{i1},a_{i2},\ldots$. Many of these attributes (like primary genre) are categorical and some of them (like rating) are continuous.
Approach 1: Consider $A_1$: I can form groups of movies having the same value of $A_1$, e.g. all movies with $A_1=a_{12}$ form a group. The other attributes in a group can vary freely. I can then calculate the mean of the revenues of all movies within a group, and then take the variance of means of all groups.
This will give me the "variation in average revenue as we change $A_1$ values". If this variation is high, that means changing $A_1$ significantly affects the average revenue - so $A_1$ should be a highly ranked attribute.
Approach 2: Again consider $A_1$: fix the values of all other attributes $A_2,\ldots,A_n$ and look at movies with the same values for $A_2,\ldots,A_n$ but different values of $A_1$. Find the variance in revenues of such movies - call it "$A_1$ variance". The attributes $A_i$ with highest "$A_i$ variance" will be ranked higher.
Approach 3: Train some ML model (not sure which one) with revenue as target variable and attributes as features. Then look at feature importances to get attribute importances.
A few queries:

What assumptions do I need to check for approach 1? e.g. minimum size of a group, distribution of other attribute values within a group, etc.
Are there any potential flaws or gotchas in approaches 1 and 2?
What approach would you prefer out of the three?
What ML model should be used for approach 3? I'm confused because there are plenty of regression models: Linear regression, GB regression, random forest regressor, etc.


texts: Neural Net for regression task on images only learning mean of training data

texts:
I'm trying to train neural networks to extract parameters out of input data.
In particular determining the width, height and color of bricks (see attached pictures as example inputs).

I've already tried CNNs operating on the raw images and NNs operating on extracted features ( 2 describing edges and a color histogram ).

Both approaches seem to learn the mean of the training data. The color is mostly gray and the width and height between the two extremes. The output does not depend on the input as the NN always outputs the "same" values (only very small fluctuations).

As I'm operating on generated data I have a perfect correlation between input and desired output and infinite training data. The used loss function is mean squared error (also tried the absolute error).

Can someone explain why this is happening or how to avoid this training behavior?

I'm using the library tiny-cnn.
My current approach (using features) uses 5 fully connected layers, decreasing layer size from 1200 to 17(amount of parameters).

    net << fully_connected_layer<sigmoid>(1200, 600)
    << fully_connected_layer<sigmoid>(600, 300)
    << fully_connected_layer<sigmoid>(300, 150)
    << fully_connected_layer<sigmoid>(150, 60)
    << fully_connected_layer<identity>(60, 17);

I've test several CNN structures, always ending in 2 fully connected layers. I tried different kernel sizes from 3 to 11.

This is one of the last I tried:

net << convolutional_layer<tan_h>(100, 100, 5, 3, 4)
    << average_pooling_layer<tan_h>(96, 96, 4, 2)
    << convolutional_layer<tan_h>(48, 48, 7, 4, 8)
    << average_pooling_layer<tan_h>(42, 42, 8, 2)
    << convolutional_layer<tan_h>(21, 21, 7, 8, 16)
    << average_pooling_layer<tan_h>(15, 15, 16, 3)
    << convolutional_layer<tan_h>(5, 5, 5, 16, 32)
    << fully_connected_layer<relu>(32, 64)
    << fully_connected_layer<identity>(64, size);

5-conv, average, 7-conv, average, 7-conv, average(3x3), 5-conv, full, full

I've mainly used RMSProp and adagrad with standard learning rates but also tried higher and lower rates.

Example inputs:


texts:
I have a dataset which has only categorical values. As I came across a few articles people suggested that KNN / Random forest would work for dataset like this. Though in R it couldn't handle as if contains more than 53 categories.

Whereas in Sklearn, I used  linear regression against each column to the column which I was predicting. After which I created a weighted average of all the outputs. But I'm pretty sure it's not the right way.

Can anyone suggest a better way?

PS : The dataset contains 4 columns  with only categorical values. With 200 - 250 different categories in each column.


texts: How to create a Time Series Training Dataset with variable sequence length

texts:
I have time series data with variable sequence lengths.
So something like:

date        value label
2020-01-01  2     0     # first input time series
2020-01-02  1     0     # first input time series
2020-01-03  1     0     # first input time series
2020-01-01  3     1     # second input time series
2020-01-03  1     1     # second input time series

how is it possible to create a training dataset (numpy arrays) of shape [samples, time_steps, n_features] when time_steps is not consistent?

Additional Info: The model that is going to be trained is an LSTM which is capable to handle variable input lengths.


texts:
We use log-likelihood (called as lambda) to reduce the risk of numerical underflow (in context of sentiment analysis using Naive Bayes).
What does "reduce the risk of numerical underflow" means?


texts: A Loss of 55.2164 with a sparse_categorical_crossentropy in a sequential neural network?

texts:
I'm following Aurélion Géron's book on Machine Learning.
The following code tries to evaluate a neural network with  a sparse categorical cross entropy loss function, on the Fashion Mnist data set. How come I get such a strange value for loss?

import tensorflow as tf
from tensorflow import keras
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

fashion_mnist=keras.datasets.fashion_mnist

(X_train_full,y_train_full),(X_test,y_test)=fashion_mnist.load_data()

X_valid , X_train = X_train_full[:5000]/255.0 , X_train_full[5000:]/255.0
y_valid , y_train = y_train_full[:5000] , y_train_full[5000:]

model=keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28,28]))
model.add(keras.layers.Dense(300,activation="relu"))
model.add(keras.layers.Dense(100,activation="relu"))
model.add(keras.layers.Dense(10,activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",optimizer="sgd",metrics=["accuracy"])
history=model.fit(X_train,y_train,epochs=30,validation_data=(X_valid,y_valid))

#some instructions with *history* for plotting a graph

model.evaluate(X_test,y_test)

and the output I get is [55.21640347443819, 0.8577]

How come I get a loss over 1?


texts: Custom Named Entity Recognition using deep learning

texts: I have a dataset with two columns. First column has some text (news article) and the second column contains names of people (not exactly English names) present in those news articles (first column). I've tried to build a custom named entity recognizer using Spacy but it isn't working. Can I use deep learning approach to identify the names in unseen news articles ? (Test data)


texts: removing words based on a predefined vector

texts:
I have the dataset test_stopword and I want to remove some words from the dataset based on a vector. How can I do that in R?

texts <- c("This is the first document.",
       "Is this a text?",
       "This is the second file.",
       "This is the third text.",
       "File is not this.")

test_stopword <- as.data.frame(texts)
ordinal_stopwords  <- c("first","primary","second","secondary","third")


texts: Machine Learning over distributed operating system

texts:
I want to implement Distributed operating system mutual exclusion using Machine learning algorithms but I am not getting dataset anywhere to perform.
Is there anyway i can get the required dataset?
I already tried UCI and amazon public dataset site.


texts: Identifying and Accounting for trend/seasonality in Predictor Variables

texts:
I'm currently working with a dataset that has been collected over several years, and I suspect my predictor variables are changing over time for their predictive power.

I could go back year by year and run the data the same way each time to see how efficient each predictor is, then trend the predictive power over time manually. There has to be a better way.

Can anyone point me towards the technique I should cram on?


texts: Convert a csv file so that all attributes will become columns with corresponding element

texts:
so I have this dataset in a csv file. I would like to convert it into a matrix form.

The data currently looks like this:

University, adelphi
State,  newyork
Control,  private
No-of-Students, 5-10
Male:Female Ratio:, 30:70
Student:Faculty Ratio:, 15:1
Sat Verbal,  500
....
Academic-Emphasis,  business-administration
Academic-Emphasis,  biology

I would like to convert it so that the Attributes become columns and the elements on each corresponding row. Like in a excel sheet.

Example:

University | State   | Control | No-of-Students| ect.
adelphi    | newyork | private | 5-10          |

Thank you very mcuh for your help!


texts:
Hello this is my first machine learning project, I got a dataset with 18.000 rows and I have a column with 4244 values missing.

I don't know why the values are missing since when it's appropriate there's a 0 value in it.
The dtype of the column is int64 I consider this column usable and would like to implement it to the model.

Could you please help me with how to deal with this problem, or lead my to a resource to teach me how to deal with this ?


texts: PCA on Neural Networks dimensions reduction?

texts:
The dataset which was extracted from the database consists of more than 50         columns, I call these columns dimensions, can I call them dimensions?
Obviously, I have to do dimension reduction on them. But since PCA like algorithms often do axis rotating to generate some new axises. I don't think I will PCA algorithm in dimensions reduction. So I calculated the correlations between these columns(parameters), and filtered these who has a high value and some other rules. So can I still call it dimensions reduction? Since I only did some parameters filtering
The reason I don't use PCA like algorithms is because I want to implement Neural Network classification, and I need the real parameters.

Please comment on these, anything even criticizing is welcomed.


texts:
New to ML in general and have been Googling on this. I am working on a dataset that will rate each customer features to a credit worthiness class attribute.
Is it possible to add class weights in the 'mlp()' function to deal with imbalance of class attributes? I have read the official document for RSNNS package. There is a parameter that is initFunc = "Randomized_Weight" but the documentation does not offer any other initFunc options.
Can this be done on the mlp() parameters, or I have to do a class weightage before the mlp() step?


texts: Using BERT for input embeddings in a seq2seq model

texts:
I'm currently trying to implement a paper that describes using BERT to embed inputs into a seq2seq model.

"For word vectors, we use the deep
contextualized word vectors from ELMo (Peters
et al., 2018) or BERT (Devlin et al., 2018). The
answer tag follows the BIO2
tagging scheme." source

The issue I'm running into is the first input into the decoder. Traditionally this is the start token, but BERT's start token is both a sentence representation & not consistent. Depending on what is being embedded the start token representation will change. e.g. embed("101") will have a different representation of "101" than embed("101 2000 2004 1102").

This creates issues for the model during inference as there is no ground truth to get a start token from. Using just the embed("101") as the start token causes the model to predict the same thing every time when otherwise it would not. It seems like there is a good number of people who use BERT to embed their inputs but I don't see anyone addressing this particular issue. I have considered the possibility of using BERT exclusivity as the encoder of a model and taking the [cls] token from that embedding as the first input to the decoder. However, this contradicts the paper listed above.


texts: Free API for historical weather US data?

texts:
I am trying to retrieve a free R-Python API that provides historical weather data in US.

In fact wunderground API is no longer available.

Any suggestion?


texts: Validation loss diverging away from the training loss

texts:
I used the XLNET for a sentiment classifier in determining whether a comment is positive or negative. I was able to get good results

But when I plotted the validation and training losses I saw this

I think this means that the model is overfitting? But I am not exactly sure. If there is any suggestions I would really appreciate it.


texts: Multi-output regression problem with Keras

texts:
number of features: 12 , -15 < each feature < 15

number of targets: 6 , 0 < each target < 360

number of examples: 262144

my normalization: I normalized the features so that they are between 0 and 1. I normalized the targets so that they are between 1 and 10.

This is the model that I am using:

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(6, activation='linear')
])

model.compile(optimizer="rmsprop", loss='mean_squared_error', metrics=['accuracy'])

model.fit(training_x, training_y, epochs=10, batch_size=100)

This is the best result that I have got (training):

235929/235929 [==============================] - 8s 33us/step - loss: 8.9393e-04 - acc: 0.6436

testing:

loss: 0.00427692719418488

acc: 0.033187106618348276

I get almost 0% accuracy on the test set! I need a model to solve this ML problem.


texts:
In the darknet deep learning framework .cfg files we see parameters like
angle, saturation, exposure

These parameters are used for data augmentation in classic image classification problems.
Does Darknet automatically perform image augmentation automatically for object detection by setting up these parameters?


texts: How can I improve the accuracy of my model? (Cab Cancellation Prediction)

texts:
I am trying to predict based on several parameters like trip type, car type, source of booking, start time, lead time (start- book) and a few other params whether or not a customer will cancel. From the code below the accuracy of default.ct the 1st classification I do is giving me an accuracy of 75%. deeper.ct the deeper tree that I am generating is giving me an accuracy of 70%. Progressively the accuracy of the pruned tree also is remaining the same. Boosting with adabag package is taking way too long because I've nearly 5,00,000 observations across 19 variables. xgboost is giving me the best mlogloss value at about 0.43.

What can I do to improve the accuracy of the model?

    # Generate classification tree
    default.ct <- rpart(tag ~ ., data = train.df, method = "class",
    control=rpart.control(minsplit=2, minbucket=1, cp=0.001))
    summary(default.ct)$used
    printcp(default.ct)

   # generate confusion matrix for training data
   prp(default.ct, type = 1, extra = 1, under = TRUE, split.font = 1, varlen =
  -10)
  default.ct.point.pred.train <- predict(default.ct,train.df,type = "class")
  confusionMatrix(default.ct.point.pred.train, train.df$tag)

    deeper.ct <- rpart(tag ~ ., data = train.df, method = "class", cp = 0,
   minsplit = 1)
   # count number of leaves
   length(deeper.ct$frame$var[deeper.ct$frame$var == "<leaf>"])

   ## Use cross-validation to prune the tree
   cv.ct <- rpart(tag ~ ., data = train.df, method = "class", cp = 0, minsplit =
   5, xval = 5)
   # use printcp() to print the table.

   printcp(cv.ct)
   # Use variable c to store accuracy data for different cp and print it out
   c <- list()

   for (i in 1:nrow(cv.ct$cptable)){
   pruned.ct <- prune(cv.ct,
                  cp = cv.ct$cptable[i])
   pruned.ct.point.pred.train <- predict(pruned.ct,valid.df,type = "class")
   c[i] <- confusionMatrix(pruned.ct.point.pred.train, valid.df$tag)$overall[1]
   }

    # prune the tree with second large cp and use it to predict validation data
    pruned.ct <- prune(cv.ct, cp = cv.ct$cptable[2])
length(pruned.ct$frame$var[pruned.ct$frame$var == "<leaf>"])


texts: How to correctly train CNN with patches taken by labeled images, if the source image contains both positive and negative samples?

texts:
I have patches (tiles) taken from very large histopathological images. These images are labeled as healthy (negative) or tumor (positive).
If the image-level label is negative, then 100% of the source image is healthy, there are no tumor regions, and every patch should be labeled as negative.
If the image-level label is positive, then there are tumor regions among the healthy tissue. We don't know if a particular patch represents the healthy region or the tumor region.
With no further information, we label every patch of that image as positive before feeding it to the CNN.
In this way, I get a low accuracy since many samples that are labeled as positive are actually negative, and the CNN won't be able to generalize enough.
I was wondering if there are better ways to approach this problem. I have researched several papers and articles, but none of them went close to this problem, even if I thought it wasn't so uncommon.
The closest solution is the "Hotdog/Not Hotdog Classifier", which can't be applied since we don't have isolated samples of the positive class.
Do you have any ideas or resources?
Thank you


texts:
I see that LSTM is very powerful reconstructing time series it was fed with, but my issue is:

=> Can LSTM predict future values without requiring real (training/testing) data?

My objective is simple: Modelling a univariate time series to predict itself.

ARIMA, SARIMA and Holt Winters were used, but Machine Learning (using LSTM) approach for forecasting seems fake in practice. Am I wrong about my point?

My dataset is a 2 years duration series observed daily.

I've evaluated initially 1.5 year as training set, defined by trainingX (a rolling window of 180 days duration) and trainingY (a rolling window of 30 days duration).

I have tried:
- A single LSTM layer with 30 units, returning sequences
- A single LSTM layer varing of 1 unit to 1024 units and a dense layer with 30 units
- Stacked LSTM with 2 to 4 layers with 1 to 1024 layers and a dense layer with 30 units
- All of them were modified by sigmoid, relu and tanh activation functions

Besides,
Using a rolling window prediction, a method that releases the first "L" values of the trainingX set and append the first "L" prediction values at the end of the same trainingX set, I get random predictions converging to a constant or to zero.

The training and predict batch were varied of 1 to 250.

Have anyone experienced a reconstructed (lagged) output time series when wanting a forecasting output?


texts: Why Does XGBoost Keep One Feature at High Importance?

texts:
I am training an XGboost model for binary classification on around 60 sparse numeric features. After training, the feature importance distribution has one feature with importance > 0.6, and all the rest with importance <0.05.
I remove the most important feature, and retrain. The same distribution forms; the most important feature has importance > 0.6, and the rest have < 0.05. I continued to remove the most important feature and retrain, remove and retrain, remove and retrain, etc. My f1-score started to drop, but every time there was one feature more important than the rest.
Also worth noting, when I removed the most important feature and retrained, the new most important feature was not the second most important feature from the previous training.
I cannot explain this behaviour intuitively. Does anyone know why this pattern arises?


texts: Data sets that have strings and numerical data all in one column

texts:
Here's a code I wrote for pre-processing a data set. It works
import numpy as np
import pandas as pd
from sklearn import svm

%matplotlib inline
import matplotlib.pyplot as plt

from sklearn.impute import SimpleImputer
import seaborn as sns; sns.set(font_scale=1.2)

stock=pd.read_csv("C:/Users/Dulangi/Downloads/winequality-red.csv")
stock.head()
X= stock.iloc[:,0:5].values
y= stock.iloc[:,5].values

g=sns.lmplot('alcohol','quality',data=stock,height=7, truncate=True, scatter_kws={"s":100})
imputer = SimpleImputer( strategy = "mean")

imputer = imputer.fit(X[:,1:2])
imputer.fit_transform(X[:,1:2])

imputer = imputer.fit(X[:,4:5])
imputer.fit_transform(X[:,4:5])

I want to know what if i have both strings and numeric data in one column, how to pre-process such data to have all numeric data?


texts: Where do I start? I have a massive dataset (from web scraping) and want to predict y from 20 variables

texts: I'm trying to do my best to recreate an algorithm that a company is using by sucking in all of its outputs and adding in some relevant variables (I won't be able to perfectly recreate it as I'm missing inventory data). I'm guessing this is a job for machine learning but I was wondering where I start? Are there specific models I should start with? Software I should use? Courses I should dive into?


texts: Find the most representative samples from a class

texts:
I have a dataset where each sample has 187 dimensions. The normal class has 70.000 samples while the other 4 have at most 5.000. Ideally, I would like to keep at most 15000 samples from the normal class which are representative enough. In other words I want to discard redundant samples from the normal class keeping the most beneficial ones.
Has anyone tried something similar?


texts: Implementation of the paper 'Perceptual Generative Adversarial Nets for small object detection'

texts:
I studied the research paper on Perceptual Generative Adversarial Nets for small object detection.

There they have detailed the structure of Generator network as given in the picture below:

I am new in the field of GAN. I am having problem in designing the generator function. Can anyone help me out?


texts:
I have a list of some 100 millions of strings, each of different length. Examples:
    nsdgnlnesef   ngmrlxkvgrmksefsfnlj   rnrfnmsbanana   housemgslen    assistremovecouch

As you can see the strings vary in length and some can include english words, some not, some both words and random characters. All use the characters [a-z]. It is possible to characterise the strings are being humanly created or by a algorithm with a reasonable certainty.

Later I aim to also classify between different sub-types of algorithms generating these strings (multiclass classification).

The 100 millions strings I have are already labeled, based on other analysis. But because this is a very slow task, I will use Machine Learning to try and predict the origin of new strings as we get new data.

The methods I have started with are bigrams, Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU). I will eventually also run trigrams.

As both time and accuracy are very important factors of the method I eventually choose, I want to make sure I test as many as possible. But since there is a forest of methods, I hope to get some helpful advice here.

I imagine methods based on features, such as Random Forest and Support Vector Machine could be useful, but I haven't them used before, and I am not certain if they or other methods could handle 100 millions of strings in an efficient way.

Any suggestions for methods to look at are very welcomed.


texts: Data Cleaning Techniques - What to drop?

texts: I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, and the product ID. However, some of the data is missing. Some store IDs do not have continuous data for the 55 weeks, and some item IDs do not have continuous data. Also, combinations of store IDs and item IDs have missing data points. What is the most efficient way to decide what to drop and what to keep for the best results in my model? If the product I am forecasting is continuous that is a good start, but I am not sure what the best combination of stores and items to drop is.


texts: Hindsight Experience Replay, how to define a partially-known End-Goal

texts:
One of the requirements of the Hindsight Experience Replay is supplying the DQN with a state and a goal (the desired end-state) that we hope to end up in:

<currState, goal>    <-- For Hindsight Experience Replay;
<currState>          <-- This would be an input for a usual DQN;

This paper allows to quickly learn when the rewards are sparse. In other words when the rewards are uniform for most of the time, with only a few rare reward-values that really stand out.

Question:

Let's say I want to have the player be killed by monsters in my game. Thus, my "goal state" must include a value of 0 for player's hit-points. However, the state-vector also includes his position (xyz coordinate), rotation vector, IDs of equipped items:

inputVec = <hp,  x1,x2,x3,  q1,q2,q3,  chestID, handsID, headID, feetID>

I don't want to impose a specific position of a player, etc - I just want him dead. I only know what his 'hp' should be (should be zero), I don't care about the other values.

Therefore, I can't provide a perfectly well-defined goal vector - does this mean I can't use Hindsight Experience replay?

Edit:
my understanding is that components of currState and goalState must have identical components. We can't have these 2 vectors be of different sizes or store different things

Edit after accepting the answer:

As @lfelipesv mentioned, page 4 tells us:

  We assume that every goal $g ∈ G$ corresponds to some predicate $f_g : S → \{0, 1\}$ and that the agent’s
  goal is to achieve any state s that satisfies $f_g(s) = 1$. In the case when we want to exactly specify the
  desired state of the system we may use $S = G$ and $f_g(s) = [s = g]$

  The goals can also specify only
  some properties of the state, e.g. suppose that $S = \mathbb{R} ^2$
  and we want to be able to achieve an arbitrary
  state with the given value of x coordinate. In this case $G = \mathbb{R}$ and $f_g((x, y)) = [x = g]$.


texts: What are the input and output channels of a convolution in PyTorch?

texts: From the documentation of Pytorch for Convolution, I saw the function torch.nn.Conv1d requires users to pass the parameters "in_channels" and "out_channels". I know they refer to input channels and output channels but I am not sure about what they mean in the context of convolution. Could someone explain this to me?


texts: How do I find the relevant features out of 11,000+ possibilities?

texts:
While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many?

I know that we need to find out which features are relevant, but not sure how to do this.


texts: Two steps optimization of a credit card limit

texts:
I have a problem similar to what is on the title but not the same, the problem on the title allows me to explain the dynamics of my need.
I have to determine how much is the optimal value for a variable called QUOTA or LIMIT for a credit card.
The goal of the model is to allow me to minimize the probability of default, given this variable and others which caracterize my costumer. Which is the best way to determine this?
I have thought two possibilities:

Train a model with all the exogenous and the LIMIT as variables for any classification model (with Y=Default/No Default)
Train a model with all the exogenous variables but not a LIMIT (with Y=Default/No Default), then train another model with the resulting probability and LIMIT as the only two variables (with Y=Default/No Default).

Any of this possibilities sound right?
With both models, the application dataset should repeat costumers with two or more possible values for the LIMIT variable and obtain the default probability of every customer repetition. Does this sound correct?


texts: Using Machine Learning to Predict Musical Scales

texts: It's possible to use Machine Learning techniques to cluster songs into musical-scale groups? I mean: "this song was written in C"... or "this song was written in Am" etc. I made a fast search about the subject and I found no software that can do this. If you know some software, or research (academic papers), related to that subject, could you link it here for me? I'm very interested in that subject but I'm not sure from where I can begin. I have a little experience with Random Forests and Neural Networks, maybe I can accomplish the classification task with one of those algorithms, but, again, I'm not sure which kinda of features I should pass to the algorithm. Thanks in advance.


texts:
I am using onehotencoding and RandomForestRegressor to predict scores of a set of soccer games.
How can I use it into predict? I am sure I am doing it wrong at the moment as I am getting all predict values to be 1 (Probably because I am filling all NaN values as 1 for splitting and fitting)
What dataset should I pass as I am encoding a few columns and then transforming it?
My code is as below
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeRegressor

# Pandas needs you to define the column as date before its imported and then call the column and define as a date
# hence this step.
date_col = ['Date']
df = pd.read_csv(
    r'C:\Users\harsh\Documents\My Dream\Desktop\Machine Learning\Attempt1\Historical Data\Concat_Cleaned.csv'
    , parse_dates=date_col, skiprows=0, low_memory=False)

# Clean dataset by dropping null rows
data = df.dropna(axis=0)

# Column that you want to predict = y
y = data.Full_Time_Home_Goals

# Columns that are inputted into the model to make predictions (dependants), Cannot be column y
features = ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result']
# Create X
X = data[features]

# Split into validation and training data
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

# Define and train OneHotEncoder to transform numerical data to a numeric array
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(train_X, train_y)

transformed_train_X = enc.transform(train_X)
transformed_val_X = enc.transform(val_X)

# Build a Random Forest model and train it on all of X and y.
# To improve accuracy, create a new Random Forest model which you will train on all training data
rf_model_on_full_data = RandomForestRegressor()

# Define columns we want to use for prediction
columns = ['Home_Team', 'Away_Team']
test_data = test_data[columns]
# Renaming Column Names to match with training dataset
test_data = test_data.rename({'Home_Team': 'HomeTeam', 'Away_Team': 'AwayTeam'}, axis=1)
# Adding NaN columns to dataset to match the training dataset
test_data['Full_Time_Result'] = np.nan
test_data['Full_Time_Away_Goals'] = np.nan
test_data['Full_Time_Home_Goals'] = np.nan
# Aligning dataframe to model defined
test_data_features = test_data[features]
# Filling all NA values as Encoder cannot handle nan values
df = test_data.fillna(1)

# Define Y for Fitting
Y = df

# We need nY as that would be the column used for splitting
ny = df.Full_Time_Home_Goals

# We need to encode and transform dataset so we have converted all nan to 1 and we are defining a new model as the
# val_x values are confusing, we will use n_
train_n_X, val_n_X, train_n_y, val_n_y = train_test_split(Y, ny, random_state=1)

# Since we have text again, we will need fitting and transforming the data
enc.fit(train_n_X, train_n_y)
transformed_train_n_X = enc.transform(train_n_X)
transformed_val_n_X = enc.transform(val_n_X)

# Fitting and then we will be using predict
rf_model_on_full_data.fit(transformed_train_n_X, train_n_y)

# Predicting. This step needs correction as predict should be on the new dataset and not just on on column.
test_preds = rf_model_on_full_data.predict(transformed_val_n_X)

print(test_preds)

What should go into predict() to get the outcome that I want?
Files used here


texts: Error in using fit() on RandomForest Classifier where X was a pandas.DataFRame object

texts:
On using fit() method on sklearn.ensemble.RandomForestClassifier I am getting a value error that says.

ValueError: could not convert string to float: 'male'

The data-set used is the one in Titanic:Machine Learning from Disaster competition on Kaggle.
Here is the link- https://www.kaggle.com/c/titanic
Can someone please help me how to deal with this, why is it occurring and how to prevent it in future.

Note-There are no NaN in my DataFrame for train_X, i.e I have replaced all NaN with df.fillna(df.mean()), also I cross-checked that no NaN values exist by using

train_X.isnull().sum()

where, train_X is the training data for features.
Please Help!!


texts: How to merge two CNN deep learning model using weighted sum and weighted product in Keras?

texts:
I am using Keras to create a deep learning model and I want to merge two CNNs by using weighted sum or weighted product.

How can I merge two CNNs using weighted sum and weighted product?


texts: Are there any projects to work on the Internet to fight against COVID?

texts:
Are there any open source projects that a novice data analyst and mathematician can do, to fight against covid-19 epidemic? I mean, I know that the best I can do is to stay away from people and now I have a laptop and plenty of time to work on some project. But the thing is that I don't know what kind of computations would be useful to save people.

I have Ubuntu, LibreOffice, R, Python, and Sagemath installed on my computer.


texts: How to Deploy a Scikit-learn Model into the Cloud?

texts:
I have generated a common Sklearn model for text classification which I want to make accessible in the cloud (there is no provider preference) as an API.

So far the closest solution that I managed to find is in this tutorial but it seems quite complex (getting the venv.zip dependencies package at the beginning is unclear, for instance) and specific (in my case NLTK and an external Stanford segmenter is involved for preprocessing and I cannot figure out where to put all these modules and how to invoke them).

Is there a decent and robust way to solve this challenge?


texts: Keras custom metric doesn't work as loss function

texts:
Referencing my previous question here.
I've managed to get my angular error metric working with tf.py_function;
ang_error = tf.py_function(angular_error, [data_true, data_pred], tf.float32)

but when I use it for a loss function, I get the following error:
InvalidArgumentError:  shape of dy was [] instead of [3]
 [[node gradient_tape/angular_error/strided_slice_1/StridedSliceGrad
 (defined at <ipython-input-15-3ce535aaa0f9>:23) ]]
 [Op:__inference_train_function_1113]

Function call stack:
train_function

The last layer where I want to use this metric as a loss function if defined as:
orientation = Dense(3, activation="linear", name='orientation')(base_output)


texts: Is there a way to save the current state of your experiment so that you can pick up from where you left off in Sklearn similar to checkpoints in Keras?


texts: Is it possible for a ML model to perform worse than random classification?

texts:
I implemented different machine learning algorithms on a matrix with binary data to predict a univariate target with two classes.

random forest (accuracy = 62.01)
Neural Network(acc= 58.9)
svm-radial kernel (accuracy = 58.02)
linear discriminant analysis(accuracy = 57.9)
logistic regression(accuracy = 57.6).

My baseline accuracy is 52.55. But in case of Naive Bayes in same setting gives only 48.5 accuracy that identifies only one class in y. predict.

Is it possible for a machine learning model to behave worse than a random classification?


texts: In reference to Gamma Gamma spend model, is this in any way related to Pareto/NBD, BG/NBD or BG/BB model?

texts:
The reason I ask this question is as follows:

I am currently in a remote intern position.
I've been asked to calculate the Customer lifetime value (past + future) for each customer based on monthly cohorts.
I can choose from P/NBD, BG/NBD or BG/BB model whatever is appropriate.

I came across the Gamma-Gamma spend model. Is this in any way associated with the above models? How?

According to this (https://segment.com/docs/guides/warehouses/how-do-i-forecast-ltv-with-sql-and-excel-for-e-commerce/), they just use average transaction value for each customer and multiply by expected conditional transactions. Is this a sufficient approach by changing individual averages to monthly cohort averages?


texts: Linear SVM in matlab and python giving different results

texts:
I have a particular dataset on which I am getting different results when using a linear SVM in matlab and sklearn toolbox.
The data has been normalized in matlab and imported into python from a mat file.
The codes used in Matlab is
acc = 0;
for i = 1:10
   [train,test] = crossvalind('HoldOut',Y,0.2);
   mdl = fitcsvm(X(train,:),Y(train),'KernelFunction','linear');%,'BoxConstraint', 10,'KernelScale',0.001);
   predictions = predict(mdl,X(test,:));
   C = confusionmat(Y(test),predictions);
   acc(i) = (C(1,1)+C(2,2))/((C(1,1)+C(1,2)+C(2,1)+C(2,2)));
end
acc = sum(acc)/10;

The code used in python is
clf_opt = svm.SVC(C=10,gamma=0.001,kernel='linear',random_state=0, tol=1e-5)
clf_opt.fit(X,y)
cvs_svm = cross_val_score(clf_opt,X,y,cv=StratifiedKFold(10)).mean()

For matlab SVM I am getting an accuracy of around 77% and in python around 60%. The choice of parameters of C=10 and gamma = 0.001 was reached after doing a GridSearchCV in python.
I went through existing posts in google for reasons of difference in LinearSVM in matlab and python but none of them worked out. I also tried out X = StandardScaler().fit_transform(X) in python but changed accuracy by 0.5 %.
I am getting comparable classifier accuracies on standard datasets (eg.IRIS) but the results  are differing in this dataset only. The dataset is attached in the link below
https://ufile.io/qs7jy
The link is a compressed file in 'rar' format and  contains three files

Python_Dataset_X  - Can be loaded with pickle
Python_Dataset_Y  - Saved as np array
Matlab_Dataset.mat - Contains the X matrix as table and Y array.

Any assistance would be appreciated.


texts: Calculate feature weight vector for one-hot-encoded data frame in R

texts:
I have the following data frame with one categorical and two numerical columns:

    V1  V2  V3
1   A   1   3
2   A   3   5
3   B   3   3
4   C   2   3

I have turned this into the following dummy variables:

    V1.1  V1.2  V1.3 V2  V3
1   1     0     0    1   3
2   1     0     0    3   5
3   0     1     0    3   3
4   0     0     1    2   3

Now, I want to apply clustering to this latter set. I guess that I could get better results if I downweight the dummy variable columns (proportionally to their number) because with equal weights the distance based clusters will be distorted.

My question is, how could I get the following weight vector from the new data set:

0.33, 0.33, 0.33, 1, 1


texts: Neural network accuracy in Torch depends on compute power?

texts:
I am new to machine learning and have quite good understanding of basic concepts.

I was implementing a 3 Layer neural network on MNIST dataset with 784, 100, 10 neurons in input, hidden, output layer respectively. I did not use any regularization here.

Firstly I trained the network on Intel i5 4th generation quad-core CPU with 4GB of ram which gave me 64% of accuracy.
Then I trained the exact same network, with exact same code on Intel i5 7th generation quad core CPU with 8gb of ram which gave accuracy of about 89%
This is the link to the implementation.

My question is in Torch, does the compute power effect the accuracy of the network? or is it something else that I am missing which has resulted in this huge change.

I did not use any different weight initialization method than the default provided in the torch libraries, so that is ruled out.
I also did not use anything else which may effect the network to change its accuracy to this extent.


texts: How to predict multiple set of coordinates for signboards text localisation through neural network

texts:
I am creating a signboard translation model from scratch. I have images of signboards where there are multiple texts and I have the corresponding set of coordinates for multiple texts. I want to create a regression model which will try to predict the coordinates if there is text in the image.
I am really stuck at a place. In some cases, I have multiple words in the image, so each word will have its own set of coordinates. So, how can I make a model such that if there is a single word then it will output single set of coordinates, but if there are 5 words then it should give me 5 set of coordinates? The number of output may vary with each image. What kind of neural net should I use? I don't want to use sliding window approach. Please help me out.


texts: Need help understanding LSTMs' backpropagation and carousel of error

texts:
I'm having a hard time trying to derive the maths behind LSTMs and vanishing gradients.

I had a of help from LSTM forward and backward pass, but I got stuck in page 11 from LSTM forward and backward pass.

Given the image:

We can form system of equations,
$$
        \begin{bmatrix}
        a^t \\
        i^t \\
        f^t \\
        o^t \\
        \end{bmatrix}
=
        \begin{bmatrix}
        tanh(W_cx^t+U_ch^{t-1}) \\
        \sigma(W_iX^t+U_ih^{t-1}) \\
        \sigma(W_fx^t+U_fh^{t-1}) \\
        \sigma(W_ox^t+U_oh^{t-1}) \\
        \end{bmatrix}
=
        \begin{bmatrix}
        tanh(\hat  a^t) \\
        \sigma (\hat i^t) \\
        \sigma (\hat f^t) \\
        \sigma (\hat o^t) \\
        \end{bmatrix}
$$
We can then represent this as $z$:

$$
z=
        \begin{bmatrix}
        \hat  a^t \\
        \hat i^t \\
        \hat f^t \\
        \hat o^t \\
        \end{bmatrix}
=
        \begin{bmatrix}
        W^c & U^c \\
        W^i & U^i \\
        W^f & U^f \\
        W^o & U^o \\
        \end{bmatrix}
*
        \begin{bmatrix}
        x^t \\
        h^{t-1} \\
        \end{bmatrix}
$$

We can find out the backprop derivation for $z$ from page 10 from LSTM forward and backward pass

$$
\delta z=
        \begin{bmatrix}
        \delta \hat  a^t \\
        \delta \hat i^t \\
        \delta \hat f^t \\
        \delta \hat o^t \\
        \end{bmatrix}
=
        \begin{bmatrix}
        \delta a^t \odot (1-tanh^2(\hat a^t)) \\
        \delta i^t \odot i^t \odot (1-i^t) \\
        \delta f^t \odot f^t \odot (1-f^t) \\
        \delta o^t \odot o^t \odot (1-o^t) \\
        \end{bmatrix}
$$

However the next part at page 11 from LSTM forward and backward pass is where I'm confused.

Given $\delta z$, we need to find $\delta W$, $\delta h^{t-1}$,

1) The author wrote down $\delta I^t = W^T * \delta z$:

If we do some linear algebra variables moves:

$$z = W^T * I^t$$

Multiply both sides with $I^{t^{-1}}$

$$I^{t^{-1}} z = W^T$$

Multiply both sides with $z^{-1}$

$$I^{t^{-1}} = z^{-1} W^T$$

Somehow this doesn't match with the author's formula?

2) Let's ignore 1), and try to solve for $\delta I$

$$
\delta I =
        \begin{bmatrix}
        \delta x^t \\
        \delta h^{t-1} \\
        \end{bmatrix}
$$
$$
\delta I = \frac{dE}{dI} =
        \begin{bmatrix}
        \frac{dE}{dx^t} \\
        \frac{d}{dh^{t-1}} \\
        \end{bmatrix}
$$

But $\frac{d}{dh^{t-1}}$ depends on a lot of the equations in $z$

Do I solve for them individually and add them up?

Note: $\frac{dE}{d\hat i_t}$ can be found at page 10 from LSTM forward and backward pass

$$h_{t-1}^{i_t}=\frac{dE}{dh_{t-1}^{i_t}}=\frac{dE}{d\hat i_t}\frac{d\hat i_t}{h_{t-1}}=\frac{dE}{d\hat i_t}\frac{d}{dh_{t-1}}i_t(1-i_t)$$

Replace $i_t$ with $\sigma(W_iX^t+U_ih^{t-1})$, replace $\frac{dE}{d\hat i_t}$ with $\delta \hat i_t \delta i_t$

$$=\delta \hat i_t \delta i_t \frac{d}{dh_{t-1}} \sigma(W_iX^t+U_ih^{t-1}) (1-\sigma(W_iX^t+U_ih^{t-1}))$$

It looks solvable, then my question is I would get 4 equations like the above, do I add them all together in the end to get $\delta h^{t-1}$? For example:

$$
\delta I = \frac{dE}{dI} =
        \begin{bmatrix}
        \frac{dE}{dx^t} \\
        \frac{d}{dh^{t-1}} \\
        \end{bmatrix}
=
        \begin{bmatrix}
        ignore \\
        h_{t-1}^{i_t}+h_{t-1}^{a_t}+h_{t-1}^{f_t}+h_{t-1}^{o_t} \\
        \end{bmatrix}
$$

Since my logic was to find the total error contributed from $h_{t-1}$ so you need to add them together?

3) Finding for $W$ looks even like a bigger task, however, I'm not sure where to start on this?

4) How does this relate to the error carousel? I mean after derivation of all the weights and $h_{t-1}$, I'm not sure how this leads to avoidance of vanishing gradients? I read somewhere that the weights are constant 1 or something along the lines like that?

I know this is kinda long, but feel free to ask for clarification if my question does not make sense. I think I've tried to solve for this almost for half a month now.

Appreciate any sort of guidance. Thanks.


texts: XGBoost hardware requirements. CPU vs. memory

texts:
I am working on predictive models with ML using very roughly 10-50 million records (currently testing with less) and around 10 explanatory variables per model.
When outlining hardware requirements for a good VM-setup, it is often difficult for me to say, which additional computational power improves how well. When I look at the task manager (working on a Windows 64bit machine) during training time of my XGBoost models, the CPU is always at 100% using all the cores.
Looks like parallelization works fine, but the training still takes a lot of time.
It seems logical for me to thus ask primarily for more CPU (that is many more cores). However memory does also play a role, because, if I understand it correctly, the algorithm stores/compresses data differently, if less memory is available.
Here is my question: should I appreciate the offer for a considerable amount of additional memory (that seems easier to get) and be okay with just a modest improvement on the CPU side, or should I ignore all that just ask for more CPU?


texts: Question about BERT embeddings with high cosine similarity

texts: Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?


texts: F1 on test set using Naive Bayes classifier decreases as training set size increases. Why?

texts: I've done a simple naive bayes classification task with a very small data set. As the training set size increases from 100 data points to 300 data points, the F1 score on the test set decreases. But why? Is it likely overfitting? What are the first places I should examine to explain this behavior?


texts: How can I convert my predictions to text after predicting using RNN?

texts:
I'm building PoS tagger for our language. I give tokens to the words and tags using Tokenizer(). Functions for word and tag are different.
# encode Y
tag_tokenizer = Tokenizer()
tag_tokenizer.fit_on_texts(tags)
Y_encoded = tag_tokenizer.texts_to_sequences(tags)

I have simple bidirectional LSTM to predict the algorithm.
bidirect_model = Sequential()
bidirect_model.add(Embedding(input_dim     = VOCABULARY_SIZE,
                             output_dim    = EMBEDDING_SIZE,
                             input_length  = MAX_SEQ_LENGTH,
                             trainable     = False
))
bidirect_model.add(Bidirectional(LSTM(64, return_sequences=True)))
bidirect_model.add(TimeDistributed(Dense(NUM_CLASSES, activation='softmax'))
bidirect_model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['acc'])
bidirect_training = bidirect_model.fit(Xtrain, ytrain, batch_size=128, epochs=10, validation_data=(Xvalidation, yvalidation))

loss, accuracy = bidirect_model.evaluate(Xtest, ytest, verbose = 1)
print("Loss: {0},\nAccuracy: {1}".format(loss, accuracy*100))

The result is pretty good, the accuracy is more than 94%. But when it comes to actually change the output from tokens to words it's a little struggle.
tag_tokenizer.sequences_to_texts(ytest.argmax(2))

When I try to change the ytest to text it works just fine but when I use the same method with prediction it doesn't convert it correctly.
ypred = bidirect_model.predict(Xtest)
tag_tokenizer.sequences_to_texts(ypred.argmax(2))

it returns the correct amount of the tags but it can return only one tag name for all of them. (For example noun for each tag, but there should be different PoS tags as well). I noticed that when I predict using LSTM the data actually changes to different format:
This is ypred:
array([[[9.96039510e-01, 3.83606239e-05, 3.69549962e-05, ...,
         5.24025600e-05, 9.90346816e-05, 4.23052406e-04],
        [9.97439384e-01, 3.10516843e-05, 2.83600530e-05, ...,
         3.12017110e-05, 5.57345738e-05, 2.51515419e-04],
        [9.98397648e-01, 2.60055913e-05, 2.21154296e-05, ...,
         1.76890717e-05, 2.89566851e-05, 1.39188676e-04],
        ...,
        [1.67124510e-01, 2.64671326e-01, 1.25262424e-01, ...,
         4.61455435e-03, 3.27696186e-03, 2.74900091e-03],
        [1.39361277e-01, 2.67008126e-01, 1.36608765e-01, ...,
         4.63735918e-03, 3.30406637e-03, 2.69036856e-03],

But actually my ytest is:
array([[[1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],
        ...,
        [0., 1., 0., ..., 0., 0., 0.],
        [0., 0., 1., ..., 0., 0., 0.],
        [0., 0., 1., ..., 0., 0., 0.]],

       [[1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],
        [1., 0., 0., ..., 0., 0., 0.],

I tried to round the numbers but that didn't work as well.


texts: What does "freedom" mean in the RankNet paper?

texts:
In the RankNet (Learning to Rank using Gradient Descent) paper, paragraph 3.1, it said

given the consistency requirements, how much freedom is there to choose the pairwise probabilities?

What does it mean?


texts: Pandas dataframe resample aggregation by mills too slow

texts:
Given this test data:

import pandas as pd
import numpy as np

data = {'date': ['2014-05-01 18:47:05.069722', '2014-05-01 18:47:05.119994', '2014-05-02 18:47:05.178768', '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.280592', '2014-05-03 18:47:05.332662', '2014-05-03 18:47:05.385109', '2014-05-04 18:47:05.436523', '2014-05-04 18:47:05.486877'],
        'battle_deaths': [34, 25, 26, 15, 15, 14, 26, 25, 62, 41], 'prisioners': [3, 4, 3, 2, 2, 6, 4, 5, 2, 8]}
df = pd.DataFrame(data, columns = ['date', 'battle_deaths', 'prisioners'])

Set index datetime:

df = df.set_index(pd.DatetimeIndex(df['date']), drop=True)
del df['date']

It works when I want to resample to the milliseconds, but it takes too long...

timeit df.resample('1L').sum()

I guess is because is aggregating all the milliseconds with NaN data, but when I drop it ..

timeit df.resample('1L').sum().dropna()

It takes even longer

guessing again that the dropna is done at the end... There is any way which by dropping the NaN samples will accelerate the process?


texts:
I am working on a problem of entity extraction which requires me to extract variables of interest from a text document. My challenge is that the text contains multiple entities of a variable, for ex. For a variable "name", there are multiple names but I only have to extract a particular name of interest. Likewise there are other variable with multiple entities. I have 2 questions regarding this.

How should I prepare my data for training?. Should I annotate all the entities for a variable or just the entity of interest. For ex: Out of names like "james", "dave" and "harry", if name of interest is "james", then should I annotate only "james" or all the names for training.

Is there any particular method to approach such problem?


texts: Is it possible to somehow improve the prediction?

texts:
I have a set of points, which is represented on the chart in black.
I take the first 978 points and I want to predict the remaining 22.
Blue indicates prediction using a recurrent neural network.
Green indicates forecast using ARIMA with orders (7, 0, 2).

The forecast segment looks like this:

Here is the code ARIMA:

model = ARIMA(arr, order=(7,0,2))
model_fit = model.fit(disp=0)
output = model_fit.forecast()
forecast = model_fit.predict(start=len(arr), end=len(arr) + 22)
res = np.append(arr, list(forecast))

Here is the code recurrent neural network:

def split_sequence(sequence, n_steps):
    X, y = list(), list()
    for i in range(len(sequence)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the sequence
        if end_ix > len(sequence)-1:
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

def model():
    n_steps = 3
    # split into samples
    X, y = split_sequence(arr, n_steps)
    # reshape from [samples, timesteps] into [samples, timesteps, features]
    n_features = 1
    X = X.reshape((X.shape[0], X.shape[1], n_features))
    model = Sequential()
    model.add(Bidirectional(LSTM(50, activation='relu'), input_shape=(n_steps, n_features)))
    model.add(Dense(1))
    model.compile(optimizer='adam', loss='mse')
    # fit model
    model.fit(X, y, epochs=100, verbose=1)
    # demonstrate prediction
    x_input = array([arr[-3], arr[-2], arr[-1]])
    x_input = x_input.reshape((1, n_steps, n_features))
    yhat = model.predict(x_input, verbose=0)
    return yhat

while len(arr) < 1000:
    yhat = model()
    arr.append(yhat)

The neural network is trained separately for each of the predicted values and adds a value to the array arr.

arr - array for input.

res - result which consists of arr array and 22 predicted values via ARIMA.

Please check if the code for the neural network is written correctly.
Is it possible to somehow improve the forecast, having previously processed this graph? Any ideas on this?
Or such data is not predicted?
I tried to predict 10 values instead of 22, and the result is still very bad.


texts: Does "feature importance" depend on the model type?

texts:
I was working on a small classification problem (breast cancer data set from sklearn), and trying to decide which features were most important to predict the labels. I understand that there are several ways to define "important feature" here (permutation importance, importance in trees ...), but I did the following: 1) rank the features by coefficient value in a logistic regression ; 2) rank the features by "feature importance" from a random forest. These don't quite tell the same story, and I'm thinking that a feature that might be "unimportant" in a linear model could be very discriminative in a non-linear model that can "understand" it.
Is that true in general? Or should "important" features (those that contribute most to a classification score) be the same across all types of models?


texts: Python sklearn PCA transform function output does not match


texts: Making predictions from keras with SciKit

texts:
Ive been thinking about combining some processes between keras and Sci-kit Learn and am looking to the this group to either validate my process or tell Im crazy. Im creating a simple Regression problem using 17 inputs like this:

Creating test/train here:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.25, random_state=7)

Building the network here:

model = Sequential()
model.add(Dense(34, input_dim=17, kernel_initializer='normal', activation='relu'))
model.add(Dense(17, kernel_initializer='normal', activation='relu'))
model.add(Dense(8, kernel_initializer='normal'))
model.add(Dense(1, kernel_initializer='normal'))

compiling the model with:

model.compile(loss='mean_squared_error', optimizer='adam')

fitting the model here:

model.fit(X_train,y_train,validation_data=(X_test,y_test), epochs=100, batch_size=10)

Now that Ive fit the model is the any reason I can use some of the SciKit functions and do the following?

Make predictions

y_pred = model.predict(X_test)

Assess the model results:

mse = mean_squared_error(y_test, y_pred)
rmse = sqrt(mse)
r2score = r2_score(y_test,y_pred)

Am I way off-base here?


texts: How can I explain the cause of different performances for two different LSTM models and improve the performance?

texts:
I've built two different models for Load Forecasting. Dataset has six features. The performance evaluation metric is the Mean Absolute Percentage Error(MAPE). Both models are based on LSTM. Here is the first model and performance.

model = Sequential()
model.add(LSTM(20, input_shape=(1,6),init= 'uniform', return_sequences=True)) #input shape (1,5) to (1,6)
model.add(LSTM(20,init= 'uniform', activation='relu'))
model.add(Dense(256,init= 'uniform', activation='relu'))
model.add(Dense(256,init= 'uniform', activation='relu'))
model.add(Dense(1,init= 'uniform', activation='relu'))

For 30 epochs:

TRAIN loss = 3.7549, TEST LOSS : 3.5419
Time Requered: 5.9089 minutes

Second model is stacked-LSTM. Here is the code:

model = Sequential()
model.add(LSTM(20, input_shape=(1,6),init= 'uniform', return_sequences=True)) #input shape (1,5) to (1,6)
model.add(LSTM(20,init= 'uniform', activation='relu',return_sequences=True))
model.add(LSTM(20,init= 'uniform', activation='relu',return_sequences=True))
model.add(LSTM(20,init= 'uniform', activation='relu',return_sequences=True))
model.add(LSTM(20,init= 'uniform', activation='relu'))
model.add(Dense(1,init= 'uniform', activation='relu'))

For 30 epochs:

TRAIN loss =  5.0825, TEST LOSS : 5.1821
Time Requered: 5.7295 minutes

In both case I have compiled models in this way:

model.compile(loss='mean_absolute_percentage_error', optimizer='adam', metrics=['acc'])

Definitly, second model is working better. But why second model(stacked-lstm) is not working well? And how can I improve better than that? For the first model after 150 epochs gradient vanishes whene loss is around 3.01. I'm aiming for less than 2.


texts: Incorporate luck in statistical modelling

texts:
I was wondering if it is possible (and if yes, how is it done) to incorporate a luck component in statistical models.

So let’s assume I’d perform a regression on the goal difference between two teams in order to model the outcome of a sports match. How could I also model a random luck component that would make the underdog draw or win in certain rare cases?

And what if I took it a step further and tried to model the match outcome using neural networks? Is is possible to include luck in this case?


texts:
I work in the Oil & Gas industry.

I have been trying to build a ts forecasting model with covariates, and the model R code is as follows:

#Getting R libraries:
library(readxl)
library(ggplot2)
library(forecast)
library(timeSeries)
library(tseries)
library(MTS)

#Create a time series object:
myts <- ts(dataset, start = c(2005,1), end = c(2019,12), frequency = 12)

#Illustrate out of sample forecasting with covariates, splitting the data:
train <- window(myts, end = c(2018,12))
test <- window(myts, start = c(2019,1))

#Fitting the time series forecasting model:
covariates <- c("Income","Prices","Sites","Vehicles")
fit <- auto.arima(train[,"Volumes"], xreg = train[,covariates])

#Forecasting from test data:
mytsfcast <- forecast(fit, h = 6*12, xreg = test[,covariates])

autoplot(mytsfcast)

However, I have been trying to forecast the retail volume sales 12,24,36, etc months out. The model only generates the following result:

Model fit results:

Please may I ask that you kindly advise on how I can get my model to forecast beyond end = c(2019,12). I am missing something?


texts: Word representation that gives more weight to terms frequent in corpus?

texts:
The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I use One-class SVM to detect outliers. Interesting enough the tf-idf performs worse than a simple count-vectorizer. First I was confused, but later it made sense to me, as tf-idf discounts attributes that are most indicative of a valid document. Therefore I was thinking of a new approach that would weight words that always appear in documents more, or rather assign a negative weight for the absence of such words. I have preset dictionary of words, so there is no worry that irrelevant words such as(is, that) will be weighted.
Do you have any ideas about such representations? The only thing I could imagine would be subtracting the document frequency from the attributes that are zero in a certain document.


texts: Which machine learning algorithm to choose?

texts: I want to choose an unsupervised algorithm which learns to predict $n$ outputs from the data, for eg. 4 coordinates (pixels) in an image. What algorithm should I choose? I think it's a 2-class classification to divide the set of points in the image as belonging to output (1) or not (0), maybe logistic regression to give the probabilities of a point being the output point. But I am confused because classification algorithms are part of supervised algorithms where we have labelled data. Should I use clustering to find 2 groups of points that can be output or not? Maybe anomaly detection to find the 4 odd points out?


texts: Predicting the Evolution of a Vector with a Neural Network (RNN?)

texts:
I want to predict the evolution of a 2D sequence (third dimension being time) and use the predicted sequence (along with several previously predicted sequences and some scalar values changing with time) to predict the next 2D sequence, and ultimately use that prediction to predict the following sequence in time.

Example of a 2D time-slice of the data:

I want to train my neural network to predict a new 2D "profile" (a time-slice of a 3D profile) based on the last ~3 profiles and a time-varying scalar parameter. What neural network architecture(s) would be capable of predicting the next 2D profile in time? I suspect RNNs, but which sub-models would work for this (e.g. seq2seq, NMT, etc.)?

Help would be greatly appreciated!


texts: Find non overlapping area between two kde plots in python

texts:
I was attempting to determine whether a feature is important or not base on its kde distribution for target variable. I am aware how to plot the kde plot and guess after looking at the plots, but is there a more formal doing this? Such as can we calculate the area of non overlapping area between two curves?
When I googled for the area between two curves there are many many links but none of them could solve my exact problem.
NOTE:
The main aim of this plot is to find whether the feature is important or not. So, please suggest me further if I am missing any hidden concepts here.
What I am trying to do is set some threshold such as 0.2, if the non-overlapping area > 0.2, then assert that the feature is important, otherwise not.
MWE:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = sns.load_dataset('titanic')

x0 = df.loc[df['survived']==0,'fare']
x1 = df.loc[df['survived']==1,'fare']

sns.kdeplot(x0,shade=1)
sns.kdeplot(x1,shade=1)

Output

Similar links

Fill area of overlap between two normal distributions in seaborn matplotlib
Python: Overlap between two functions (PDF of kde and normal)
Fill area-between two curves in python


texts:
I have seen 'seed' parameter in many codes of Keras.
I have a code example here:

validation_generator = data_generator.flow_from_directory(
    train_data_dir,
    target_size = (img_width, img_height),
    batch_size = batch_size,
    shuffle = True,
    class_mode = 'categorical',
    seed = 42,
    subset = 'validation')

Can you explain what is it? And when to use it?


texts: Intuition of "Head" in Attention models (Transformer)?

texts: I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!


texts: Is it good practice to always remove highly correlated variables?

texts:
1- Would it always be beneficial to remove highly correlated features prior to training a model? If not, why not.

2- Would you perform One Hot encoding where applicable, prior to removing highly correlated features ? (taking into account a dummy variable may be highly correlated with another variable)

Thanks


texts: Multilabel Classification - increasing the confidence

texts:
I have a basic multilabel topic classifier(Tfidf vectorizer with OneVsRest Classifier) built on some customer reviews. I observed that there are some classes with right features but it still predicts with very low confidence.

What is the reason for such a classification in general?
How do I boost the confidence?


texts: Scalar input to neural network whose existence is conditional

texts:
I am developing a neural network to determine if comments posted on a blog-type website are appropriate or not (to reject spam, poorly written comments, etc). I use Keras with Tensorflow to achieve this by defining a number of scalar features such as message length, number of words, fraction of words that are in English dictionary... All are normalized by appropriate quantities. These numbers are the input to a neural network defined as follows:
inp = Input(shape=(n_inputs, ))
hidden_1 = Dense(hidden_size, activation='relu')(inp)
hidden_2 = Dense(hidden_size, activation='relu')(hidden_1)
out = Dense(num_classes, activation='softmax')(hidden_2)
model = Model(inputs=inp, outputs=out)

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=[f1_score])

I would like to add some inputs that do not always exist. For example, a comment may be posted after another comment from the same user, and in that case the delay between the comment and the previous one would be a relevant quantity to consider as a user may be spamming by sending quickly several comments in a row.

How can I use this quantity that does not always exist?
Would it be good to use two quantities: a binary indicating if the
post is not the first one from that user, and a second number
indicating the delay from the previous comment, but filled with an
arbitrary value (0, 1, 0.5, the mean delay on all my dataset?) if
the post is the first one?

Edit
I think this is related to the problem of missing data, see this thread.


texts:
I am trying to use apriori() on this dataset.
After cleaning it, I made all attributes categorical with as.factor(). Then I uset these instruction:

chocolate_rules <- apriori(chocolateApriori, parameter=list(minlen=3, supp=0.1, conf=0.7), appearance=list(default="lhs", rhs=c("RatFactor=1", "RatFactor=2", "RatFactor=3", "RatFactor=4", "RatFactor=5")))

where RatFactor is categorial rounded "Rating" from the original dataset (and has only 5 possible values).

It seems to work, but when I call "chocolate-rules" the result is set of 0 rules.
Can you explain me why? Or can you help me getting another result?


texts: Machine learning approach for predicting set members

texts:
Say I have a large training dataset containing sets of 40 items each, and each item in the set is unique (so every training input is a set $S=\{i_1, i_2, ..., i_{40}\}$), and there are more than 40 unique items that can be part of a set.

I would like to be able to predict which items are a probably member of a set, given some incomplete set. So let's take the following example:

Training data:
$S_1 = \{1,2,3\}$,
$S_2 = \{3,4,5\}$,
$S_3 = \{6,7,8\}$
Say I then have an input $S_4 = \{3\}$, I would want the approach to give back that 1, 2, 4 and 5 are more probable set members than 7, 8. Ideally with some probability value.

I've considered the following:
Using the apriori algorithm to learn some association rules. I wasn't sure how to interpret the support or lift as a probability of set membership.

Training a Multilayer perceptron on the input (probably one-hot encoded) to learn weights corresponding to the various input items. However, if I were to simply give the 40-item sets as input and output then the network would just learn to copy the input, giving no information about possible other set members. I've thought about giving all variations of the 40 item set as input, with the 40 item set as output, but this would result in $2^{40}$ possibilities per input which would be massive.

Is there some machine learning approach or data structure that could help in this situation?


texts:
Does anyone know what does the num_negatives mean? What is it for?
def get_train_samples(train_mat, num_negatives):
    user_input, item_input, labels = [], [], []
    num_user, num_item = train_mat.shape
    for (u, i) in train_mat.keys():
        user_input.append(u)
        item_input.append(i)
        labels.append(1)
        # negative instances
        for t in range(num_negatives):
            j = np.random.randint(num_item)
            while (u, j) in train_mat.keys():
                j = np.random.randint(num_item)
            user_input.append(u)
            item_input.append(j)
            labels.append(0)
    return user_input, item_input, labels

This occurs here:
user_input, item_input, labels = get_train_samples(train_mat, NUM_NEGATIVES)

And here:
hist = model.fit([np.array(user_input), np.array(item_input)], np.array(labels),
                 epochs=EPOCHS, verbose=VERBOSE, shuffle=True, batch_size = BATCH_SIZE,
                 validation_data=([np.array(val_user_input), np.array(val_item_input)]),
                  callbacks=CALLBACKS)


texts: Why might trees work so much better than boosting classifiers?

texts:
I am predicting 10 classes label encoded using scikit-learn with 6 factors, 1.2M cases.
DecisionTreeClassifier
RandomForestClassifier
ExtraTreesClassifier
give accuracies (and precision and recall) of 0.9

AdaBoostClassifier
GradientBoostingClassifier
give accuracies of 0.2

Any pointers on the huge discrepancy?

(I am doing gridsearchcv).
Code:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

def output_metrics():
    from sklearn.metrics import accuracy_score, precision_score, recall_score
    print("Accuracy:",accuracy_score(y_test, y_pred))
    print('Precision', precision_score(y_test, y_pred, average=None).mean())
    print('Recall', recall_score(y_test, y_pred, average=None).mean())

from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import GridSearchCV

tree_para =  { 'n_estimators': [16, 32] }
clf = GridSearchCV(AdaBoostClassifier(), tree_para, cv=5)
model= clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)

output_metrics()


texts: What features from sound waves to use for an AI song composer?

texts:
I am planning on making an AI song composer that would take in a bunch of songs of one instrument, extract musical notes (like ABCDEFG) and certain features from the sound wave, preform machine learning (most likely through recurrent neural networks), and output a sequence of ABCDEFG notes (aka generate its own songs / music).

I think that this would be an unsupervised learning problem, but I am not really sure.

I figured that I would use recurrent neural networks, but I have a few questions on how to approach this:
- What features from the sound wave I should extract so that the output music is melodious?
- Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?
- Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?


texts: How to build a symmetric similarity model on top of embeddings?

texts:
I have two equal length vectors that come out of two identical embedding layers.

I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's plausible that different coordinates are dependent wrt overall similarity). I want to learn this using examples of good and bad pairs, without actually learning the initial embedding.

What I'd like to do is to somehow combine the two vectors using another layer, and then connect this layer to an output layer to get the final decision (similar/not similar).

The trivial way is to add another layer, and fully connect the two concatenated embedding vectors to this new layer. The downside is that the model is not symmetric, which makes the search space bigger than it should be.

Is there a better way?

Ideas I have so far:

Cross product the two vectors, then do the learning on the $n \times n$ output (easy enough, but might have too many weights to learn).
Create two symmetrical neural nets, where the weights are symmetrical (as if mirror image of each other). Intuitively this can allow arbitrary combinations of coordinates from both sides to feed the next layer (how can equalizing the weights be done in keras?).

Is there a better way?

Bonus points: Beyond symmetry, how does one instigate transitive and reflexive properties? By just adding a ton of trivial $(x, x)$ examples?


texts: Interpreting the results based on Granger Causality test


texts: Machine learning goal: given a population of 100,000 students, predict a group of 3,000, and minimize the median grade of that group

texts:
In other words, I am looking to predict students that will fail out of school before it happens. The data includes socioeconomic status and other related variables.
I have tried an XGB binary classification (both tree and forest), but the problem is that it doesn't penalize severely wrong answers (predicting that a student will be in the bottom 3% in terms of grades, but they're actually A+ students). The result is that the average grades of the predicted students is quite low, but the median grades aren't actually that bad - there are a few extremely bad students that pull down the average but not the median.
I have tried a XGB regression (both tree and forest), but the problem is that I can't get the model to focus on the bottom 3%. It seeks to reduce error for all predictions. I couldn't care less about telling the difference between an A student and a B student, I only need to consistently identify the bottom 3%ile.
I was thinking that perhaps this could lend itself to reinforcement learning instead of supervised, but I know nothing about reinforcement... WOuld it be possible to make a reinforcement model where the goal is to minimize the median grades of the 3% of students predicted? Or are there any other machine learning techniques that would work?


texts: Machine learning. Input: set of distributions, Output: distribution

texts:
I have a set of features where typical machine learning techniques do not work very good. All features have very different distributions, some heteroscedasticity is also present.

The distribution shapes still convey a lot of information. For example one can see that if the value of a feature is below a threshold highly likely the target variable can never be higher than a value.

I know about bayesian estimates but they do not work well with categorical features (some are) and also they require that all features independently contribute to the target variable. This does not hold true for my features.

Do you know any other machine learning technique that can learn from the same of the features (or in other words from their distribution) and then output a distribution of how the target variable might look like?

Thanks a lot
Alex


texts: What is the num_initial_points argument for Bayesian Optimization with Keras Tuner?

texts:
I've implemented the following code to run Keras-Tuner with Bayesian Optimization:
def model_builder(hp):

    NormLayer = Normalization()
    NormLayer.adapt(X_train)

    model = Sequential()

    model.add(Input(shape=X_train.shape[1:]))

    model.add(NormLayer)

    for i in range(hp.Int('conv_layers',2,4)):
        model.add(Conv1D(hp.Choice(f'kernel_{i}_nr',values=[16,32,64]), hp.Choice(f'kernel_{i}_size',values=[3,6,12]), strides=hp.Choice(f'kernel_{i}_strides',values=[1,2,3]), padding="same"))
        model.add(BatchNormalization(renorm=True))
        model.add(Activation('relu'))
        model.add(MaxPooling1D(2,strides=2, padding="valid"))

    model.add(Flatten())
    model.add(Dropout(hp.Choice('dropout_flatten',values=[0.0,0.25,0.5])))

    for i in range(hp.Int('dense_layers',1,2)):
        model.add(Dense(hp.Choice(f'dense_{i}_size',values=[500,1000])))
        model.add(Activation('relu'))
        model.add(Dropout(hp.Choice(f'dropout_{i}_others',values=[0.0,0.25,0.5])))

    model.add(Dense(hp.Choice('dense_size_last',values=[100,200])))
    model.add(Activation('relu'))

    model.add(Dense(2))
    model.add(Activation('softmax'))

    opt = Adam(learning_rate=lrn_rate_init)

    earlystop = EarlyStopping(monitor='val_loss',patience=8,restore_best_weights=True)

    model.compile(loss='categorical_crossentropy',optimizer=opt,metrics=['accuracy'])

    return model

tuner = BayesianOptimization(model_builder,objective='val_loss',num_initial_points=??,max_trials=tuner_trials,directory='BayesianOptimization/',project_name='BayesianOptimization')

What do the num_initial_points argument does exactly and what should I set it to in my case?
Reading the documentation I see the description

The number of randomly generated samples as initial training data for Bayesian optimization

but not being an expert I don't exactly get what it means and how it will impact the optimization process.


texts:
I'm training a model with CV and then I'm testing the predictions on a new test set.
Am I doing the right thing or is it necessary to test the predictions on a new dataset using Cross-Validation?
Thank you!


texts: Why is torch complaining about an empty tensor?

texts:
The structure_loss method is supposed to return a loss for ground truth vs predicted masks:
import numpy as np
import torch
import torch.nn.functional as F
import torch.nn as nn
def structure_loss(pred, mask):
    weit = 1 + 5 * torch.abs(nn.functional.avg_pool2d(mask, kernel_size=31, stride=1, padding=15) - mask) # here
    wbce = F.binary_cross_entropy_with_logits(pred, mask, reduce='none')
    wbce = (weit * wbce).sum(dim=(2, 3)) / weit.sum(dim=(2, 3))

    pred = torch.sigmoid(pred)
    inter = ((pred * mask) * weit).sum(dim=(2, 3))
    union = ((pred + mask) * weit).sum(dim=(2, 3))
    wiou = 1 - (inter + 1) / (union - inter + 1)
    return (wbce + wiou).mean()

a = torch.Tensor(np.random.uniform(0,1, (352, 352)))
b = torch.Tensor(np.random.uniform(0,1, (352, 352)))
print(structure_loss(a,b))

However, I test it on  dummy data I get
 File "binary_cross_entropy_for_random_array.py", line 8, in structure_loss
    weit = 1 + 5 * torch.abs(nn.functional.avg_pool2d(mask, kernel_size=31,
stride=1, padding=15) - mask) # here
RuntimeError: non-empty 2D or 3D (batch mode) tensor expected for input

Process finished with exit code 1

How to generate tensors suitable for this method?


texts: Determining which entries in input vector have the greatest effect on output vector

texts:
I'm looking for a solution in Python. I'm doing a project where I've built a densely packed neural network which takes a 48-dimensional input vector and the last layer is a 2-dimensional linear output. I'm looking for an algorithm that can determine how much each scalar in the input vector contributes to the output vector.

Specifically, my input vector is 24 consecutive pairs of position and momentum. For example the input would be $(x_1,v_1,x_2,v_2,...,x_{24},v_{24})$ and my output is the next position and momentum pair $(x_{25},v_{25})$. I've hypothesized that the more recent (higher index) pair should impact the output more. Is there any technique that I could use to quantitatively verify this?


texts: Similarity between 2 statistical distributions

texts:
Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in shapes(like one is symmetric and the other is skewed) etc. and gives similarity between [0,1]?
I am using Hedges' index for the same but it does not give a similarity index between 0 and 1. It can be greater than 1 as well, so it is difficult to interpret it.
Also, no pattern of the data is known beforehand, if it helps in any way for the answer.


texts: Is it possible to use LSTM for time series forecasting for future months with test data having only NaN values?

texts:
my dataset is a univariate time series with one column as months, other column having demands for the corresponding months. My test dataset has NaN vals only for all the months. Can LSTM be used in this case and if so, how?
I tried using LSTM but only got NaN values. If you could help me out, that'd be great. Thanks!


texts: Transitioning to Data Science from Actuarial Science

texts:
The title says it all, I want to transition from actuarial science to data science.
My background: I have a BS and MS in pure mathematics, both with high GPA (3.94) and graduation with honors. My areas of study were highly theoretical (Algebraic Geometry), though I took some statistics courses, computer science, and economics as well. I am savvy with R, Python, Sql, etc. Unlike some pure-mathy types I do communicate well and I do well at translating business questions into quantitative problems.
My motivation: I'm not really motivated by money, as most know actuarial science is already fairly high paying. I think I would enjoy a career in data science more than actuarial science. From the research I have done, the skill set and projects in the data science domain seem more intriguing to me. I have always been intrigued by deep learning, machine learning, etc. but since it wasn't directly relevant to my studies in school or my career I have never built a strong background in these areas.
My Attempts: I have sent out some applications for data science positions but haven't had any luck yet. I am located in the Seattle area and being a tech-y city I imagine there are a lot of competitive applicants for any jobs that get posted. I usually get a "thanks but no thanks" response within a week.
The question: How can I make myself more competitive and at least land some interviews. Since I have only worked in actuarial science (3 years experience), my resume and work history are only actuarial (however my last title within the actuarial domain was actually "Actuarial Data Scientist", we were making web applications for insurance clients that let them view models we built for policyholder behaviour and other things of interest.)  Most of the job postings request NLP, deep learning, machine learning experience, to which I have little exposure. I have been self-studying with PyTorch. What types of things can I be self-studying to make my skill set more desirable, and how can I communicate these skills through my resume? It seems uncommon to have a resume section titled "things I have learned in my spare time".
Obviously there are loads of things I could be studying and learning, so I scan job postings to try and narrow down the most commonly requested skills. What would be the best things I could start with, and then how could I incorporate these skills into my resume or cover letter?


texts: About 1M rows of data. Should I restrict myself to few columns as well?

texts:
I'm trying to build a predictive model from about 1 million rows of data. My goal is to predict a certain numerical value.

I have the intuition that I should use very few numerical binary columns so I don't get data points that are too separated, a.k.a., the curse of dimensionality. Is this true? Besides, is it the same for numeric columns?


texts: Dealing with unbalanced error rate in confusion matrix

texts:
Here is the confusion matrix I got when I was playing with Forest Type Cover Kaggle dataset : Link.

In the matrix, light color and higher numbers represent higher error rates, so as you can see, lots of mis-classification happened between class 1 and 0 .

I wonder what kind of methods I can use to reduce these two error rates though improvements have been made through combing two classifiers, Random Forest and Extra Tree. Will stacking help in this case?

Data can be found on
https://www.kaggle.com/c/forest-cover-type-prediction/data


texts: emission probability using hmmlearn package in python

texts:
I am learning hmm and try to implement it in  Python hmmlearn package(http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples).

However I am not quite understand what the documentation says:

  Classes in this module include MultinomialHMM, GaussianHMM, and
  GMMHMM. They implement HMM with emission probabilities determined by
  multimomial distributions, Gaussian distributions and mixtures of
  Gaussian distributions.

Does this means when we try to estimate the model given an observation sequence, the  emission probability can only be these three kind? what if I want to type in the emission probability by meself?


texts: Why is the accuracy of a LinearSVC not the same as the SDGClassifier?

texts:
I'm fine tuning parameters for a linear support vector machine. There are multiple ways to do it, but I wanted to compare LinearSVC and SDGClassifier in terms of time. I expected the accuracy score to be the same but, even after fine tuning with GridSearchCV, the score of the LinearSVC is lower. I tried changing up parameters many times, but the maximum with LinearSVC I can get is 41.176 versus 41.503 of SDGClassifier. Why?
The code:
    class SVMSentiment(Base):
"""Predict sentiment scores using a linear Support Vector Machine (SVM).
Uses a sklearn pipeline.
"""
def __init__(self, model_file: str=None) -> None:
    super().__init__()
    # pip install sklearn
    from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer
    from sklearn.linear_model import SGDClassifier
    from sklearn.svm import SVC, LinearSVC
    from sklearn.pipeline import Pipeline

    self.pipeline = Pipeline(
        [
            ('vect', CountVectorizer()),
            ('tfidf', TfidfTransformer()),
            #('tfidf', TfidfVectorizer()),
            ('clf', LinearSVC( loss='hinge',
              penalty='l2', max_iter = 10,

             #SGDClassifier(
                #loss='hinge',
               # penalty='l2',
              #alpha=1e-3,
              #  random_state=42,
             #max_iter=100,
                #learning_rate = 'optimal',
                #tol=None

            )),
        ]
    )

def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    from sklearn.model_selection import GridSearchCV
    from sklearn.svm import SVC
    from sklearn.linear_model import SGDClassifier
    from sklearn import svm
    from sklearn import preprocessing
    from sklearn.preprocessing import LabelEncoder, OneHotEncoder
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.svm import LinearSVC
    train_df = self.read_data(train_file, lower_case)
    param_range = [0.001, 0.01, 0.1, 1, 10, 100]
    parameters = {
        # 'vect__ngram_range': [(1, 1), (1, 2)],
        'tfidf__use_idf': (True, False),
        #'clf__alpha': [0.0001, 0.001, 0.01, 1, 10, 100],
        'clf__max_iter': [10, 100, 1000],
        'clf__tol': [0, 0.0001, 0.001, 0.01],
        'clf__loss':['hinge'],
        'clf__penalty': ['l2'],
        'clf__C': param_range

    }

    parameters1 = {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['linear'], 'clf__tol' : [0, 0.01]

                  }
    lr = LinearSVC()
    print(lr.get_params().keys())
    gs_clf = GridSearchCV(self.pipeline, parameters, cv=5, n_jobs=-1)
    gs_clf = gs_clf.fit(train_df['text'], train_df['truth'])
    print(gs_clf.best_score_)
    for param_name in sorted(parameters.keys()):
        print("%s: %r" % (param_name, gs_clf.best_params_[param_name]))

    # estimator_svm.best_score

    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)

    test_df['pred'] = learner.predict(test_df['text'])
    return test_df


texts:
I want to perform drift detection on data with multiple input values (x0, x1, x2, x3). I'm using an adaptive window algorithm found from sci-kit found here.

Doing this

from skmultiflow.drift_detection.adwin import ADWIN
adwin = ADWIN()
adwin.add_element(np.array([1, 2, 3, 4]))

Results in ValueError: setting an array element with a sequence.

But doing this

from skmultiflow.drift_detection.adwin import ADWIN
adwin = ADWIN()
adwin.add_element(np.array([1])

works just fine.

Any idea how to do the first thing?

My current solution is to just use 4 different drift detectors, but I would like to use one.


texts: How does one use activation function with greater than [-1;1] range for binary classification?

texts:
In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes.

They explain (§ 4.5) why an activation function that only reach target values asymptotically may result in drawbacks : instability and ouptu saturation.

On of their proposed solution is to (carefully) use activation function that have a range greater than [-1,1]. For that purpose they propose to add a linear term to the standard tanh activation function, or add a multiplicative constant >1.

This make some sense, however, I am wondering how to use that ouptut. I am used to rescale [0,1] scores to calibrate models in probability. But I don't know what I would do with a 1.2 score. Perform as usual with a monotonous function to transform score in probabilities ? Add another layer ?


texts:
From my limited dabbling with data science using R, I realized that cleaning bad data is a very important part of preparing data for analysis.

Are there any best practices or processes for cleaning data before processing it? If so, are there any automated or semi-automated tools which implement some of these best practices?


texts: How to calculate classification accuracy with confusion matrix?

texts:
I have Train and Test data, how to calculate classification accuracy with confusion matrix ? Thanks

@attribute outlook {sunny, overcast, rainy}
@attribute temperature {hot, mild, cool}
@attribute humidity {high, normal}
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}

Train:

1   sunny       hot     high    FALSE   no
2   sunny       hot     high    TRUE    no
3   overcast    hot     high    FALSE   yes
4   rainy       mild    high    FALSE   yes
5   rainy       cool    normal  FALSE   yes
6   rainy       cool    normal  TRUE    no
7   sunny       cool    normal  FALSE   yes
8   rainy       mild    normal  FALSE   yes
9   sunny       mild    normal  TRUE    yes
10  overcast    mild    high    TRUE    yes
11  overcast    hot     normal  FALSE   yes
12  rainy       mild    high    TRUE    no

Test:

overcast    cool    normal  TRUE    yes
sunny       mild    high    FALSE   no

Rules found:

(humidity,normal), (windy,FALSE) -> (play,yes) [Support=0.33 , Confidence=1.00 , Correctly Classify= 4, 8, 9, 12]
(outlook,overcast) -> (play,yes) [Support=0.25 , Confidence=1.00 , Correctly Classify= 2, 11]
(outlook,rainy), (windy,FALSE) -> (play,yes) [Support=0.25 , Confidence=1.00 , Correctly Classify= 3]
(outlook,sunny), (temperature,hot) -> (play,no) [Support=0.17 , Confidence=1.00 , Correctly Classify= 0, 1]
(outlook,sunny), (humidity,normal) -> (play,yes) [Support=0.17 , Confidence=1.00 , Correctly Classify= 10]
(outlook,rainy), (windy,TRUE) -> (play,no) [Support=0.17 , Confidence=1.00 , Correctly Classify= 5, 13]


texts: Creating a training dataset from analytical solution

texts:
I am currently redesigning an inverse problem on an experimental technique, but I am having doubts about how to create a training dataset. Here is the problem I am trying to solve:
I have already created a model in order to solve the forward problem, such that if I input $x_1,x_2,...,x_n$, the model solves a generalized eigenvalue problem and returns the expected output $y_1,y_2,..,y_m$.
Thus, what I am trying to do is to generate random $(\vec{x},\vec{y})$ pairs in order to create a training data set that will allow me to solve the inverse problem. Considering this, I am confused as if the random $(\vec{x},\vec{y})$ pairs should be considered as synthetic data (since they come from the actual analytical solution). If so, should I generate the training data by following a probability distribution and adding noise?
Thanks in advance for the help!


texts: Is over fitting okay if test accuracy is high enough?

texts:
I am trying to build a binary classifier. I have tried deep neural networks with various different structures and parameters and I was not able to get anything better than

Train set accuracy : 0.70102
Test set accuracy : 0.70001

Then I tried machine learning algorithms such as KNN and Decision Trees etc. And I found that Random forest Classifier from Scikit-learn with n_estimators=100 gave me

Train set accuracy : 1.0
Test set accuracy : 0.924068

I tried adjusting other parameters such as max_depth, criterion But the decrease in training set accuracy also caused the test set accuracy to drop. Like

Train set accuracy : 0.82002
Test set accuracy : 0.75222

My question is, is this

Train set accuracy : 1.0
Test set accuracy : 0.924068

acceptable ? Even thought the model is over fitting, the test set accuracy is better.