texts:
I am looking a python lib named deap, but stuck at beginning.

The first paragraph says:

  The problem is very simple, we search for a 1 filled list individual.

What does 1 filled list means? Search a 1 filled list, from where? individual list or individual 1?

Google One Max Problem only gives some information which seems to be useful:

There is a Max One Problem

I can understand this, but is it same as One Max Problem? If so, I have a question that why need evolutionary algorithm to "evolve" our population until eventually the target emerges. ? If I am a Medical Researcher, I should already have the entire DNA(gene list) , what I need to do is just search in that list, not evolve a random list.
There is a Maximum_satisfiability_problem

This is understandable too, seems relate to One Max Problem , another saying?


texts: How to understand clearly the feature importance computing in random forest model

texts:
I want to know the details of feature importance computing of random forest with gini index. I refer to this Chinese blog:https://www.jianshu.com/p/7a876bb876b9. But I don't understand clearly these mathematical forms in blog.

for example

As the node m after twos branches(Il, Ir) splitting, the gini index is computing as following

And  feature Xj in the decision ith tree ,and be in nodel of M set, and the feature importance of Xj in the ith tree is computing as following:

And I confusing about M set, what's the meaning of M set? I think that the  feature Xj in ith tree just one node not a M set? If you guys have detailed computing method, please tell me!


texts: Recognizing circled numbers on a piece of paper

texts: I've built a handful of CNN using tensorflow, keras, pytorch for recognizing text/number/objects in an image. What I'm trying to figure out how to do now is how to recognize numbers on a piece of paper that are circled by a pen or pencil. So on a piece of paper, there are two rows of 6 sets of numbers, and someone will go through and 2-3 sets of numbers, and I'd like to be able to take a picture or scan the paper, and have a model be able to recognize which number sets were circled. What would be the best way to start going about this?


texts: Determine useful features for machine learning model

texts:
I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:

What quantitative metrics can I use to determine that a feature will be useful to the learning model?

I have been comparing the distribution of the target mean over the feature groups, to the target mean of the dataset as a whole. For example, take a binary feature X and a binary target. Let's say the target has a mean of 0.10 when taken over the entire dataset.

To analyze the feature X, I take the target mean for each group within feature X.

mean (X=0) = 0.07
mean (X=1) = 1.15

In this way, I can observe the effect of a feature on the target.

I know that there must be some stronger metrics which people use to determine the strength of a feature. In school I used p tests to determine the statistical significance of a variable. Is there an analog in DS/ML?


texts: Cost sensitive learning and class balancing

texts:
I am facing a classification problem with classes that are really imbalanced (more or less 1% of positive cases). In addition, the "cost" of a False Negative (FN) is much higher than the cost of False Positive (FP).
Considering so, I decided to insert the weights into my classification model. Which is the best way to compute these weights? Ideally, the weights have to take into account both the data unbalancing and the miss-classification cost.
I am not interested in putting a 0/1 label to each record, but just in ordering the test dataset according to the output score. The idea is to contact the records with the highest score to offer a product.
I would like to use these weights using, for example, the sample_weights parameter available in most of scikit-learn classification algorithm (here the documentation).
Is it a good idea to oversample/downsample the data and then use the weight only to control the miss-classification cost? Or would it be better to use weights that are able to consider the whole situation? Are there any known ways to compute these weights?


texts: best way to regularize gradient boosting regressor?

texts:
i am testing gradient boosting regressor from sklearn for time series prediction on noisy data (currency markets).

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html

surprisingly, the the gradient boosting regressor achieves very high accuracy on the training data - surprising because the data is so noisy. however, it performs poorly on the test set.

this is clearly a case of overfitting, so i'm wondering what parameters i can change to regularize the gradient boosting regressor.

so far i've tried max_depth, reducing it to 1 (from the default of 3). this seems to work pretty well in increasing accuracy on the validation set.

does anyone know what other parameters i could tweak, to improve performance on the validation/test set? thanks


texts: I'm working on a particle physics dataset and want to know what libraries I would need to implement GANs and other generative algorithms like  in Python.


texts: Difference between subgradient SVM and kernel SVM?

texts:
What is the difference between subgradient svm and kernel svm?

From my understanding subgradient svm is a linear classifier that uses hinge loss and kernel svm  uses some kernel function for non linear classification. I was wondering, if subgradient is only a linear classifier, could I use a tree of non linear svms to separate non linear data? I would essentially do binary classification separate 1 class vs the rest then the child node of the tree would work on the rest and separate the next class from the rest and so on.  Any general idea or feedback would be great.


texts: Keras multi-gpu seems to heavily load one of the cards


texts: How to fill NaN based on the previous row that contains a specific value

texts:
I would like to fill the missing data using the value from the previous trading day of the same stock. In this example, the AAPL stock should be 100.
I've tried with fillna but I am not able to pick a specific row based on the stock.
My data:
date       stock   price
22/12/20   MSFT    87
22/12/20   AAPL    99
22/12/20   FCA     81
23/12/20   MSFT    90
23/12/20   AAPL    100
23/12/20   FCA     80
24/12/20   MSFT    91
24/12/20   AAPL   *NaN*
24/12/20   FCA     78


texts:
My data consists of multiple NHL seasons where the total number of goals scored by each player is kept track.

I intend to compare the distribution of goals scored between each season.

My thinking is that I need to normalize goals scored before I can properly compare distributions since not every season has had the same number of games played.

Is it in fact necessary to normalize the data in order to fairly compare the distributions?


texts: ValueError: y should be a 1d array, got an array of shape (285, 30) instead

texts:
I am using this data set below and I am trying to find the support vector machine of the data set. Also
I have my code and error below as well.
http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer

import numpy as np
from sklearn import svm,datasets
breastcancer = datasets.load_breast_cancer()
#print(breastcancer)
everydata = breastcancer.data
#print(everydata)
everytarget = breastcancer.target
traindata = []
traintarget = []
testdata = []
testclasses = []
#Class 0 data separtion
for i in range(0,140):
    traindata.append(everydata[i])
    traintarget.append(everytarget[i])
for i in range(140,212):
    testdata.append(everydata[i])
#Class 1 data separation
for i in range(212,357):
    traindata.append(everydata[i])
    traintarget.append(everytarget[i])
for i in range(357,569):
    testdata.append(everydata[i])
traindata = np.concatenate((everydata[:140, :],everydata[212:357, :]),axis=0)
traintarget = np.concatenate((everydata[:140],everydata[212:357]),axis=0)
testdata = np.concatenate((everydata[140:212, :],everydata[357:569, :]),axis=0)
print(len(traindata))
print(traintarget)
print(testdata)
dd = svm.SVC(kernel='linear')
dd.fit(traindata,traintarget)
decide = dd.predict(testdata)
print(decide)

Why am I getting this error is my concatenation incorrect. The results are supposed to output 0s and 1s at the end.


texts: Implementing PatterNet in Python as it is in MATLAB

texts:
NOTE: This question was first posted in Cross Validated Stack Exchange but I was instructed to move it off that website as it was not a good fit.
I am new in implementation of machine learning, neural networks in python. I am trying to implement patternet in python as it is in MATLAB; more specifically to keep the network configuration settings. Below is what I get when I use paternet(10):
    Neural Network

              name: 'Pattern Recognition Neural Network'
          userdata: (your custom info)

    dimensions:

         numInputs: 1
         numLayers: 2
        numOutputs: 1
    numInputDelays: 0
    numLayerDelays: 0
 numFeedbackDelays: 0
 numWeightElements: 230
        sampleTime: 1

    connections:

       biasConnect: [1; 1]
      inputConnect: [1; 0]
      layerConnect: [0 0; 1 0]
     outputConnect: [0 1]

    subobjects:

             input: Equivalent to inputs{1}
            output: Equivalent to outputs{2}

            inputs: {1x1 cell array of 1 input}
            layers: {2x1 cell array of 2 layers}
           outputs: {1x2 cell array of 1 output}
            biases: {2x1 cell array of 2 biases}
      inputWeights: {2x1 cell array of 1 weight}
      layerWeights: {2x2 cell array of 1 weight}

    functions:

          adaptFcn: 'adaptwb'
        adaptParam: (none)
          derivFcn: 'defaultderiv'
         divideFcn: 'divideind'
       divideParam: .trainInd, .valInd, .testInd
        divideMode: 'sample'
           initFcn: 'initlay'
        performFcn: 'crossentropy'
      performParam: .regularization, .normalization
          plotFcns: {'plotperform', plottrainstate, ploterrhist,
                    plotconfusion, plotroc}
        plotParams: {1x5 cell array of 5 params}
          trainFcn: 'trainscg'
        trainParam: .showWindow, .showCommandLine, .show, .epochs,
                    .time, .goal, .min_grad, .max_fail, .sigma,
                    .lambda

    weight and bias values:

                IW: {2x1 cell} containing 1 input weight matrix
                LW: {2x2 cell} containing 1 layer weight matrix
                 b: {2x1 cell} containing 2 bias vectors

My intention is not to create the exact net data structure, I would like to mainly implement the activation functions to different layers of my network as it is listed in above. Let's assume I can initialize the weights (network parameters).
I followed this post Getting Started with Deep Learning and Python as well as Multi-layer Perceptron to start. In the latter link, I learned that I can implement MLP by utilizing MLPclassifier from sklearn.neural_network library:
MLPClassifier(activation='relu', alpha=1e-05, batch_size='auto',
       beta_1=0.9, beta_2=0.999, early_stopping=False,
       epsilon=1e-08, hidden_layer_sizes=(5, 2), learning_rate='constant',
       learning_rate_init=0.001, max_iter=200, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
       solver='lbfgs', tol=0.0001, validation_fraction=0.1, verbose=False,
       warm_start=False)

However, its setting is not exactly the one that I am trying to have. Reading MLPclassifier document page did not help me further.
I would appreciate it, if I get to know how to implement it as such? I would be happy to be directed to any post or link which has already discussed this matter.


texts: what is the effect of initial weights on model training for different algorithms

texts: If I do a model training on a dataset with three different algorithms Logistic Regression (L1), SVM(S1) and a Neural Net(N1). If I train the models again with the same data set and same parameters used previously for each of the models, everything same except for the initial weights, and built 3 models Logistic Regression (L2), SVM(S2) and a Neural Net(N2). Training can be considered to be long enough. So now my question is, is L1 = L2, S1 = S2 and N1 = N2 ?


texts: Could I add a one hot encoding to each feature representing "has data" versus "has no data"

texts:
I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and this feature doesn't have data for each of my features. Would a CNN be able to take advantage of this or should I just use another method to approximate the result.

Note The reason why I'm looking for an alternative is that I have a very small dataset and I assume extrapolating would insert to great of a bias on whatever method I use determine the missing value.

Sample Data

0.0,0.3,0.2
0.4,Nan,0.2
0.2,0.6,Nan
Nan,0.1,0.5
0.0,0.2,0.0
0.9,Nan,Nan

Would be converted to

0.0,1,0, 0.3,1,0, 0.2,1,0
0.4,1,0, 0.0,0,1, 0.2,1,0
0.2,1,0, 0.6,1,0, 0.0,0,1
0.0,0,1, 0.1,1,0, 0.5,1,0
0.0,1,0, 0.2,1,0, 0.0,1,0
0.9,1,0, 0.0,0,1, 0.0,0,1

Since my dataset is already relatively small I don't particularly want to remove entries. This would reduce my data by roughly a third.

If this is a 'foul ball out of left field' (makes no sense and would cause more errors) how would you deal with a small data set with a decent amount of holes.

EDIT: As per @zachdj suggestion in comments

Would be instead converted to this data set which is much smaller.

0.0,1, 0.3,1, 0.2,1
0.4,1, 0.0,0, 0.2,1
0.2,1, 0.6,1, 0.0,0
0.0,0, 0.1,1, 0.5,1
0.0,1, 0.2,1, 0.0,1
0.9,1, 0.0,0, 0.0,0


texts:
I have met that question online and I wanted to know where sampling can simulate complex processes and why?

Why is sampling useful in machine learning?

Sampling can increase the accuracy of the model
Sampling can simulate complex processes
Sampling is lower cost
Sampling can save lots of time

I can select multiple ones. As far as I know, sampling is lower cost and can save lots of time but, can it simulate complex processes?


texts: How much time do scikit classifiers take to classify?

texts:
I am planning to use scikit linear support vector machine (SVM) classifier for text classification on a corpus consisting of 1 million labeled documents. What I am planning to do is, when a user enters some keyword, the classifier will first classify it in a category, and then a subsequent information retrieval query will happen in within the documents of that category catagory. I have a few questions:

How do I confirm that classification will not take much time? I don't want users to have to spend time waiting for a classification to finish in order to get better results.
Is using Python's scikit library for websites/web applications suitable for this?
Does anyone know how amazon or flipkart perform classification on user queries, or do they use a completely different logic?


texts: What techniques are there to train custom sentence classification models with reasonable memory footprint?

texts:
We are currently working on tasks that involve user-inputted data (e.g., question-answers, short-answer-grading), with a framework that will allow them to be improved through active learning. However, we are running into an issue for determining best-practices for training these custom models.
In most frameworks, the common suggestion is:

Download an enormous pretrained model (BERT, etc.)
Put your data into a task which will retrain the model for a few epochs
Use your new, specialized, and enormous model

However, this quickly breaks down when you have many such models. You can't let every single user have their own copy of BERT.
As an alternative, there are options for training layers on top of the pre-trained model. So the ideal process is instead:

Download your enormous pretrained model.
Train a task which builds a custom layer/layers on top of the enormous pretrained model.
Store that top-layer model for use on that task

Are there any recipes or projects that represent a good approach to do this which rely on active efforts? We have done some basic ad-hoc layer approaches to training classifiers for this purpose (e.g., you can put a Logistic model or another layer of neural network on top of embeddings), but my hope is that there are approaches out there that are more systematic for training NLP classifiers that add lightweight layers to large-corpus pretrained models.


texts:
I'm working on event-forecasting problem for a supermarket, and the data frame I have looks like (sorry for formatting):

    Start date   End date   product   Y   event_params
0    21.07.18    30.07.18      A     100     value_1
1    22.07.18    25.07.18      B     200     value_2
2    13.07.18    20.07.18      C     140     value_3`

Before training the model I calculate Y column using daily information about each product. The issue is that it appears, that on the events with crossing dates I calculate the same amount twice. My final goal is to get a weekly information on Y column, so I my prediction will have a big error, just because there is no clarity in the training data frame.

My idea was to transpose it somehow, so that during the training I have clarity for the events with intersection.

Smth like this:

product  week1    week2   ...?
   A    value_1  value_1   1
   B    value_2  value_2   2
   C    value_3  value_3   3`

The question is what will be the best way to process each event params? I can't add them as columns, since there might be up to 20 intersecting events (so that df will have lot's of NaN values). Is there any other way of solving the problem with preserving original event parameters (no averaging etc.)?

Maybe you can give me any other clue of how to process the data like this.


texts: Improve the loss reduction in a neural network model

texts:
The following code is to train a neural network model of a given dataset (50,000 samples, 64 dim).

from keras import Sequential
from keras.layers import Dense
from keras.optimizers import Adam

X, y = process_dataset()
model = Sequential([
    Dense(16, input_dim=X.shape[1], activation='relu'),
    Dense(16, activation='relu'),
    Dense(1, activation='sigmoid')
])

'''
    Compile the Model
'''
model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.01), metrics=['accuracy'])

'''
    Fit the Model
'''
model.fit(X, y, shuffle=True, epochs=1000, batch_size=200, validation_split=0.2, verbose=2)

In the beginning, you can see below that the val_loss gets reduced from one epoch to another very well.

Epoch 82/1000
 - 0s - loss: 0.2036 - acc: 0.9144 - val_loss: 0.2400 - val_acc: 0.8885
Epoch 83/1000
 - 0s - loss: 0.2036 - acc: 0.9146 - val_loss: 0.2375 - val_acc: 0.8901

When the model takes many epochs, the loss change becomes so small, especially when the number of epochs increases.

Epoch 455/1000
 - 0s - loss: 0.0903 - acc: 0.9630 - val_loss: 0.1317 - val_acc: 0.9417
Epoch 456/1000
 - 0s - loss: 0.0913 - acc: 0.9628 - val_loss: 0.1329 - val_acc: 0.9443

Kindly, I have two questions:

What does this phenomenon mean? i.e., the loss begins to decrease very well at the beginning but not much reduction by the time the training epochs takes a lot of iteration.
What is the possible solution for this?

Thank you,


texts: How can we model the class which maximizes the event probability?

texts:
I’d like to ask about a case when we would like to predict the best class of some input variable, so that the probability of event will be maximized.

For example the advertisement type for a given customer, which will maximize the probability of purchase.
In collected data we have many various ad types, customer descriptors and informations if the purchase was made or not.

One solution which comes to my mind is to treat the ad type as an input variable, train regular probability model, then make predictions with all ad type configuration and then pick one giving the best estimation.

What are the other options?


texts: pandas count values for last 7 days from each date


texts: How do you generalize from stochastic to batch learning using gradient descent?

texts:
I understand pretty OK how to derive the formulas and implement stochastic gradient descent for a deep neural network (even though the total derivative magic for hidden layers is a bit pushing my limits).

I'm struggling to grasp the fundamentals of generalizing the method to batch or mini batch training.

From what I understand, in batch training you feed forward all of your batch's examples and then backpropagate the error.

In stochastic backpropagation most computations involve taking the derivative of the error with respect to neurons' activations. But in the case of batch training there is one activation per training example. Do you average them or something ? Or is the math radically different?

Is there a simple example somewhere that doesn't involve tensors?


texts:
So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data for that particular input in training data set.

 x1          x2           x3          y
 10          15           20          0
 20           7           10          1
 5           10           4           0

So imagine this is an example training set, we send these inputs to activation function, and for the first input it returns correct output (0).

But for second output it again returns 0, so the weights are adjusted until the activation function returns 1.

So now my doubt is, if the new updated weights returns the wrong output for the third input, its weight gets changed again, but will there be a situation where these weights will not satisfy for the previously tested inputs, like for example the first input in this case.

Is there a chance that new weights will return 1 for the first input, which is wrong?


texts: Finding the worst affected industry due to COVID in terms of unemployment

texts:
My goal is to find the worst affected industries from COVID—19 in terms unemployment.
In terms of the data I will use for this task, I have a time series county-wise unemployment rate data of each month and business distribution data.
Business distribution data contains number of establishments in each county by their respective industries. (Manufacturing -121, Accommodation and Food Services -564, Construction-32 etc.)
Unemployment rate data gives monthly unemployment rate in each county.
From this data, what would your recommendation be to find the worst and least affected industry?
Any help is highly appreciated. Thanks in advance.


texts:
I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution.  Is this a feature of the algorithm (e.g., all logit probabilities are log-normal), or might it be a result in my data?

Thanks.


texts: software for workflow integrating network analysis, predictive analytics, and performance metrics

texts:
I am hoping that there is some existing software for what I want to accomplish, as I'm not a big fan of reinventing the wheel.

In general, I would like a software package that can serve as a workflow that integrates network analysis, predictive analytics, and performance evaluation.

More specifically, information about a criminal network along with other relevant data would be used to predict monetary flows across the network.  Law enforcement case data combined with information about network monetary flows would direct resource use in disrupting flows across the network.  These law enforcement performance metrics along with metrics related to criminal networks would be used for strategic decision making.  It would also be ideal to have dashboards showing some of the metrics of interest.

I have identified some potential software candidates: Dataiku DSS, RapidMiner, KNIME, Orange Data Mining, Watson, and Statistica.  I am not too familiar with them and I wonder if something else may immediately come to mind for those more familiar with these applications.

Any direction is greatly appreciated.

-Ted


texts:
Why does the complexity of KNearest Neighbors increase with lower value of k? And when does the plot for k-nearest neighbor have smooth or complex decision boundary? Please explain in detail.

And also , given a data instance to classify, does K-NN compute the probability of each possible class using a statistical model of the input features or just gets the class with the most number of points in favour of it?


texts: When should you balance a time series dataset?

texts: I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?


texts: fastai - using 'untar_data' function in kaggle kernel

texts:
I have recently started with fastai lesson 1 and I am using kaggle to run the course notebooks. While going through the ‘lesson1-pets’ notebook we use untar_data(URLs.PETS) to get the data.

What I want to understand is where does this data get downloaded to?
As I can observe, after running the untar_data(URLs.PETS) function, it says downloading… ,the data gets downloaded, but nothing gets added to the data section of kaggle kernel.

Note: I am able to run the whole notebook in kaggle and just want to understand the working of untar_data function.


texts: How to model a conditional demand forecast model as ex-ante forecast for a moving population?

texts:
Goal: I am trying to forecast demand from a specific population given a specific promotion on a certain period of time frame.
The data is of the format:

date | promotion % | sales in 1000s(y)
01-Jan | 20 | 10
02-Jan | 25 | 12
...
01-Oct | 23 | 130

Now given that demand at any given point of time t depends on time t-1, I tried to model this as time series model ,and to capture the impact of promotion, I would add an exogenous variable w/ promotion.
So, this would typically be modelled as an ex-ante forecast.
The problem starts when population to whom the promotion is offered has to bear a specific criteria(say # of transactions > 2 ). And there are new customers acquired daily, who might or might not bear that specific criteria.
In summary,
if I look back from 1st-OCT , I have 1000 customers with # of transactions > 2.
The demand generated over time from these folks would look like:

date | promotion % |  # of customers causing demand | sales in 1000s(y)
01-Jan | 20 | 100 | 10
02-Jan | 25 | 150 | 12
...
01-Oct | 23 | 1000 | 130

Now the forecast needs to be on these 1000 customers only for different promotion values in coming days (02-Oct, 03-Oct,...,07-Oct) i.e.
Forecast for:

date | promotion % |  # of customers causing demand | sales in 1000s(y)
02-Oct | 20 | 1000 | ?
02-Oct | 30 | 1000 | ?
03- Oct| 20 | 1000 | ?
03- Oct| 30 | 1000 | ?

Problem:
Given, that I am tracking demand generated only due to the folks that I have information so far(1000 customers), in the validation period, the demand curve changes, and does not have the trend and seasonality components as before. This can be seen below:

I went thru https://otexts.com/fpp2/forecasting-regression.html, and the solution that came to mind was to be able to predict the number of customers(prediction of predictors) and to translate this into pure ex-ante forecast problem.
But the forecast of predictors(# of customers) is difficult and I would want to avoid a convoluted design as far as I can.
Another way would be to model this as a regression problem, w/ addition of lag variables as features.
I also researched a couple of research papers but so far there is no obvious way of modelling this, or I have limited terminology with me to be able to find this.
The actual question:
Is there a way to model time series for a growing population causing demand and do a conditional forecast?


texts: Handling categorical features in Factorization Machines algorithm - Feature Hashing vs. One-Hot encoding

texts:
For solving a prediction problem I'm willing to use the Factorization Machines, a model that in addition to learning linear weights on features, learn a vector space for each feature to learn pairing interactions between features in this new space.

I was told that performing the hashing trick to convert categorical features to 1-of-k binary features (using sklearn’s DictVectorizer, which returns sparse matrix) can destroy feature interaction and I should try regular one-hot encoding instead.

Can anyone explain why?


texts: How to adapt the LBFGS algorithm to accept a different data input parameter in Spark?

texts:
I´m trying to use the LBFGS algorithm to minimize the parameters of a model used for Customer Life Time Value analysis (https://github.com/CamDavidsonPilon/lifetimes). The lifetimes lib in python uses a minimize function that has the data arrays as input parameters. The format is something like:

date,id
2014-03-08 00:00:00,0
2014-05-21 00:00:00,1
2014-03-14 00:00:00,2

We need to implement this same function in scala, but the implementation available here (https://github.com/apache/spark/blob/v2.2.0/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala) has a sligthly different format for the data input:

   * @param data - Input data for L-BFGS. RDD of the set of data examples, each of
   *               the form (label, [feature values]).

Question is, how could I change the scala implementation to accept the data as just plain arrays or in the format given by the lifetimes lib in python?


texts: Why do I need to create a dataframe just to add a column?

texts:
So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is there some function I'm missing to add new columns? Or is it that in actuality only one dataframe is created and the new ones just point back to the columns of the old dataframe with the addition of the new column?
I'm using pyspark btw.
Help is appreciated.


texts: How to apply the Fourier or Wavelet transformed data to LSTM model

texts:
I am dealing with a PHM RUL problem, a time series problem of machine signal. I consider to apply Fourier transform or Wavelet Transform to my sensor feature and train the LSTM model. But I have some problems when I using these transformed sensor data.
my data looks like:

sensor 1    sensor 2   sensor 3    RUL
   1           1.5         3        5
   2           2.5         2        4
   3           3.5         1        3
   4           4.5         0        2
  ...          ...        ...      ...

First 3 column is my features(x), the last column is my RUL(y). The data have n rows which are time sequential. Every x mapping to one y.

I have some ideal to employ the transformed sensor data to my training model.

Simply treat the transformed sensor data as a new feature.
Concat the transformed sensor data after original time series data.
Only feed the transformed sensor data to my training model.
Create two LSTM model. The original data and transformed sensor data feed in their own model respectively, then combine the result by something like ensemble.

I can't make sure which one is more reasonable. Actually, all these ways will face a problem when I mapping my transformed sensor data(x) to my original RUL(y). That is, the transformed sensor data often have difference length(rows) from the original data. I can not mapping x to y if their length is different.

Any advise about how to apply the transformed data to LSTM model will be appreciated !


texts: RL: Collecting States (training data) in real-life. Must use fixed timestep?

texts:
I am using a Reinforcement Learning agent to play a 3D game, but have trouble with collecting the "current and next state" pairs.

To decide what action to perform, the network must perform a forward pass.

It performs forward pass in time $t$, but in the meantime the actual environment might have evolved significantly.  In computer games this means something like 10 frames or more have already passed (a varying amount).

The situation is worsened if I run, say 100 games at once on the same computer.

I don't have the ability to stop the game at each frame to do forwardprop. Anyway  it wouldn't be possible were I to train, say a real-life robot to walk.

Question:

Should I stick to a 'fixed timestep' approach, only asking to provide an action every 0.1 seconds? While it computes next action, I could pretend the network keeps outputting the most recent action for all the skipped frames. Good idea?

If that's the only option, then should I avoid at all costs situations where forward prop takes more than the 'fixed timestep'? (more than 0.1 sec in my case) So it's better to choose say, 0.2 seconds just to be safe.

Seems quite unreliable - is there a better way to do it?

Is there a paper that explores the alternatives? (I guess such a paper will be about real-life robot training)


texts: How to group by ID, and output without repeating IDs?

texts:
Input

UserID  Profile
101     apple
102     banana
103     pie
101     banana
102     pine
103     berry

Output

UserID  Profile
101     apple
        banana
102     banana
        pine
103     pie
        berry


texts: GBM: small change in the trainset causes radical change in predictions

texts:
I have build a model using transactions data trying to predict the value of future transactions. The main algorithm is Gradient Boosting Machine. The overall accuracy on the testset is fine and there is no sign of overfitting. However, a small change in the training set creates radical change in the model, and in the predictions. But even when the testset change a little the overall accuracy is stable.

The time period is from 2005 to today and when a single day is added to the dataset predictions change drastically (e.g. +/- 10%). If multiple training are perform on the same training set, the predictions are the same.

I have test Light GBM(2.1.0) and XGBoost(0.60) with Python 3.6 on Windows 10. A seed is set and I train the model on CPUs. I have tried to increase the number of iterations to a high number and adding a specific seed to the bagging parameters.

This blogpost discuss brefly that issu without giving any solutions.


texts: Simple LSTM model quickly learns and overfits

texts:
I am training a multi-class LSTM classifier on approximatively 700k documents of 40 words.
My classes are very umbalanced, some have 2 or 3 samples while the biggest class has 48548 documents.
My data was originally unlabeled so I previously trained another model to cluster them, that process was: Cleaning, etc -> LSTM auto-encoder -> DBSCAN clustering on codes -> Automatic merge using difflib's ratio and a threshold -> Manual checks to filter, remerge, drop noisy cluters, etc -> Automatic duplicate removal inside the classes using FastDamereauLevenshtein (for example biggest class went from 100k -> 48k thanks to this).
Now, my classes are still very clean. Although umbalanced, texts are very similar to each other inside a single class, sometimes it's just a difference of 1 or 2 words. So this should be an easy job for another classifier to learn.
I start by creating my training and validation sets, using a 90%-10% ratio (10% only, because some classes have like 3 documents, so I don't want to drop more documents from them).
I use a tokenizer to build a vocabulary and fit the sequences:
max_words = 20000
max_len = 40

tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(X)
sequences = tokenizer.texts_to_sequences(X)
X = pad_sequences(sequences, maxlen=max_len)

Then, to deal with the problem of umbalanced classes, I found a solution online to smoothen the class weights and assign more weight to the classes that are less populated:
class_weights = {}
k = 0
for k_class in range(n_classes):
    N = np.count_nonzero(Y_encoded==k_class)
    class_weights[k] = N
    k+=1

# Smoothen it
import math
total = df_train.shape[0]
keys = class_weights.keys()
class_weights_smooth = dict()
mu = 0.5

for key in keys:
    score = math.log(mu*total/float(class_weights[key]))
    class_weights_smooth[key] = score if score > 1.0 else 1.0

Thanks to this, the weight of my biggest class (48k documents) is 1, and the weight of the lowest class is ~11. 11 is still very small compared to 48k but at least better.
Now here's the architecture of my classifier:
from tensorflow.keras.layers import Dropout, Embedding, LSTM, Dense
from tensorflow.keras.models import Sequential

model = Sequential()
model.add(Embedding(max_words, 50, mask_zero=True, input_length=40))
model.add(LSTM(40, return_sequences=True))
model.add(Dropout(0.5))
model.add(LSTM(40))
model.add(Dropout(0.5))
model.add(Dense(n_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

I don't think this model is very complex. It has only 30k parameters if you exclude the 1,000,000 parameters of the embedding layer.
Training is pretty slow, however it does seem to quickly overfit to the training data:
Epoch 1/20
4999/4999 [==============================] - 943s 189ms/step - loss: 1.0499 - accuracy: 0.9229 - val_loss: 1.8765 - val_accuracy: 0.5728
Epoch 2/20
4999/4999 [==============================] - 941s 593ms/step - loss: 0.3578 - accuracy: 0.9748 - val_loss: 1.4938 - val_accuracy: 0.6635

It goes on, and the training accuracy eventually reaches >99% while the validation accuracy still lacks behind. I have to train it for almost 20 epochs so it can reach 95% on validation accuracy but the model will be ultra-overfitted.
I'm quite lost and don't know what to really do now. How can I improve this?

Should I make the model simpler or more complex?
Is the embedding OK in size and vocabulary size?
???


texts: The connection between optimization and generalization

texts:
Optimization algorithms such as gradient descent or particle swarm can find a minima in a function.

On the other hand, learning methods such as back-prop define learning as an optimization problem and are used to learn weights in deep neural networks and etc.

We know that DL models can learn/memorize basically anything in the training data [2], even random noise, which brings us to the question of then how we can achieve generalization (as defined in [5]) *.

In [7, 1, 3] authors tried to show the connection between generalization and the loss landscape (sharp minima/flat minima) plus its effect from the batchsize.

The effect of training data on generalization studied in [4] shows that models that have high influence from more data points generalize better than models that rely/influenced on less data points.

On the other hand [6] sheds some light on the role of the model's parameters from a neuroscience point of view in terms of effect of neurons in generalization (Networks that generalize better are harder to break with neuron deletion.).

Although all of these papers investigated generalization up to a certain point, the effect of optimization used in the learning algorithm on the generalization is not yet clear.

Is there any study done in this direction? What are the possible related work on this? Or do you have an answer to this question?

*:
Let me point out that although many believe explicit regularization is crucial for generalization, [2] already explains explicit regularization (l1/l2/dropout) does not play a big role in generalization. Many tricks that were known as a generalizer, are shown to be a myth. They also show that interestingly, SGD works as an implicit regularizer which might be a connection to the effect of optimization alg. to generalization.

[1]: Li, Hao, et al. "Visualizing the Loss Landscape of Neural Nets." arXiv preprint arXiv:1712.09913 (2017).

[2]: Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).

[3]: Keskar, Nitish Shirish, et al. "On large-batch training for deep learning: Generalization gap and sharp minima." arXiv preprint arXiv:1609.04836 (2016).

[4]: Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." arXiv preprint arXiv:1703.04730 (2017).

[5]: Kawaguchi, Kenji, Leslie Pack Kaelbling, and Yoshua Bengio. "Generalization in deep learning." arXiv preprint arXiv:1710.05468 (2017).

[6]: Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, Matthew Botvinick, "On the importance of single directions for generalization." arXiv preprint arXiv:1803.06959 (2018). https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/

[7]: Dinh, Laurent, et al. "Sharp minima can generalize for deep nets." arXiv preprint arXiv:1703.04933 (2017).


texts:
I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example:

Dataset A: Company name: Facebook
Dataset B: Company name: Facebook, Inc
Dataset C: Company name: facebook

Some other signals like company url exists, but in terms of name matching wondering if text similarity is a good approach for this grouping problem.


texts: Deep learning and label noise. Best practices for the real world

texts:
Unlike MNIST or other benchmark datasets collected data often come with subpar, inaccurate labels.
What are the best practices to help the neural networks to don't overfit the noise? Things that comes to my mind:

gradient clipping

regularization

adjustment to the loss function (e.g. Patrini Making Deep Neural CVPR 2017 paper), but they typically assume the noise to be independent from the features. Have you used them successfully?

dataset distillation. How did you do that? Do you have references you can share?

I found few papers on the topic, but I found them quite academic (e.g. they simulate the noise by randomly flipping the labels, they assume very large datasets and computational resources). Has anyone applied them successfully?


texts: Why there is Different accuracy for two same trained model?

texts:
Trained the same model twice with the same dataset, the same parameters (Epochs, Batch Size, Learning rate, etc..). But both trained model shows different train as well as test accuracy on the same dataset.
(code is same for two models)
Why both models show different accuracy?

Train the model
Test accuracy = 87.98%

Again Train the model
Test accuracy = 67.18%


texts:
Can machine learning algorithms predict random number generators.
key
A= 1 2 3
B= 4 5 6
C= 7 8 9
Example to catch a number sequence
4 8 8
I would select B C C
That would give me 27 number combinations
4 7 7
4 8 7
4 9 7
5 7 7
5 8 7
5 9 7
6 7 7
6 8 7
6 9 7
4 7 8
4 8 8 match
4 9 8
5 7 8
5 8 8
5 9 8
6 7 8
6 8 8
6 9 8
4 7 9
4 8 9
4 9 9
5 7 9
5 8 9
5 9 9
6 7 9
6 8 9
6 9 9
Using the example above can a sequence prediction algorithm predict the next sequence
Data
B C C
C C A
A C A
C A A
C B B
C C A
B C C
A C B
C B A
B B B
A A C
C A C
B A C
A C A
B C C
B A A
C B C
A C C
A A C
C C B
B B C
B C C
A A B
B A A
A B B
A B A
C C C
C C A


texts: size of deep learning object detection pre-train models

texts:
I am benchmarking five different models,

YOLO v2
YOLO v3
Cascade R-CNN
Faster R-CNN
Retina Net

In my comparison, I would like to know what the sizes of each model, to discuss the potential overfitting (if some are larger than others). It can be either the number of layers, the number of parameters. For Yolo models these are stated in the paper, e.g  YOLOv2 has 106 layers with 51,000,657 parameters, whereas YOLOv3 has 349 layers and 65,252,682 parameters.

I looked into the other models, but couldn't find similar numbers in their own papers. Therefore, I appreciate if you help me to figure out whether those three are simpler than YOLOs or where they stand in terms of complexity.


texts: How to assess nested cross validation results in comparison to non-nested results?

texts:
I have a nonlinear regression model scoring genes from scores between 0 to 1 as to whether they are likely to cause disease. Training data is ~700 gene samples by 53 features.
Currently I get results with xgboost which look like:
r2 Nested CV Average: 0.807
MSE Nested CV Average: -0.016

Non-nested Results:
XGBR Train r2: 0.949 Test r2: 0.805
XGBR Train MSE: 0.002 Test MSE: 0.018
r2: 0.895
Predicted r2: 0.871

Should I be concerned that the non-nested training result is overfitting in comparison to the test result, or should I only rely on my 5-fold nested cross-validation to determine overfitting is minimised?
For reference, the XGBoost model I use is tuned like this:
xgbr = xgboost.XGBRegressor(random_state=seed, objective='reg:squarederror')
xgbr_params = {
    'max_depth': (1, 10),
    'learning_rate': (0.01, 0.5),
    'n_estimators': (20, 50),
    'reg_alpha': (1, 10),
    'reg_lambda': (1, 10),
    'gamma': (0, 0.5),
    'min_child_weight': (1, 5),
    'subsample': (0.1, 1),
    'colsample_bytree': (0.1, 1)}

#Best parameter output:
xgbr = xgboost.XGBRegressor(random_state=seed, subsample=0.8258568992489053, min_child_weight=1,
n_estimators=50, gamma=0.0, objective='reg:squarederror', colsample_bytree=1.0,
learning_rate= 0.3987519903467713, max_depth=4, reg_alpha=1, reg_lambda=10)


texts: What's wrong with my backpropagation through time (BTT) calculation or how to multiple a scaled vector and a matrix without matching dimensions?

texts:
I am trying to make a pretty simple RNN from scracth, using only Numpy library of Python.
At this moment I am having troubles with BTT as I do not know how to proceed with situation when a multiplication of a scaled vector and a matrix, without matching dimensions, should be made.
Probably, I made a mistake somewhere and this situation is just a consequce of it. However, I was not able to identify any.

For the presentation purposes I have written my calculation on a piece of paper. Here's the screenshot of it->>>Calculation

So, could anyone please point to the mistake I made. Or if there is not any error, then how should I proceed? How to muplitply vectors/matrices when they do not have matching dimensions?
I know there is a technique called padding, when '1's are added to match dimensions but I am not sure it is relevant here.

Anyway, thanks for the reading and your answers!

UPDATE: I figured out my mistake. I need to multiply 1 by 4 vector with 1 by 5 vector, not 3 by 5 matrix as it is matrix with all inputs of this particulat training example. Here I need only X3 to calculate how change in Wx affect H3. Outer product will produce a matrix with exact dimensions required - 5 by 4 matrix


texts: Does it makes sense to apply feature scaling on timestamp

texts:
I was wondering if it makes sense to apply normal standardization on a feature like timestamp ?
The data that I process are network packets.

Thank you


texts: How do I read accuracy and loss curves for a CNN?

texts:
I am using transfer learning and after evaluating the model I have the following plots which compare test and train accuracy and test and train loss:

what I see is that between test and train accuracy there is a large gap, but how do I interpret this result?

I don't think I am doing bad since I have a test accuracy of 0.880000.

I am trying to do image classification and in my training set I have 3200 images, while in the test set I have 800 images. As pretrained network I am using VGG16.

Can somebody please help me? Thanks in advance.

[EDIT]Also the other metrics such as Recall,F1 score and Precision are good:


texts: How to deal with overestimation of small values and underestimation of high values in XGBoost?

texts:
I'm running XGBoost to predict prices on a cars dataset, I was wondering what alternatives are there for this kind of problem where smaller values are overestimated and higher prices underestimated.

I tried applying log to prices since it has a skewed to the right distribution, but still having this undesirable effect.

Also, as a bonus question, log(price) improved the prediction score, the mean relative error or MRE calculated as mean(ABS(RD)) by 2 percent, if anyone has the intuition onto why this could have happened that would be great.

In the image below RD is the relative difference between predictions and the actual values, and the price bucket is a bucketized variable where the number indicates the price low interval bound over 1000.


texts: Is it possible to automate generating reproducibility documentation?

texts:
First, think it's worth me stating what I mean by replication & reproducibility:

Replication of analysis A results in an exact copy of all inputs and processes that are supply and result in incidental outputs in analysis B.
Reproducibility of analysis A results in inputs, processes, and outputs that are semantically incidental to analysis A, without access to the exact inputs and processes.

Putting aside how easy it might be to replicate a given build, especially an ad-hoc one, to me replication always possible if it's planned for and worth doing. That said, it is unclear to me is how to execute a data science workflow that allows for reproducibility.

The closet comparison I'm able to think of is documentation generators that generates software documentation intended for programmers - though the main difference I see is that in theory, if two sets of analysis ran the "reproducibility documentation generators" the documentation should match.

Another issue, is that while I get the concept of reproducibility documentation, I am having a hard time imagining what it would look like in usable form without just being a guide to replicating the analysis.

Lastly, whole intent of this is to understand if it's possible to "bake-in" reproducibility documentation as you build out a stack, not after the stack is built.

So, Is it possible to automate generating reproducibility documentation, and if so how, and what would it look like?

UPDATE: Please note that this is the second draft of this question and that Christopher Louden was kind enough to let me edit the question after I realized it was likely the first draft was unclear. Thanks!


texts: Can ridge regression be used for feature selection?

texts:
I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to zero, only really close to it.

So can ridge regression cause any coefficients to become zero? Can the number of zeros in a weights vector change from zero to something else?
Or maybe to put it simply, are the number of zero coefficients using ridge regression monotonically increasing or possibly decreasing?

Thanks.


texts:
The loss used in REINFORCE algorithm is confusing me.

From Pytorch documentation :

loss = -m.log_prob(action) * reward

We want to minimize this loss.

If a take the following example :

Action #1 give a low reward (-1 for the example)
Action #2 give a high reward (+1 for the example)

Let's compare the loss of each action considering both have same probability for simplicity :

p(a1) = p(a2)
=> m.log_prob(a1) = m.log_prob(a2)
And loss(a1) = -m.log_prob(a1) * reward(a1) = m.log_prob(a1)
And loss(a2) = -m.log_prob(a2) * reward(a2) = -m.log_prob(a2) = -m.log_prob(a1)
Then loss(a1) < loss(a2) since m.log_prob(X) < 0

Here I don't understand this conclusion : the loss being minimized, it means a small loss is good compared to a high loss.

So here it would mean action #1 is good compared to action #2 ? But the reward says the opposite !


texts: How should I sample from a mixture distribution?

texts:
Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf for the distribution $D_i$ corresponding to density $p_i$. I would like to sample from such a distribution.
The method I currently employ is an implementation of rejection sampling. I construct a proposal function $M*u(x)$, where $u \sim \text{Unif}(lb,ub)$ ($lb,ub$ are constructed such that at least 99% of the cdf of each $D_i$ is contained within, using icdf) and $M$ is constructed by finding the $\max_{x \in [lb,ub],i} p_i(x)$. Because such proposal function envelopes $f$, I am able to sample $f$ by choosing $x \in X \sim \text{Unif}(lb,ub) \times \text{Unif}(0,M)$ and rejecting if $\pi_2 x > f(x)$ [$\pi_2 x$ being the second coordinate of $x$].
However, doing this is quite slow. Not only is the proposal function inopportune (it is scaled uniform, which likely will lead to many rejections), but the construction of $M$ is very slow, as maximizing a function is a non-trivial task. Is there a more efficient way to sample such a distribution? I had considered icdf sampling, but constructing the icdf for $f$ seems non-trivially difficult. Is this impression incorrect? Or perhaps is there some other effective method? If it is helpful, I am implementing this in python and am currently using the scipy and pytorch libraries.


texts: How are weights organized in Conv2D layers?


texts:
In which cases is it better to use a Decision tree and other cases a KNN?

Why use one of them in certain cases? And the other in different cases? (By looking at its functionality, not at the algorithm)

Anyone have some explanations or references about this?


texts: XGBoost increase the error when changing evaluation function

texts:
I have changed the eval function of XGBoost to rmsle and the optimisation increase the error after the iteration [2] instead of decreasing it. If I change to the default eval function, RMSE, this does not happen.

This is the code of RMSLE used:

def evalerror(preds, dtrain): # this is compatible with DMatrix
    labels = dtrain.get_label()
    assert len(preds) == len(labels)
    labels = labels.tolist()
    preds = preds.tolist()
    terms_to_sum = [(math.log(labels[i] + 1) - math.log(max(0,preds[i]) + 1)) ** 2.0 for i,pred in enumerate(labels)]
    return 'error', (sum(terms_to_sum) * (1.0/len(preds))) ** 0.5

This is the parameters of XGBoost used:

param = {'bst:max_depth':1, 'bst:eta':0.025, 'silent':False, 'objective':'reg:linear','eval_metric':'rmse' }
bst = xgb.train( param, d_train, num_rounds,early_stopping_rounds=20, evals=eval_list, verbose_eval=True, feval=evalerror)

and this is the evaluation:

[0] eval-error:0.836219 train-error:0.835095
Multiple eval metrics have been passed: 'train-error' will be used for early stopping.

Will train until train-error hasn't improved in 20 rounds.
[1] eval-error:0.809301 train-error:0.806747
[2] eval-error:0.792647 train-error:0.78908
[3] eval-error:0.803355 train-error:0.798805
[4] eval-error:0.803261 train-error:0.79835
[5] eval-error:0.809352 train-error:0.804283
[6] eval-error:0.810453 train-error:0.805126
[7] eval-error:0.811059 train-error:0.805646
[8] eval-error:0.815261 train-error:0.809722
[9] eval-error:0.820237 train-error:0.814521
[10]    eval-error:0.823378 train-error:0.817408
[11]    eval-error:0.824981 train-error:0.81868
[12]    eval-error:0.826607 train-error:0.820176
[13]    eval-error:0.827813 train-error:0.821358
[14]    eval-error:0.827625 train-error:0.821007
[15]    eval-error:0.823347 train-error:0.816547
[16]    eval-error:0.824362 train-error:0.81752
[17]    eval-error:0.82529  train-error:0.818321
[18]    eval-error:0.824621 train-error:0.817463
[19]    eval-error:0.824103 train-error:0.816766
[20]    eval-error:0.814759 train-error:0.807234
[21]    eval-error:0.807961 train-error:0.800186
[22]    eval-error:0.808398 train-error:0.800246
Stopping. Best iteration:
[2] eval-error:0.792647 train-error:0.78908

It may be the case that I need to adjust my objective function to this evaluation metric?


texts: Keras Warning (Single Feature Regression)

texts:
# Regression Example With Boston Dataset: Standardized
import numpy as np
from pandas import read_csv
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# load dataset
dataframe = read_csv("housing.csv", delim_whitespace=True, header=None)
dataset = dataframe.values
#print(dataset.shape)
# split into input (X) and output (Y) variables
X = np.array([1,3,4,5,6,7,8,9,11,12,13,14,15,16,17,18,30,100])
Y = np.array([5369,1850,1396,1111,935,885,845,714,658,658,659,659,661,656,657,662,663,668])
X=X.reshape(-1,1)
# define base model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(13, input_dim=1, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model
# evaluate model with standardized dataset
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=1, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Standardized: %.2f (%.2f) MSE" % (results.mean(), results.std()))

I am trying to do a simple regression using Keras. But for some reason, it does not seem to work (Many warnings and extremely high RMSE) probably because I have low number of samples. Can I make any changes to this code such that the # of samples does not create an issue.


texts: I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?


texts: How does shuffling make data identically distributed?

texts:
It makes sense that gradient descent algorithms (like stochastic GD, mini batch GD) work better when we shuffle data. (It makes instances in a dataset independent) (for example if dataset sorted by labels, in each mini batch we have data that related to a specific class so weights prone to a class(we have weights and biases that minimizes loss function in respect to one of labels, ... that's make GD have more volatilities).
but how shuffling make instance in dataset identically distributed? and how it helps GD?


texts: Why adding combinations of features would increase performance of linear SVM?

texts:
I have a dataset of ~5000 elements represented by vectors composed by ~30 binary values (0 or 1)
on which I am performing binary classification with SVM with linear kernel (I use the Scikit learn lib).

For curiosity, I tried to add an extra feature that consists in a AND between two others (remember that all my features are boolean). The result was that the performance of the SVM improved. I was surprised by this improvement because the AND operation is equivalent to a multiplication, therefore I would expect that my SVM, as every linear classifier, was somehow naturally already taking into account mutiplications between features.

What is wrong with my theoretic understanding of SVM ?


texts: How is the standard deviation of VAE's obtained?

texts:
I am trying to build a Variational Autoencoder. I was looking at various codes online and found most of them in some way or another copy Francois Chollet (Google researchers) code.

Now my main question with this code is this part:

As you can clearly see the $log(\sigma)$ is the output from a Dense layer. Where did this assumption come from (the output is $log(\sigma)$ and not $\sigma$? How is it possible that we generate a random normal distribution with standard deviation $\sigma$ like this? Is it due to the way the computer generates Normal Distributions?


texts: Cross Correlation Between Input-Output Sine Waves

texts:
I am writing an algorithm to estimate the frequency transfer function of the system. For this, I want to use the Cross-correlation Between Input-Output Sine Waves method. There are a few things I don't understand about method:
I - What does capital N mean?
II - Is and Ic scalar or vector(depend on N)?
These are my thoughts :
I - N represents the number of outputs I collected for a specific w value with a sampling periode.
II - Scalar.
Could you help?


texts: Derivative of activation function calculating the delta of the bias returns always 0

texts: Right now I'm constructing a neural network using a graph (as an assignment). And I bumped up to the problem where I needed to calculate the delta of bias for backpropagation. The thing is, if bias is supposed to always be an activated value of one, then using the derivative of the sigmoid function with one would always return 0 (as is sigmoid(x)*(1-sigmoid(x), which would then be 1*(1-1), therefore 0). But having always 0 in delta of bias makes it so that it never updates. What am I missing?


texts: How multi layer LSTM are interconnected?

texts:
I am trying to understand the layers in LSTM for my own implementation using Python. I started with Keras to getting familiarized with the layer flow.

I have tried the below code in Keras and I have the observations as follows

# LSTM MODEL
step_size = 3
model = Sequential()
model.add(LSTM(32, input_shape=(2, step_size), return_sequences = True))
model.add(LSTM(18))
model.add(Dense(1))
model.add(Activation('linear'))

And I got the below summary details for this implementation.

I tried to understand the internal structure of these layers and how these layer numbers ( 32, 18,1) mapped with matrix. I got the below matrix size from the weight matrix.

I can not see any relation between the output size and the weight matrix :(

Matrix Order

Layer 1

(3, 128)

(32, 128)

(128,)

Layer 2

(32, 72)

(18, 72)

(72,)

Layer 3

(18, 1)

(1,)

Now I am confused to find out the implementation logic inside the layer. Because I got three weight matrix in each LSTM layer and I can not connect the first LSTM(32) to the next LSTM(18) layer.

It will be thankful if any live implementation or flow diagram exists for this implementation. Thanks.


texts:
My model (Logistic Regression) has AUC score of 0.8

Am I right in stating that the probability of the model ranking a random positive sample higher than a random negative sample is 0.8?
Also, how does this interpretation change if I pick more than 2 random samples (let's say 5)? Can I say that the probability of the model to predict highest probability for the best sample (positive) out of 5 is 0.8? Or will this become 0.8*0.8*0.8*0.8 = 0.4096?


texts:
So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course

from sklearn.preprocessing import OneHotEncoder
onehotencoder=OneHotEncoder(categorical_features=[10])
Y= onehotencoder.fit_transform(X).toarray()

However i get the following error

ValueError: could not convert string to float:

A bit of information Y is converted to an object from a df using

Y=df.iloc[:,:].values

I want to oneHotencode the 10th column of y which contains string values. The type of Y in variable explorer is object and if execute

type(Y)

i get numpy.ndarray

I'm new to Pandas and sklearn and would really appreciate any help.


texts:
I am new to machine learning and cannot figure a way to solve this problem:

I have X which is always one row/record while the y can be a row or more.

Here is a simple sample from my data-set, 'i mean that i have more columns in both x and y that the presented here'

x = [[1,3,'John'],[3,5,'Sandra'],[5,2,'Leila']]
y =[

[['D',3,,'K'],['JK',8,'GH']],

[['FG',6,'DF'],['WE',4,'EF'],['CV',5,'EE']],

[['TG',9,'CB']],

]

y contains both numerical and categorical values, all the rows in y need to predicted.

I am using python to do Multi-output regression


texts: What are useful evaluation metrics used in machine learning

texts:
I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321".
 All this based on CNN.

I want to evaluate my model. I used Fscore (recall and precision). Can you advice me more metrics?


texts:
I am currently reading slides about the $k$-means algorithm. In the analysis, the professor writes

  Minimize Schwarz Criterion: $W(C) + \lambda m k \log R$

$W(C)$ is Within-class scatter. I guess $\lambda$ is a weighting factor which has to be chosen by the developer and $k$ is the number of clusters. But what is $m$ and what is $R$?


texts: What could be the reason of having a lower RMSE than MAE?

texts: I used some machine learning algorithms in my dataset and I found that my RMSE goes low than the MAE. What are the most common reasons for that type of typical scenario. Since from my understanding the RMSE is normally higher than the MAE. But if I am wrong is it actually possible to have a lower RMSE and higher MAE? (for example: RMSE: 26, and MAE : 36)


texts: Kendall rank correlation coefficient's p-value

texts:
I'm trying to compute a p-value for a two tailed test following Wikipedia formula which indicates that:

one computes Z, and finds the cumulative probability for a standard
normal distribution at -|Z|. For a 2-tailed test, multiply that number
by two to obtain the p-value

I'm using this Rust's library which computes Tau value and then you can get the significance from this source code.
The problem is that this calculator (with default values) gives a 2-sided p-value = 0.0389842391014099. Which is far from the p-value I'm getting. The steps I'm following are these:

Compute Tau
Compute the statistical significance: Z with significance = kendall::significance(tau, x.len())
Gets the CDF from Gaussian Distribution with sigma = 1 using this GSL library's function: cdf = gaussian_P(-significance.abs(), 1.0)
Multiply that value by 2

I'm getting a very different value: 0.011946505026920469. I don't understand what I'm missing. Perhaps it's a misunderstanding of Gaussian distribution and it's sigma param. Any kind of help would be really appreciated


texts: How can I combine two single-column datasets into a single Pandas data frame?

texts:
I'd like to import the Rotten Tomatoes Movie Review dataset into a single data frame. How can I combine two datasets that are 1-column strings into a text, label shape?

Here's where I'm at so far (you can duplicate in Google Colab) :

import os
import pandas as pd

# Reset
!rm -rf "rt-polarity.csv"

def fetch_rt_polarity_data():
# Fetch Data
if not os.path.isfile("rt-polaritydata.tar.gz"):
    !wget -q http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz
    !tar -xzf rt-polaritydata.tar.gz
    !mv rt-polaritydata/rt-polarity.pos rt-polarity.pos
    !mv rt-polaritydata/rt-polarity.neg rt-polarity.neg
    !rm -rf rt-*

# Format Data
df_pos = pd.read_csv("rt-polarity.pos", encoding='latin-1', sep="\t", lineterminator="\n")
df_pos = df_pos.reset_index(drop=True)
df_pos.columns = ['text']
df_pos['label'] = 1

df_neg = pd.read_csv("rt-polarity.neg", encoding='latin-1', sep="\t", lineterminator="\n")
df_neg = df_neg.reset_index(drop=True)
df_neg.columns = ['text']
df_neg['label'] = 0

# Combine
df = pd.concat([df_pos, df_neg], ignore_index=True)
df.head(10)

df.to_csv("rt-polarity.csv")

df = pd.read_csv("rt-polarity.csv", encoding='latin-1', sep="\t", lineterminator="\n")
return df

df = fetch_rt_polarity_data();
df.head(5)


texts: How to create dictionaries out of pandas dataframes?

texts:
I import three csv-files with characteristics of countries (rows) and years (columns):

country_data_m = 'country_data_m.csv'
m_year = pd.read_csv(country_data_m, nrows=161, index_col=0, header=0, sep=';', na_values=[""])

country_data_e = 'country_data_e.csv'
e_year = pd.read_csv(country_data_e, nrows=161, index_col=0, header=0, sep=';', na_values=[""])

country_data_i = 'country_data_i.csv'
i_year = pd.read_csv(country_data_i, nrows=161, index_col=0, header=0, sep=';', na_values=[""])

The datasets look like this:

                                       1995          1996          1997  \
Afghanistan                              NaN           NaN           NaN
Albania                                  NaN           NaN           NaN
Angola                          5.538749e+09  7.526447e+09  7.648377e+09
Antigua and Barbuda             5.772807e+08  6.337306e+08  6.806171e+08

                                  1995    1996    1997    1998    1999  \
Afghanistan                        NaN     NaN     NaN     NaN     NaN
Albania                            NaN     NaN     NaN     NaN     NaN
Angola                          0.8565  0.8369  0.8173  0.7976  0.7777
Antigua and Barbuda             0.6957  0.6352  0.6513  0.6401  0.6171

                                  1995    1996    1997    1998    1999  \
Afghanistan                        NaN     NaN     NaN     NaN     NaN
Albania                            NaN     NaN     NaN     NaN     NaN
Angola                          0.0612  0.0626  0.0641  0.0655  0.0670
Antigua and Barbuda             0.1852  0.2264  0.2147  0.2147  0.2030

For each country, I need a dictionary, where the key is the year, and where the values are the variables from the three different datasets. So far, I tried this code:

afghanistan = {m_year.loc["Afghanistan", (year)],e_year.loc["Afghanistan", (year)], i_year.loc["Afghanistan", (year)]): year for year in range(1995, 2017)}

albania = {m_year.loc["Albania", (year)],e_year.loc["Albania", (year)], i_year.loc["Albania", (year)]): year for year in range(1995, 2017)}
...
zimbabwe = {m_year.loc["Zimbabwe", (year)],e_year.loc["Zimbabwe", (year)], i_year.loc["Zimbabwe", (year)]): year for year in range(1995, 2017)}

However, the code cannot find the year in the dataframes and gives me the following error:

TypeError: cannot do label indexing on <class 'pandas.core.indexes.base.Index'> with these indexers [1995] of <class 'int'>

Any help would be very much appreciated!


texts: What is the difference between "Python interactive" and "Python 3" kernels

texts:
When creating a new notebook using the web interface, Jupyter gives me two options: 'Python Interactive' and 'Python 3' (see screenshot).

However, I've been unable to find any indication on what the differences between these two kernels (which I assume they represent) are. Both are python 3.6.7 on my system. The search term jupyter combined with "python interactive" or "python 3", even with quotes, yields only generic descriptions of Jupyter, which is not really surprising. Even the Jupyter kernels list doesn't really help.

So, what is the difference?


texts: Deep Learning Model apply for prediction real values

texts:
I have a dataset following structure:

Dataset
    Trainig_Set
          Folder_ID_1
          Folder_ID_2
          Folder_ID_3
              .
              .
              .
          Foder_ID_125

Each folder has no equal images, i.e, Folder_ID_1 = 600, Folder_ID_1 = 750, ... , Folder_ID_45 = 450.

I has label value following structure:

{"Folder_ID_1": 12.1, "Folder_ID_2": 9.1, "Folder_ID_2": 10.5, ... , "Folder_ID_125": 8.5}

Question:
(1) How can apply Deep learning Model for prediction such like real value?


texts:
so that $H_0$ is equidistant from $H_1$ and $H_2$.
However, here the variable $\delta$ is not necessary. So we can set $\delta=1$ to simplify the problem.
$$w\cdot x+b=1 $$
and
$$w\cdot x+b=−1$$

Why is this assumption is taken? If it is taken, we can get the distance between two planes as $2$ directly because both are parallel and differ by $2$. how it is $\frac{2}{\left\|w\right\|}$ instead.
got equations from this https://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/


texts: What are interleaved layers of convolutions?

texts:
In the FaceNet paper, they describe the Zeiler&Fergus model like this:

  [...] the
  Zeiler&Fergus model which consists of multiple interleaved
  layers of convolutions, [...]

What do they mean by interleaved? How does that compare to the inception layers?

Especially, as the Zeiler&Fergus paper states

  We use standard fully supervised convnet models
  throughout the paper, as defined by (LeCun et al.,
  1989) and (Krizhevsky et al., 2012).

  [...]

  The top few layers of the network are conventional
  fully-connected networks and the final layer is
  a softmax classifier.


texts:
I can't find any complexity information about HDBSCAN by google or wiki.
And how about compare to OPTICS?


texts: Integer encoding and weighing when one feature consists of more names

texts: Hello I am trying to make a content based movie recommendation system and one feature is genre of the movie. I will give an integer number to each genre randomly. However, some movies are of more than 1 genre. I will use tf-idf for weigthing these features. However, I am very confused that what to do when a movie is horror and action and comedy movie at the same time. Should I multiply or add these weighted features, I have no idea. Also I do not know why we use weighting in the first place at all. Can you help me get through this? By the way, let me say that each movie will be regarded as a document and then tf-idf calculation is done on them.


texts: Trouble understanding the partial differentiation used in reinforcement learning

texts:
I am studying deterministic actor-critic algorithms in reinforcement learning.

I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in state $s$ and outputs a deterministic action $a$ based on the distribution policy $u$.

The state and action are fed into the critic. The critic sees how good it is to take a particular action from a given state using the action-value function $Q(s,a,w)$.

The critic is then updated via temporal difference (TD) learning and the actor updated in the direction of the critic

Thus it can be seen that the actor's goal is to try and maximise the state action value function $Q(s,a,w)$ by picking the best actions in the given state.

I am having trouble understanding the mathematics behind the updating of the actor.

The below equation gives how the actor is updated.

\begin{equation}
\frac{\partial l}{\partial u} = \frac{\partial Q(s, a, w)}{\partial a} \frac{\partial a}{\partial u}
\end{equation}

What I understand is that we are taking the partial derivative of $l$ with respect to $u$, and we are backpropogating the critic gradient to the actor.

It seems that $l$ is a differentiable function of the variable $a$, but I am confused when it comes to describing what is happening in the equation above as it seems to consist of two functions multiplied together.

Can someone kindly explain what is really happening in the mathematics above?


texts: How to model a Bimodal distribution of target variable

texts:
I want to regress on this target, have tried multiple transformations to bring it to normal but its not helping, read some stuffs online but none of the suggestions have worked till now.

I am attaching the residual histogram as well, somehow the residuals are normally distributed.

Thanks in Advance.


texts:
I created my custom environment in gym, which is a maze. I use a DQN model with BoltzmannQPolicy.
It trains fine with the following variables:

position of the agent
distance from the endpoint
position of the endpoint
which directions can it move to

So I don't give it an image or anything. If I train and test it in the same environment (the same maze, without changing the position of walls) it can solve it easily. But if I introduce it to a completely different environment (maze) without the training then it doesn't know what to do. I don't know if the problem is with my code, or DQN is just for solving the same environment.
Which algorithm should I use instead?


texts: Tool/dataset for matching first names and nicknames

texts: I'm trying to identify the same individuals in a large dataset where sometimes the individuals may be listed by their full first name (e.g., "Michael Douglas") and sometimes by its nickname (e.g., "Mike Douglas"). Does anyone know of a dataset that has (at least) English first names and their corresponding common nicknames? Or a tool that will do such matching for you? I searched for a while and couldn't find anything like this.


texts: Cluster elements that appear in the same lists

texts:
Suppose I have a multitude of sets with (unordered) combinations of elements and I want to determine which elements tend to appear together.

For example

Given the following sets:

{a,e,g}
{a,e,h}
{a,e,i}
{b,f,j}
{b,f,k}
{b,f,l}
{d,c,m}
{d,c,n}
{d,c,o}

Element-pairs where both elements tend to occur in the same sets will have lower distance:

# Low-distance pairs:
{a,e}, {b,f}, {d,c}

# Medium-distance pairs
{a,g}, {b,j}, {d,m}, ...

# High-distance pairs:
{g,h}, {j,n}, {b,f}, ...

Currently

I'm implementing DBSCAN with a custom distance metric. I use the following distance metric between two elements:

d(a,b) = 1 - numsets(a, b) / (numsets(a,!b) + numsets(b,!a))

Where d(a,b) denote the distance between elements a and b. While numsets denotes how many sets fulfill some conditions:

numsets(a, b) - the number of sets that contain a and b
numsets(a,!b) - the number of sets that contain a but not b

This solution should achieve the goal however it's not a pretty solution and I couldn't find this problem on SE. In terms of solving the problem, is there a more sensible distance metric? In terms of implementation, is there a nicer way to do this?


texts: Calculating similarity where order matters

texts:
How can I calculate a similarity (coefficient) where the order of the items matters and something like the Jaccard index would not be useful.

Specifically, I'm interested in comparing ingredients. Take a simplified apple pie ingredient list, for example:

Apples
Enriched Unbleached Flour
Palm Oil
High Fructose Corn Syrup
Salt
Eggs
Spices

And then compare that to:

The exact same ingredients, but in reverse order. Where spices is the top ingredient, and apples are the least used ingredient
The list, but with, palm oil replaced with vegetable oil and corn syrup replace with sugar

Even if you're naive about the fact that the replacements made in #2 are similar, the set of #2 is much more similar to the original than the set of #1. Is there an algorithm which would express that?


texts: How to improve results from ML model? (spam classification)

texts:
I am trying to build a model that predicts if an email is spam/not-spam. After building a logistic regression model, I have got the following results:
          precision    recall  f1-score   support

         0.0       0.92      0.99      0.95       585
         1.0       0.76      0.35      0.48        74

    accuracy                           0.92       659
   macro avg       0.84      0.67      0.72       659
weighted avg       0.91      0.92      0.90       659

Confusion Matrix:
 [[577   8]
 [ 48  26]]

Accuracy:  0.9150227617602428

The F1-score is the metric I am looking at.  I am having difficulties in explaining the results: I think are very bad results! May I ask you how I could improve it?
I am currently considering a model that looks at corpus of the emails (subject + corpus).
After Erwan's answer:
I oversampled the dataset and these are my results:
Logistic regression
              precision    recall  f1-score   support

         0.0       0.94      0.77      0.85       573
         1.0       0.81      0.96      0.88       598

    accuracy                           0.86      1171
   macro avg       0.88      0.86      0.86      1171
weighted avg       0.88      0.86      0.86      1171

Random Forest
              precision    recall  f1-score   support

         0.0       0.97      0.54      0.69       573
         1.0       0.69      0.98      0.81       598

    accuracy                           0.77      1171
   macro avg       0.83      0.76      0.75      1171
weighted avg       0.83      0.77      0.75      1171


texts:
I used MXNet previously to beat keras+tensorflow accuracy in CNN regression models.
Now I am trying to implement LSTM, which in keras runs fine:

from keras.layers import LSTM,Flatten,Input
from keras import Model
import numpy as np

def make_keras_stacked_lstm():
    inp = Input(shape=(100,1))
    lstm1 = LSTM(16, return_sequences=True)(inp)
    lstm2 = LSTM(1, return_sequences=True)(lstm1)
    outp = Flatten()(lstm2)
    return Model([inp], outp)

def keras_main():
    ins = np.random.uniform(size=(1000,100,1))
    outs = np.random.uniform(size=(1000,100))
    model = make_keras_stacked_lstm()
    model.compile(optimizer='sgd', loss='mean_squared_error')
    model.fit(ins, outs, epochs=1, validation_split=.1)

if __name__ == '__main__':
    # main()
    keras_main()

How can I translate this to MXNet, in either Symbol or gluon dialect? I found no "return_sequences=True" analogue.


texts:
I'm running a bidirectional LSTM. But this error is appearing:

TypeError: object of type 'Bidirectional' has no len()

What's wrong in this code? Please help.

 import pandas as pd
 import numpy as np
 from tqdm import tqdm, trange
 import unicodedata

 from keras.preprocessing.sequence import pad_sequences
 from keras.utils import to_categorical
 from keras.models import Model
 from tensorflow.keras.layers import Input
 from tensorflow.keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional

 # Defining Constants

 # Maximum length of text sentences
 MAXLEN = 180
 # Number of LSTM units
 LSTM_N = 150
 # batch size
 BS=48

 input = Input(shape=(MAXLEN,))

 model = Embedding(input_dim=n_words, output_dim=MAXLEN, input_length=MAXLEN)(input)

 model = Dropout(0.2)(model)

 model = Bidirectional(LSTM(units=LSTM_N, return_sequences=True, recurrent_dropout=0.1))(model)

 out = TimeDistributed(Dense(n_tags, activation="softmax"))(model)  # softmax output layer

 model = Model(input, out)

 model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
 history = model.fit(X, np.array(y), batch_size=BS, epochs=2, validation_split=0.05, verbose=1)


texts: Before facing this question, I always thought non-learnable problems are those which the provided data for the problem has high amount of outliers, those which don't have sufficient features or those for which the Bayes error is large because of having same features with different labels. As you can see, it seems that the data is fine because the learning should be comparable with human level inference. A human can distinguish between even or odd numbers by just looking at them. I know that we as human begins, do modulus two operation in our mind to decide whether a number is even or odd, the feature extraction part, but we are doing that with just the number itself. It is clear that we can not find a decision boundary to be able to generalize because the inputs have alternative behavior. 1 is even 2 is odd, 3 is even 4 is odd and all the other numbers in this manner. I want to know this kind of problem ,which does not have the mentioned problems which may cause an algorithm not to learn, has any special name?


texts:
I am trying to follow along with a project written by Mike Smales - "Sound Classification using Deep Learning".  In there, the author wrote a helper file called wavfilehelper.py:
wavehelper.py Code
import struct

class WavFileHelper():

    def read_file_properties(self, filename):

        wave_file = open(filename,"rb")

        riff = wave_file.read(12)
        fmt = wave_file.read(36)

        num_channels_string = fmt[10:12]
        num_channels = struct.unpack('<H', num_channels_string)[0]

        sample_rate_string = fmt[12:16]
        sample_rate = struct.unpack("<I",sample_rate_string)[0]

        bit_depth_string = fmt[22:24]
        bit_depth = struct.unpack("<H",bit_depth_string)[0]

        return (num_channels, sample_rate, bit_depth)

In his main program he calls the helper file like this:
from helpers.wavfilehelper import WavFileHelper

wavfilehelper = WavFileHelper()

However, when I run this block of code in PyCharm, it complains "ModuleNotFoundError: No module named 'helpers.wavfilehelper'"...how can I get this helper file to work in the PyCharm environment?  Do I have to put the wavehelper.py file in a special folder to be called?
Any help will be greatly appreciated!


texts:
I am beginner in data science. I have this "aids" dataset from "mdhglm" package in R.

dataset = aids, info = Repatead Measures on AIDS Data

data("aids", package = "mdhglm")

Here, I want to know what is my target variable?
I am sorry, if the question is too basic.


texts:
I've this matrix confusion:

[9779  107]
[2227  148]

What is the accuracy of my model? My doubt is because the confusion matrix is calculated based on Test dataset so how can it evaluate the accuracy of my model?

Thanks!


texts: How can I replace outliers with maximum non-outlier value?

texts:
I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no outliers).

How can I impute this value in python or sklearn? I guess I can remove the values, get the max, replace the outliers and bring them back. But hoping there’s a function for that already.

Second, is this a bad idea? I see others remove the outlier completely or replace with the mean or median. I wonder if my approach is wrong.


texts:
Our little group at uni is investigating if there is a relationship between 3 measures of social well-being (social anxiety, social connectedness and self esteem) and usage time (on-screen time in hours) on several social media platforms. Would you recommend that we use correlation or regression in this case?

Thanks in advance :)


texts: While sampling the corpus via Skip-gram, should the resulting target-context pairs be kept unique for the training?

texts:
Should I prepare my training data for word2vec Skip-gram embedding as unique target-context word pairs discovered throughout the corpus? Or should the repeated occurrences of the same pairs be present multiple times in the training data as well?
I am well aware of the subsampling method, but it is unclear whether I should downsample frequent words (so unique pairs containing them will be sampled less likely into the training) or I should downsample frequent pairs entirely.
My intuition is that if frequent pairs are presented more to the network during training as opposed to infrequent ones, then their embedding will have closer resemblance. However, if I understand right, Skip-gram seems to only differentiate between context and non-context words, and gives no importance to how "close" of a context or non-context they are.


texts: Why does MAE differ after prediction (Neural Network)?

texts:
I'm having trouble understanding what's happening in the following code.
I already have defined x_train, y_train, x_val, y_val and x_test which define my training, validation and test sets. I'm using keras library.

model = Sequential()
model.add(Dense(8,input_dim=x_train.shape[1], activation='relu'))
model.add(Dense(1, activation='linear'))
model.summary()

rms = optimizers.RMSprop(lr=0.0003)

model.compile(optimizer=rms,
              loss='mean_squared_error',
              metrics=['mae'])
history = model.fit(x_train, y_train, epochs=1000, validation_data=(x_val, y_val), batch_size = 32)

pred_train = model.predict(x_train)
pred_val = model.predict(x_val)
pred_test = model.predict(x_test)

MAE_train = np.mean(np.absolute(y_train - pred_train ))
print('Mean absolute error for  training is MAE = '+str(MAE_train))
MAPE_train = 100 * np.mean(np.absolute(y_train - pred_train )/ y_train)
print('MAPE for training is MAPE = '+str(MAPE_train))
print(pred_val)
MAE_val= np.mean(np.absolute(y_val - pred_val ))
print('MAE  val is MAE = '+str(MAE_val))
MAPE_val = 100 * np.mean(np.absolute(y_val - pred_val )/ y_val)
print('MAPE for val is MAPE = '+str(MAPE_val))

here is the result of an execution :

What I don't understand is that the val_mean_absolute_error that is shown at the end of the training differs from the one that I compute.
At first I tried to save my best model using Callbacks and then predict with it but the values also differ in this case.

What am i missing ?


texts: Overfitting in an unsupervised technique

texts:
I am trying to understand if over-fitting can happen in an unsupervised technique like kmeans clustering. Could someone help me understand if and how this would happen?

Thanks.


texts:
I have a dataset which has two columns:
title      price
sentence1  12
sentence2  13

I have used doc2vec to convert the sentences into vectors of size 100 as below:
LabeledSentence1 = gensim.models.doc2vec.TaggedDocument

all_content = []
j=0

for title in query_result['title_clean'].values:
    all_content.append(LabeledSentence1(title,[j]))
    j+=1
print("Number of texts processed: ", j)

cores = multiprocessing.cpu_count()

d2v_model = Doc2Vec(dm=1, vector_size=100, negative=5, hs=0,
min_count=2, sample = 0, workers=cores, alpha=0.025,
min_alpha=0.001)

d2v_model.build_vocab([x for x in tqdm(all_content)])

all_content  = utils.shuffle(all_content)

d2v_model.train(all_content,total_examples=len(all_content), epochs=30)

So d2v_model.docvecs.doctag_syn0 returns me vectors of all the sentences
I want to now perform clustering using DBSCAN but since I have the other price column which is numeric I am having some trouble fitting the final data to the model. I have a similar problem as described in Stackoverflow, one of my columns has an array of 100 sizes each row, and the other column is just numeric. Hence when I perform dbscan on the data I get the same error.
Is there any smart way to handle such cases? Combining doc2vec output with other numerical columns to prepare it for clustering? Something like this, where both_numeric_categical_columns is the desired input to the model:
clf = DBSCAN(eps=0.5, min_samples=10)
X = clf.fit(both_numeric_categical_columns)
labels=clf.labels_.tolist()

cluster1 = query_result_mini.copy()

cluster1['clusters'] = clf.fit_predict(both_numeric_categical_columns)


texts:
Dataset: I'm given the number of minutes individual customers use a product each day and am trying to cluster this data in order to find common usage patterns.

My question: How can I format the data so that, for example, a power user with high levels of use for a year looks the same as a different power user who has only been able to use the device for a month before I ended data collection?

So far I've turned each customer into an array where each cell is the number of minutes used that day. This array starts when the user first uses the product and ends after the user's first year of use. All entries in the cells must be double values (e.x. 200.0 minutes used) for the clustering model. I've considered either setting all cells/days after the last day of data collection to either -1.0 or NULL. Are either of these a valid approach? If not what would you suggest?


texts: Optimizing an averaged perceptron algorithm using numpy and scipy instead of dictionaries

texts:
So I'm trying to write an averaged perceptron algorithm (page 48 here for the equation) in python. Instead of storing the historical weights, I simply accumulate the weights and then multiply consistency counter, $c$, that is the variable w_accum.

My implementation initially had the weight vectors and x represented as dictionaries where a feature is in the dictionary only if it's active, that was supposed to be the most efficient way I could think of. Here is that code:

def train():
    #Initialize w, bias
    self.w, self.w['bias'] = {feature:0.0 for feature in features}, 0.0
    self.w_accum, self.w_accum['bias'] = {feature:0.0 for feature in features}, 0.0
    c = 0
    # Iterate over the training data n times
    for i in range(iterations):
        # Check each training example
        for i in range(len(x_train)):
            xi, yi = x_train[i], y_train[i]
            y_hat = self.predict(xi)
            #Update weights if there is a misclassification
            if yi != y_hat:
                ## self.w_accum += self.w*c (basically that's what I'm doing)
                self.w_accum = {feature:self.w_accum[feature]+c*self.w[feature] for feature in self.w_accum}
                c = 1
                for feature, value in xi.items():
                    self.w[feature] = self.w[feature] + yi*eta*value
                self.w['bias'] = self.w['bias'] + yi*eta
            else:
                c += 1

with predict(self, x) being defined as follows:

def predict(self, x):
    if (x.shape[1] == (self.w.shape[0]-1)): # Bias
       x = hstack([x, np.ones(1)])
    s = np.dot(x.toarray(),self.w)
    return 1 if s > 0 else -1

The problem with this implementation is that it got SIGKILL'ed on massive data sets, and I had to come up with a different solution, so I turned to scipy's sparse.DictVectorizer and ran it over the dictionaries to turn them into sparse dictionaries to use. Now my w and x are represented as csr_matrix objects. Here is what the new updated code looks like:

 def train(x_train, y_train):
        #Initialize w, bias
        len_features = x_train[0].shape[1]
        self.w = np.zeros((len_features+1,1))
        self.w_accum = np.zeros((len_features+1,1))
        c = 0
        # Iterate over the training data n times
        for i in range(epochs):
            # Check each training example
            for i in range(x_train.shape[0]):
                xi, yi = x_train[i], y_train[i]
                xi = hstack([xi, np.ones(1)])
                y_hat = self.predict(xi)
                #Update weights if there is a misclassification
                if yi != y_hat:
                    self.w_accum += self.w*c
                    c = 1
                    self.w += xi.transpose()*eta*yi
                elif yi == y_hat:
                    c += 1

and the predict function is as follows:

def predict(self, x):
    if (x.shape[1] == (self.w.shape[0]-1)): # X_{n+1} = 1
       x = hstack([x, np.ones(1)])
    s = np.dot(x.toarray(),self.w)
    return 1 if s > 0 else -1

The problem is that this code is actually SLOWER than my previous implementation! I have no idea why. The code seems to take a long time around the x.transpose() command (third to last line in train) and in the np.dot call.

I thought using a sparse dictionary was supposed to be the solution? Is it because I'm using a dot product over all the features, even the zeros? Is there a way to fix this? I thought sparse vectors were optimal for these sorts of things?


texts: fast fourier transformation on images -- python


texts: Split columns by finding "£" and then converting to minvalue, maxvalue

texts:
I have a pandas dataframe with a salary column which contains values like:

  £36,000 - £40,000 per year plus excellent bene...,
  £26,658 to £32,547 etc

I isolated this column and split it with the view to recombining into the data frame later via a column bind in pandas.

I now have an object with columns like the below. The columns I split the original data frame column I think are blank because I didn't specify them (I called df['salary']=df['salary'].astype(str).str.split()
)

So my new object contains this type of information:

[£26,658, to, £32,547],
[Competitive, with, Excellent, Benefits]

What I want to do is:

Create three columns called minvalue and maxvalue and realvalue
List items starting with £ (something to do with "^£"?
Take till the end of the items found ignoring the £ (get the number out) (something to do with (substr(x,2,nchar(x)))?
If there are two such items found, call the first number "minvalue" and call the second number "maxvalue" and put it below the right column. If there is only one value in the row, put it below the realvalue column.

I am very new to pandas and programming in general, but keen on learning, your help would be appreciated.


texts: Clustering big data by reducing data accuracy?

texts:
I have 1 million rows with 20 attributes to do hierarchical clustering. When I want to build a distance matrix on this data by dist() in R, it says that it needs 5 TB memory. I have these approaches:

Reduce the number of rows by sampling
Change the method of clustering
?

Now, do you suggest another approach?
And I have an idea, I think if I reduce the accuracy of the values and then doing "group by", then I can remove duplicated rows and have a new column with the count of duplicates for each row.
Is there any R package that can do hierarchical clustering with these data?

"group by": count number of duplicated rows and add a column that say how many times this row was duplicated in source.


texts: Is Cross Validation needed for regression if you already know the predictors in your model?

texts:
Let's say you want to model the behavior of Y = X1 + X2 and you know that this is the model you want to make.  Whether or not that approximates the true relationship well is unknown. But since you want to be able to have coefficients that explain how Xi affects Y, you build a regression model.  You don't plan on adding/subtracting predictors (since you don't have any additional data) and you don't plan on comparing this model with another (no other model allows for interpretation).
Does it make sense to still use sample splitting or cross validation?  If you do cross validation, do you average the coefficients?  Or could you just use your entire data set to train the model.
Thanks!


texts: What is purpose of partial derivatives in loss calculation (linear regression)?

texts:
I am studying ML and data science stuff from scratch. As a part of the course, I am studying how the models are derived. And for most of them, starting with the simplest - linear regression, we take partial derivatives. I understood its implementation part, however, I am a bit confused about why we need to take partial derivative there.

Is there any specific reason behind it? Can we use any other methodology to compute linear regression loss function?