texts:
I'm trying to implement a Double Dueling DQN on LunarLander and I'm facing an issue as my model is not learning so I'm trying to debug the graph and this leads me to a question regarding the prediction of the Q-value.

Q(s,a) = V(s) + A(s,a) − avg( A(s,a) )

I understand this equation in 2 different ways. Let's imagine we have a 2-actions environnent with A = [0,6, 0,4], V = 1,2 and we chose action = 0.

Based on how I did it (let's call it Numpy's style), the Q-predicted is :

Q = 1.2 + [0,6, 0,4] - 0.5 = [1.3, 1.1]

But I can also understand it as if we compute only the value on the used action:

Q = 1.2 + 0.6 - 0.5 = [1.2 + 0.6 - 0.5, 0.4] =  [1.3, 0.4]

This won't impact the error because the Q target will be the same as the Q-value except the index of the taken action but it can impact the prediction of action in the argmax as we fixed the n-action as only 1 "value" so actions are "linked".

Am I thinking the wrong way ? Is the Numpy's style right ?

Thanks for your light about it,
Nicolas


texts: What is the difference between keras.layers.core and keras.layers

texts:
We can import a Dense layer from:
1.from keras.layers import Dense
2. from keras.layers.core import Dense
Similarly for Conv2D:
1.from keras.layers.convolutional import Conv2D
2.from keras.layers import Conv2D
So what is the difference between the above lines.If they are the same why the syntax is different?


texts: Do timesteps must have the same temporal distance in training a RNN?

texts:
I have a recurrent neural network with LSTM units that I want to train with batches of 6 timesteps. Each timestep is a record of a dataset and represents the temporal aggregation over 5 minutes of data taken every 10 seconds. Unfortunately, the dataset has temporal interruptions, so sometimes two successive records in the dataset can be temporally separated by 10 minutes to a few weeks. I was wondering if the timesteps I give to the network in each batch must all be separated by the same temporal distance, that is in my case they must all be within 5 minutes of each other or it is sufficient that they are subsequent and therefore I could also give the network for example a batch with 4 records at a distance of 5 minutes from each other and then 2 taken for example two days later.

UPDATED:
The data comes from an electronic system so the time interruptions are due to the device being turned off. Rarely it is turned off only for a few minutes while most of the time it remains off for longer periods, like hours or sometimes days. Among other elements, the device includes a battery, therefore many of the features present in the dataset records are temperature, current and voltage readings which change value during temporal interruptions (little when turned off for a few minutes and significantly when turned off for hours or days). The goal of the network is to predict the value of some features two steps ahead.


texts: Choosing loss function in Keras for prediction binary_crossentropy or categorical_crossentropy

texts:
What loss function in keras should I chose for binary_crossentropy or categorical_crossentropy?

I have data like : $w1,w2,w3,w2,w2,w1,w3,w5,w9,w5,w4...$

I want to predict sequence:

input: $w1,w2,w3$  -> $LSTM$ -> get output: $w2,w2,w1$

I encoded the symbols $w1,w2...$ by obe-hot-encoding

So the input of the model is : $[[1,0,0],[0,1,0],[0,0,1]]$

The output of the model is : $[[0,1,0],[0,1,0],[1,0,0]]$


texts: Error while using the pandas_datareader package

texts:
I am trying to do a basic project where I grab some data from Morningstar or Google Finance, but when I import the package according to the usage instructions on GitHub and run Python in Pycharm, it returns the error:

ImportError: cannot import name 'is_list_like'

What should I do? Obviously I am doing something wrong.


texts: Can distortion be derived from inertia rather than recalculating it from scratch in case of kmeans?

texts:
I got this definitional difference between distortion and inertia from here:

Two values are of importance here — distortion and inertia. Distortion
is the average of the euclidean squared distance from the centroid of
the respective clusters. Inertia is the sum of squared distances of
samples to their closest cluster centre

However, when I searched for an example from here:

distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                        'euclidean'), axis=1)) / X.shape[0])

I took one inertia value from:

and 1 distortion value from :

however the values 217.64 and 3.45 aren't simply a division of 217.64 by count of variables in x (17).
Can anyone detail on what exactly is the formula for distortion?


texts:
Is there any work done on analyzing sequence of frames from a video using Deep Learning techniques?

By "analyzing" I mean like memorizing them in order to classify or predict something (e.g. by taking into account first 10 frames of a video the model can make some sort of conclusion).


texts:
I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on-

What data preprocessing steps should I perform. The dataset is
highly imbalanced (500 failures and ~40000 non-failures)
What type of Machine Learning models should I take into consideration as data is highly imbalanced?
Few days back, I read about Survival Analysis
and now I am in conundrum whether the problem would be of Survival
Analysis or Machine Learning?

I am currently working with dataset provided by BackBlaze(https://www.backblaze.com/b2/hard-drive-test-data.html).

It would be really great it I could get some direction :)


texts: Ensemble of different reservoirs (echo state networks)

texts:
Suppose I want to do reservoir computing to classify the input to the proper category (e.g. recognizing a handwritten letter).
Ideally, after training a single reservoir and testing it, there would be an output vector y with one value close to 1 and the others close to 0. However, this is not the case in practice, and I don't want to make the reservoir bigger at the moment.
I was therefore thinking of combining the predictions of a number of independent reservoirs for higher precision. After all, each reservoir does not only contain information on the 'most probable' output, but also on the 'second most probable' etc.
I have by now found out that renormalizing the output from each reservoir to a probability can be done with the 'softmax' function. But how to combine the probabilities assigned by the individual reservoirs to a total probability, with the hope that combining enough independent reservoirs would lead to a single element of the total y that is a clear outlier?
Some ways that I thought of:

Multiply the probabilities obtained through the different reservoirs (this seems the most intuitive to me)
Take the average
For each letter that has to be classified, just pick the reservoir that assigns the highest probability to one of the possibilities; or where the difference between first and second most probable possibility is largest

Are such schemes appropriate? Or should I do something else? I saw a paper where they trained a second layer to combine inputs from different reservoir, but this seems too far-fetched at the moment, something simple would be preferable.


texts: Can you please recommend papers/github or smth about object detection on RGB-D images (NOT 3d cloud points).The result should still be objects in rectangles in the 2d image, as in the usual methods for object detection like YOLO and others. All I can find is Silent Object detection methods, but it seems like not what I'm looking for.


texts:
I was reading this article https://www.di.ens.fr/~aspremon/PDF/CovSelSIMAX.pdf, whose goal is to estimate the covariance matrix from a the sample covariance matrix drawn from a distribution $X$.
' Given a sample covariance matrix, we solve a maximum likelihood problem penalized
by the number of nonzero coefficients in the inverse covariance matrix. Our objective is to find a
sparse representation of the sample data and to highlight conditional independence relationships
between the sample variables.'

The likelihood problem is only for the case where $X$ is a multivariate normal variable, so why did  the author apply their method to a random distribution in section 4. ?

Why are we seeking for a sparse covriance matrix ? Doesn't the covariance matrix have to be unique?

Thanks!


texts: CNNs: understanding feature visualization Channel Objectives (SOLVED)

texts:
I'm trying to follow a paper on deep NN feature visualization using beautiful examples from the GoogLeNet/Inception CNN. see: https://distill.pub/2017/feature-visualization/

The authors use backpropagation to optimize an input image to maximizes the activation of a particular (Inception) neuron/feature, or entire channel.

For example, Inception Layer 4a, Unit 11 is feature 12 of 192 from the 1x1 convolution path of Inception Layer 4a before filter concatenation (see: https://distill.pub/2017/feature-visualization/appendix/googlenet/4a.html#4a-11).

For Layer 4a 1x1 convolution the shapes are:

# Layer 4a
input:    [14,14,480]
output:   [14,14,512]

# 1x1 convolution
kernel:   [1,1,480]   # total 192 kernels
output:   [14,14,192] # channels [0..192] of Layer4a output
Layer4a slice:    tf.slice( layer4a_output, (0,0,0), (14,14,192) )

# Layer4a Unit 11
layer4a_unit_11 = tf.slice(layer4a_output, (11,0,0), (1,1,1)) # numpy [11,1,1]

In a related article, the authors state (see: https://distill.pub/2018/building-blocks/) ,

  "We can think of each layer’s learned representation as a three-dimensional cube. Each cell in the cube is an activation, or the amount a neuron fires. The x- and y-axes correspond to positions in the image, and the z-axis is the channel (or detector) being run."

Furthermore, they offer a diagram which super-imposes the cube of Layer4a over the input image with the (x,y) axis overlaying the image itself.

I understand that the Neuron Objective is the input image that produces the highest activation for Layer 4a, Unit 11 which can be found at index=[11,0,0] of Layer 4a output=[14,14,512]. In this case, (x,y)=[11,0]. Each [1,1,480] kernel generates a feature map of shape=[14,14,1] with a total of 196 activations.

kernel => channel or feature map and activation => neuron or feature.

Question

But what is the intuitive concept of the (Positive) Channel Objective? In this example, Unit 11 sits in the same channel as 14x14=196 other neurons, but the channel objectives for all these neurons are different. If the optimized image for the Channel Objective maximizes the sum of neuron activations for channel 0, (e.g. slice=[14,14,0] of 192 1x1 convolutions or 512 total layer 4a channels) wouldn't it be the same for all 192 neurons in the same channel? Obviously, by the examples we see this is not true.

How does the Channel Objective relate to the Neuron Objective for Unit 11?

ANSWER

  I understand that the Neuron Objective is the input image that produces the highest activation for Layer 4a, Unit 11 which can be found at index=[11,0,0] of Layer 4a output=[14,14,512].

This is where my understanding went off the rails. Layer 4a Unit 11 is actually  channel/feature 12 of 192 for the 1x1 convolution. It is NOT the 12 of 196 neuron of channel 1. My fault for confusing 192 channels with 196 neurons/channel.

Instead, as mentioned the in answer, Unit 11 is a single neuron in channel 11, usually located near the center, e.g. Neuron Objective is (x,y,z)=(7,7,11) and Channel Objective is (x,y,z)=(:,:,11)


texts: Difference between RMSProp and Momentum?

texts:
Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the difference as : "RMSProp and Momentum take contrasting approaches . While momentum accelerates our search in direction of minima, RMSProp impedes our search in direction of oscillations. "
I don't get this statement . Can someone elaborate on the difference between the two ?


texts: Very low probability in naive Bayes classifier 1

texts:
I have some training data (TRAIN) and some test data (TEST). Each row of each table contains an observed class (X) and some columns of binary (Y). I'm using a Python script that is intended to predict the probability (Pr) of X given Y in the test data based on the training data. It uses a Bernoulli naive Bayes classifier. Here is my script:

https://stackoverflow.com/questions/55187516/look-up-bernoullinb-probability-in-dataframe

It works on the dummy data that is included with the script.

On the real data, I know from experience which class some of the Y columns are indicative of. My script however is giving probability predictions like "1" where I don't think that the class is correct and "6e-77" on correct classes.

Any advice on what I can try please?

Edit

There are two problems. The very low probability is caused by the naive assumption that nothing is related to anything else. This is described here:
https://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_curve.html#sphx-glr-auto-examples-calibration-plot-calibration-curve-py

The incorrect answers are caused by my code getting confused about which class is which, as described on my Stack Overflow post.


texts: Which statistical test tells which classifier performs better than the other?

texts: I have 3 classifiers: A, B and C. According to accuracy, specificity, sensitivity, f-score, and g-mean, say classifier B performs best. Now I want to statistically validate this claim. How should I do it? Will McNemar's test be enough to validate it? Which statistical test will tell me which classifier is better than the rest? And how?


texts: Filling 2 different values for missing/NaN columns

texts:
I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN values in 6-7 columns.
I want to fill those NaN with mode() / most_frequent() of each column. But it should be filled with two different values depending on the target.
So, NaN for all those with Target=1; mode() of values with Target=1 should be filled and similarly, for all those with Target=0; mode() for of values with Target=0.

I've tried the following code:

train_df[train_df.TARGET==1].fillna(train_df[train_df.TARGET==0].mode(),inplace=True)

and

train_df[train_df.TARGET==0]=train_df[train_df.TARGET==0].fillna(train_df[train_df.TARGET==0].mode())

Both of these codes didn't worked. I was still left with the same amount of NaN values.

But, when I run this-
    train_df[train_df.TARGET==0].fillna(train_df[train_df.TARGET==0].mode()).isnull().sum()

I observed all the missing values do get filled where Target=0.

Sometimes, I get this Warning:

/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py:6287: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._update_inplace(new_data)

I'm stuck at this for a long time and don't know how to proceed. Please suggest some solution to this.

Thanks


texts:
Imagine that I've a field called date in this format: "yyyy-mm-dd" and I want to convert to number like "yyymmdd". For that I'm trying to use this:

Data_ID = FOREACH File GENERATE CONCAT((chararray)SUBSTRING(Date,0,4),(chararray)SUBSTRING(Date,6,2),(chararray)SUBSTRING(Date,9,2));

But I'm getting a list of nulls...

Anyone knows what I'm doing wrong?

Thnaks!


texts: Does anyone know how google play music's recommendation engine works?

texts:
What sort of models do they use?  Presumably some flavor of neural net.  Do they do a lot of feature engineering?  Or do they throw in huge raw matrices of oscilloscope output?  Given that it must be supervised learning, what is their target?  A thumbs up?  How do they deal with heterogeneity in the population? We're not independent nor are we identically distributed.

It seems like it works better lately.  Curious how it works.


texts:
I think they share a lot (e.g. machine learning is a subset of both, right?), but maybe both have elements the other doesn't have?

Could you name some in that case? Or is one a subset of the other?

What is the relationship between AI and data science?

For example, when it comes to the relationship of AI and ML, I always say AI is a superset of ML. And the distinguishing set is search algorithms, which I would include in AI but not in ML. Would search algorithms be included in data science?


texts: Batch Normalization and Dropout together causing incorrect segmentation results

texts:
So, I've been running a test to see how well a number of networks can perform road segmentation on a particular customer's dataset.  I am testing UNET, RDRCNN, and Tiramisu against each other.

UNET reliably trains, however RDRCNN and Tiramisu cannot.  Immediately during training, the validation loss and accuracy plateau.  After looking at the results, it looks as if the network is guessing a single pixel value; leading to inferences that are just a square of all one color.

I have tried turning the learning rate down, using a simple optimizer like SGD, and making sure I am loading the data with the same exact loader.  I have also tried using a dice/jaccard loss to account for class imbalance.  None of these have worked.

Has anyone ever experienced this before? Any help would be greatly appreciated.  Thank you!


texts: How can I predict the post popularity of reddit.com with hidden markov model(HMM)?

texts: If I get some posts on reddict.com, how can I predict whether this post will (trending/hot/popular) in the future or not? I would like to use the hidden markov model to predict it, but I don`t know how to define the hidden states and observation sequence...can anyone give me any suggestion? Thanks~


texts: SVM hyperplane equations for linearly seperable data

texts:
I'm going through Wikipedia article on SVM and came across these equations:

It says:

  With a normalized or standardized dataset, the parallel hyperplanes to
  our required hyperplane $wx+b=0$, can be described by the equations
  $wx+b=1$ and $wx+b=-1$

What is meant by normalized and standardized data here? If the data is not standardized, what equation should be used?


texts: How to down-weight non-words in text classification?

texts:
Background:

Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images.  This works well, except when it doesn't.  Many docs are old, scanned images and what Tika extracts is gibberish.

Using Spark on Hadoop and either ML or MLlib (haven't settled, though I like ML better).

So far getting best results from a pipeline using Naive Bayes that removes Stopwords, tokenizes and Countvectorizes features (no Tf-Idf).  Total bag-of-words approach.  Next best is using ML to tokenize, Tf with Idf and into LogisticregressionwithLBFGS.

Anyway, the thought occurred to me that the model uses many docs that are junk.  Literally just strings of gibberish like "mmmmmmmm aaannnammmmrrr hdhhdhhhhhjjj..."

This isn't good, but since I'm operating at scale it's just what happened.  Certainly I could pick through 10,000 training docs and remove the bad examples, but there has to be an easier way.  Is there?

The title of this question belies my brainstorm that there might be a way to discount, downweight or outright ignore tokens that aren't discernable by a dictionary.  Is there?

Open to any and all advice or approaches to get better precision out of this model.

Thanks


texts: Time series forecasting using multiple time series as training data

texts:
I am trying to forecast the total attendance (ie. the number of entrances, which is also the number of tickets bought) of a festival just two days after it started. That is, knowing how many people went to the event during the first two days, how to predict the total number of people that will have visited the festival?

I do know it seems difficult at first glance, and that I theoretically could only make pretty bad forecasts, but here's the deal : I have organized more than thirty festivals in the past, and have collected data on each of them. Specifically, on each of these festivals, I have a daily time series where I know:

the number of tickets bought daily
whether the day was a weekday or during the week-end
whether the day was a day of public and school holidays
the daily weather

I have observed that all of these time series follow multiple trends. For instance, attendance always is at its best on saturdays, and at its worse on tuesdays... Likewise, there seems to always be more people coming in the very last days of the event than in beginning. These trends are the same for almost all the festivals. When decomposing the time series, I observe close trends, and close seasonal values.

Another thing, which I guess is no good news, is that the events had different timespans : some lasted 4 days, others 5, 6, 7 and even 8 days. Some started on a monday, others on a saturday.

So my questions is: how could I use these time series as a training data to try to forecast total attendance at the event knowing the attendance of the very first days. That is to say, which model could I use to predict total attendance of the event knowing I have all of this data ? I was of course thinking of machine-learning (or deep-learning) since I have a lot of training data, but I'm unsure whether it can be easily implemented in R or Python...

In order to do forecasts, I do of course know, for the on-going festival, how long it will last, whether it is going to take place during public and school holidays or not, whether it is going to span over a week-end, and I have the weather forecasts for each day.


texts: Explanation behind the calculation of accuracy in deep learning model

texts:
I am trying to model an image segmentation problem using convolutional neural network. I came across code in Github which I am not able to understand the meaning of following lines of codes for calculation of accuracy -

def new_test(loaders,model,criterion,use_cuda):
    for batch_idx, (data,target) in enumerate(loaders):
        output = model(data)
###Accuracy
    _, predicted = torch.max(output.data, 1)
    total_train += target.nelement()
    correct_train += predicted.eq(target.data).sum().item()

model(data) outputs a tensor of shape B * N * H * W
B = Batch Size
N = Number of segmentated classes
H,W = Height,Width of an image


texts: Feature selection before or after applying filter in Time-series forecasting

texts:
I'm predicting ozone concentration based on meteorological variables and ozone value of the previous day. I applied savitzky golay filter to get rid of noise in the time-series dataset.

My question is, if I want to perform feature selection, do I do it before or after applying the filter? What is the logical order?
Because the feature importance is different before and after applying the filter.

Using XGBOOST, this is the feature importance before the filter:

And this is after the filter:

I'd really be grateful for any help or information.


texts: Integrate remote sensing & GIS data in a CNN

texts:
I am trying to develop a CNN model that takes as input a set of parameters derived from satellite imagery.

What are the necessary steps to go through in order to make the Geotiff data acceptable as a valid input for a keras model?


texts: Binary (Unary) Recommendation System with Biased Views

texts:
I would like to create a content recommendation system based on binary click data that also takes views into account.

What content a user has been exposed to, and therefore has the chance to click on, is currently biased by a rule based system that is not always documented. I do have view data (if a user saw the content on their screen, regardless of whether it was clicked.), and am wondering how to take this into account with a traditional matrix factorization recommendation system such as this item-item approach, or if there are other other better options.

Any suggestions for implementation in Python are a bonus!


texts: Predicting Multiple Values Values Using Time Series Forecasting

texts:
I want to illustrate my question with the following example:
I have a wholesale company through which I sell 200 products:
P1,P2,P3 .... P200
to a 1000 customers  C1,C2,C3 ... C1000
I keep my sales records in a database as a time series in which I keep the product_id,customer_id, amount and timestamp for each transaction over a period of 4 years.
My question is regardless of the forecasting algorithm (static or learning), can predicting the sales of all of these products made through a single model? or model for each product, or a model for each customer? If any or many of these choices can be valid why ?


texts: Unstable accuracy of CNN - When should I stop training?

texts:
I'm using caffenet for fine-tuning.
I'm doing cross validation (15 vs all) with a very small data set of about 250 images. I'm testing every 10 iterations (~2 epochs). My batch size is 50. With some sets I'm getting very unstable accuracy - Can jump from 70% to 90% and back to 70% and back and fourth. My question is: Let's say I hit 90% accuracy after 40 iterations (~8 epochs) - Does this mean that the net had reached an optimal state or could it be that it just had a lucky guess on the validation set? My final question is: Should I stop training and save the net? Thanks.


texts: How to deal with position bias in search?

texts: In search, position of the search result affects the click-through rate a great deal. How do people usually deal with this ? In practice how to remove such bias to create unbiased training data for training learning to rank model ?


texts: Does a Keras checkpointer save the best weights when using chekpoints with restore_best_weights?

texts:
I m training a sequence model in Keras using the tensorflow backend.  I've also included some callbacks to save checkpoints and revert to best weights if the model starts to overfit (which it will).

My question - when fitting using this set of callbacks, does the final checkpoint contain the version of the model with the best weight?  I know that the weights in classif_model will revert but I'm not sure if that also applies to the final saved state.

from keras import callbacks as kc

classif_model = my_model(input_shape)

# Set up callbacks
checkpointer = kc.ModelCheckpoint(filepath='results/'+name+'.h5', verbose=0)
earlystopping = kc.EarlyStopping(monitor='val_loss', patience=patience, restore_best_weights = True)
callbacks = [checkpointer, earlystopping]

# train the model
hist = classif_model.fit(x = X_tr, y = Y_tr, epochs = epochs, batch_size = batch_size,
                         callbacks = callbacks, validation_data = (X_val, Y_val),
                         verbose = 0)


texts:
I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than three). Does anyone have a suggested for how I might get started?

Is the "bag of words" approach feasible here? Later, I could add another classification stage based on document structure (perhaps decision trees).

I am somewhat familiar with Mahout and Hadoop, so I prefer Java-based solutions. If needed, I can switch to Scala and/or Spark engine (the ML library).


texts: In scikit learn, one of the parameters to set when instantiating a decision tree is the maximum depth. What are the factors to consider when setting the depth of a decision tree? Does larger depth usually lead to higher accuracy?


texts: Seemingly good results with training a CNN but bad when testing

texts:
I have an image classification task, and I am using Keras for a network with CNN layers, with what seems like good results in training, translates to poor performance in testing. Upon training, I quickly see an increase in accuracy and validation accuracy to around the following level:

4678/4678 [==============================] - 2s 427us/step - loss: 0.0607 - acc: 0.9795 - val_loss: 0.1605 - val_acc: 0.9590

Now my first reaction was that this might be too good to be true, turns it was. In testing the model, it had very bad performance.

Bashar Haddad suggests that there may be a data imbalance problem, but with the following data specifications of data, before being split into 25% validation, I don't think that's my problem:

(data is being fit consecutively in a for loop)
average number of datapoints in training: 4500
average number of datapoints in testing: 1500
average number of classes: 46 (max 49)

and most of the classes had a good amount of datapoints in each.

Based on this, I see the user found increasing batch size and decreasing learning rate helped, and I fit the model in the following way...

varying batch size from 100 -> 500
varying epochs size from 50 -> 250
cnn_model.fit(X_train, y_train,
                        batch_size=500,
                        epochs=50,
                        verbose=1,
                        shuffle=True,
                        validation_data=(X_test, y_test)
            )

Varying batch_size and epochs had little to no effect.

With both accuracy and validation accuracy high, and also very little difference between them while training, I am confused as to why using this model on test data results in such bad performance.

What could be reasons for this? And possible solutions to solve it?


texts: Improving model prediction for small single data sets by using multiple data sets to fit

texts:
Have a project I'm working on and am running into an issue.
Essentially I have lots of data sets with a small amount of data. These data sets represent locations in an area. I have one classification data set (y) with categorical values (number from 1 - 6), the location is in the middle of the rest of the data sets.
The issue with these points is that I don't get a lot of data per point (maybe 100 rows).
My goal was to fit a data set (physically closest to the target classification data set) and generalize it to the rest of data. It doesn't work very well, so now I'm trying to improve the model performance by adding 'k' data sets also close to the class data set.
Initially :
model.fit(X_1,y_class)
model.predict(X_2)

These data sets all share the same features and share the same rows as the classification data set.
With basic sklearn models I'm unable to take multiple inputs to fit, and one input to predict.
Essentially what I'm trying to do is the following :

model.fit([X_1,X_2,...X_k],y)

model.predict(X_5)

Where :
All features are numeric (floats)
Each X matrix is about 100 points long with a continuous index [0:100].
I only have one test point (with 100 observations) for each group of points, so it's imperative I use as much data close to the test point as possible to improve the model with the goal is to generalize the data to future points.
Is there another model or technique I can use for this? I've done a bit more research into NN models (not familiar so would prefer to avoid), and found that Keras has the ability to take multiple inputs to fit using their functions API, but can I predict with only one input after it has been fitted to multiple?
https://stackoverflow.com/questions/55233377/keras-sequential-model-with-multiple-inputs


texts:
I use Faster and Mask R-CNN for various problems, including training GANs. I'm particular interested in local features that Region Of Interest module in Faster R-CNN extracts from feature layer(s). Essentially it takes a prediction (or ground truth) from Region Proposal Network (RPN),  a feature map, and crops and resizes the feature map using the predicted coordinates/size.

What I need is an explanation/reference to better understand what these local features are, what they represent. I realize that feature maps in ConvNets are a black box, but if anyone has an understanding of local RoI features, it would be good to know.


texts:
I was reading about the solution to this OTTO Kaggle challenge and the first place solution seems to use several transforms for the input data X, for example Log(X+1), sqrt( X + 3/8), etc. Is there a general guideline on when to apply which kind transformations to various classifiers?

I do understand the concepts of mean-var and min-max normalization. However, for the above transformations, my guess is that Log and Sqrt are used to compress the dynamic range of the data. And the x-axis shift is just to recenter the data. However, the author chooses to use different methods of normalization for the same input X when feeding into different classifiers. Any ideas?


texts:
I'm trying to solve a simple problem and seeking advise/suggestions - I have lot of paper documents in my drawers which I want to store on cloud (S3) as PDFs.

My idea blueprint is -

1) scanning all of them using printer

2) store it locally in organised folder structure

3) upload them to S3 cloud - (python code using Boto3 ?)

Questions -

1) Does this sound good to start with? Please advise.

2) Is S3 the best for this in AWS stack? Or, anything else? (Apologies, completely new to AWS stack, hence aware of all its services and offerings.)

Fyi, I'm currently working on AWS Free Tier.

Thanks in advance!
Ranit

ps. This's my first post/question on AWS and my SO account is blocked to ask questions. Sincere apologies if it lacks essential details.


texts: In a neural network, does gradient vanish during a great number epochs as well, rather that only vanishing through different layers?


texts: How to use two different datasets as train and test sets?

texts:
Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. The problem that I've encountered, now that I'm trying to make my own classification algorithm (the text sends a positive/negative message) regards the training and the testing datasets. In all the examples that I've found, only one dataset is used, a dataset that is later split into training/testing. I have two datasets, and my approach involved putting together, in the same corpus, all the texts in the two datasets (after preprocessing) and after, splitting the corpus into a test set and a training set.

datasetTrain = pd.read_csv('train.tsv', delimiter = '\t', quoting = 3)
datasetTrain['PN'].value_counts()

datasetTest = pd.read_csv('test.tsv', delimiter = '\t', quoting = 3)
datasetTest['PN'].value_counts()

corpus = []
y = []

# some preprocessing
    y.append(posNeg)
    corpus.append(text)

from sklearn.feature_extraction.text import TfidfVectorizer
transf = TfidfVectorizer(stop_words = stopwords, ngram_range = (1,1), min_df = 5, max_df = 0.65)
X = transf.fit_transform(corpus).toarray()

# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.11, random_state = 0)

The reason why I've done this is because I'm working with the Bag of Words model and if I'm creating from the beginning X_train and X_test (y_train, y_test respectively) and not using the splitting function, I get an error when running the classification algorithm:

X_train = transf.fit_transform(corpustrain).toarray()
X_test = transf.fit_transform(corpustest).toarray()

...

classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)

ValueError: Number of features of the model must match the input. Model n_features is 2770 and input n_features is 585

I'm kind of new at this and I was wondering if anyone could please guide me in the right direction?


texts: How do I deal with missing values in a data set?

texts:
I am trying to build a binary classification model which predicts whether a patient would me infected with a certain disease at the the end of his hospital stay or not. The features that I have are results of different standard medical tests. But the issue is almost all of these results have around 60% - 80% missing values as not all the tests are relevant for all the patients. So how do I deal with the missing values as dropping them is not an option here.
Also since the medical test results lie on scale ranging from low to high, should i converted them to categorical variable with High, Low, Medium, Null (for missing data), based on the standard medical test ranges?


texts: How to interpret this 'Zipf' (survival, war casualties) plot?

texts:
Help me interpret this chart please?

What do the X and Y axes mean here? (I asked the author directly but they didn't respond for some reason.)

Any thoughts on what 'Surivival Function' might mean here?

War Casualties from 1800 until today, n data points


texts: Best practice for computing week-over-week change in MAU metric

texts:
When we calculate week over week change for a metric like "Monthly active users", we often refer to a calculation of SUM(This week) - SUM(Last week)/SUM(Last week), which means we will run total of this week's number first, and then compare to the total from last week.
There is another situation when we can have change any date comparing 7 days ago(week), So each date during a week will have a change value.

Are there any best practice for this "week over week" concept. I've also heard that someone is using "weekly change" referring to the similar thing. Is there any difference? Which calculation I should chose?


texts: select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations

texts:
I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, colsample_byTree, min_child_weight, and subsample). I did a literature review to specify the range and interval of values for each parameter. Using those ranges and intervals, the parameter space generated around 62,500 parameter combinations. I am using R caret::train function to generate the best hyperparameter combination for my dataset. However, the amount of simulations (62,500) is too much. I read about the Latin hypercube sampling (LHS) and I think that is what I need to reduce the number of simulations by applying initial selection of hyperparameters using LHS. But I am having trouble implementing the approach in my dataset. My goal is to generate a manageable number of hyperparameter combinations (i.e., ~500) using LHS, and then use caret::train function to select best parameters. I would like to ask for help in implementing LHS using my parameter space.
nrounds <- seq(from = 200, to = 1000, by = 200)
maxdepth <- seq(from = 2, to = 10, by = 2)
eta <- c(0.01, 0.05, 0.1, 0.2, 0.3)
gamma <- seq(from = 0, to = 0.4, by = 0.1)
colsample_bytree <- seq(from = 0.4, to = 1, by = 0.2)
min_child_weight <- seq(from = 1, to = 5, by = 1)
subsample <- seq(from = 0.6, to = 1, by = 0.1)
dataGrid <- expand.grid(nrounds, maxdepth, eta, gamma, colsample_bytree, min_child_weight, subsample)


texts: Predicting future value with regression Model

texts:
I have a set of predictor variables and another target variable .

Now I am really confused on what method to use to forecast the target variable .

For e.g my data set have customer profit(which is my target variable) and a set of predictor variable(balances of different account) for one year for each customer .

Now I need to predict profit of next 5 years .I am confused in the part that I dont have the data(predictor variables) for future .

What are my possible choices of modelling .Please assist .


texts: I'm a high school senior who is new to data science, and would like to get into natural language processing. I currently know nothing about NLP, and the information online can be overwhelming. What are tokens? What are they used for / why do we need to tokenize text?


texts:
I have a problem where I am trying to apply a one-class svm to detect outliers. I am training on a dataset of true cases using a one-class radial svm and then predicting for both false and true cases. It is worth noting that the true cases I am training on do contain a proportion of misclassified cases. This is the nature of the problem and not something that can be corrected for or identified easily.

I have applied hyperparameter tuning through a grid search with cross-validation to get the best values of nu and gamma for the model. However, when I do this and predict on all cases, I get poor separability between true and false cases based on the resulting decision values. I actually get far better separability from using a fixed value for C and gamma of 1 and 0.05 respectively.
My intuition is that this lack of separability is down to the misclassified cases in the true training data. As such, my question is should hyperparameter tuning be applied to a one-class svm on data with misclassified values? If so, should nu and gamma both be tuned?  Any papers discussing this problem would be great as I haven't found any resources relating to it.

I am aware that nu is related to C by nu = A+B/C where A, B are constants that are difficult to calculate and that

  The parameter nu is an upper bound on the fraction of margin errors
  and a lower bound of the fraction of support vectors relative to the
  total number of training examples

So nu of 0.01 would mean at most 1% of your training examples would be misclassified. A good discussion on nu is here. I am searching values 0.0001 0.0010 0.0100 0.0500 0.1000 0.1500 0.2000 0.2500 0.3000 0.3500 0.4000 0.4500 0.5000.
I am also aware that

  Gamma is a parameter of the RBF kernel and can be thought of as the
  ‘spread’ of the kernel and therefore the decision region. When gamma
  is low, the ‘curve’ of the decision boundary is very low and thus the
  decision region is very broad. When gamma is high, the ‘curve’ of the
  decision boundary is high, which creates islands of
  decision-boundaries around data points.

A good post on gamma with intuitive visualisations is here. I am searching across gamma values of 1x10^-04 1x10^-03 1x10^-02 1x10^-01 1x10^+00 1x10^+01 1x10^+02 1x10^+03 1x10^+04 1x10^+05

So should I hold nu constant at a high value to assume a high number of misclassified cases? Also should I search across different values of gamma?


texts: Multiple step prediction for non-time series

texts:
I have a public EHR dataset which contains info on
a) lab tests b) diagnosis c) surgical procedures d) drugs prescribed etc
Now, using the above data elements, I would like to predict the below
a) When will be the next 3 visits of the patient to the hospital?
b) What will be the reason for his next 3 visits?
For example - Patient A's next 3 visit will be on Jan 21st 2017 (for disease A) , Dec 15th 2017 (for disease B) and Mar 16th 2018 (due to disease C).
I think what I am trying to do is multi step forecasting but not just using a plain time series but with a data which has chronology of events occurred for a patient in hospital.
While I understand date level predictions may be too detailed, any other approach on how this can be done and how to formulate this problem would really be helpful.
Can experts here provide me the list of steps and techniques to follow? So, I can learn methods that you list and be aware of the tasks involved in this pipeline?


texts: What Naive Bayes method is being used in this example?

texts:
I'm reading an old book called "Data Mining Practical Machine Learning and Techniques";  It uses a tool called Weka to automate examples as well, but I'd prefer to use Python to do so.

I'm quite new to Machine Learning, and I'm just trying to line up examples with a computer API; I'm not certain of the type of Naive Bayes being used in the example (which I implemented in a spreadsheet from the book).  From the look of it, based on other examples in the book, it doesn't look to be multinomial-bayes, but I don't seem to be able to find an API that lines up with it either.

The example tries to determine if a sports team will go outside and play or not (yes, no) based on the Outlook (sunny, overcast, rainy), Temperature Classification (hot, mild, cold), Humidity Classification (high, normal) and if it is Windy or not (TRUE, FALSE).

The values are INPUT from data rows in the Weather Data Table 1.2 on page 11.

The processing of the INPUT begins and then in Table 4.2 The weather data with counts and probabilities they tally up probabilities and totals based on possible attribute values and totals.

They then add a new day row to the input and process which does not have a Will the team play value, but which does have all the other attributes with a single value for each.  The likelyhood is calculated for yes and no, and then in the Output a probability value for yes and a value for no are calculated.

I've included the spreadsheet here.

I realize this is really basic Machine Learning, but I just wondered if anyone could point me towards a python library or python algorithm that would handle such a problem...I've actually also tried to get it to work in Weka, and run into a similar issue in which I appear to be able to find a Naive Bayes algorithm without churning out the correct result presented in the book and in my spreadsheet.

So does this particular Naive Bayes method have a name and equivalent API?


texts:
I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this blog, and not really understanding how the quantile parameters affect the model (I am using the suggested parameter values on the blog).
When I apply this code to my data, I obtain nonsense results, such as negative predictions for my target values while my target values are always over 10K.
I don't understand how should this code vary according to my data and would really appreciate any help.

Differences in my data to the data that is used on the blog are:

My distribution is Poisson like.
I have over 100 features.

Note: I tried tuning the delta, threshold and var parameters, but they don't seem to have a controllable effect on the results and predictions remains nonsense.


texts: Histogram is extremely skewed to the left

texts:
everyone.
Hope you are all okay.

I am pretty bad at visualizations in Python.
I am working on a movie dataset and now I want to understand how values of variable 'vote_count' for every movie in the dataset are distributed.
So I am using very simple code:

plt.hist(df_movies.vote_count,bins=7)
plt.show()

And the output looks the following way:

As it is easy to notice the distribution is skewed very much to left.
I realize that I should  focus on plt.xticks(), but I tried to play around and the histogram still looked awful.

Could you please give me some advices how to make it more understandable and useful to get insights.

Thanks a lot in advance!


texts: "Optimal range for loss function": Myth or Truth?

texts:
I am currently working on a regression problem using a deep neural network that given a volume 32x32x256 in input need to generate a second volume of the same dimensions in output, this is not a segmentation problem and the volume generated by the net is formed by floats (to be more specific, im tryng to remove noise from a noisy x-ray beam).
As a loss function i am using the MSE beetwen the volume generated by the net and a ground volume already calculated without noise.
The volumes have been normalized on values between 0 and 1, before giving them to the net.
When we started the first trainings we were getting very low values in MSE (about 3*10^-4), but the deep-net was converging (on values near 5*10^6).
One of my co-worker stated that those values (of the loss) were far to low to get an optimal result with a framework as Keras adn Adam as optimizer, and proposed to change the loss function to get result within a range of 100 - 0.01, he proposed to do the sum of the square difference instead of the mean.
At first i was really confused by his statement, but a did a test:
Instead of normalizing the volumes on values between 0 and 1 i normalized them between values 0 and 1000, before giving them to the net.
In this test the MSE went from values of about 30'000 and coverged of values near 600, after denormalizing the data this new neural network gave result much more accurate of the first network, and just by changing the starting normalization!
Do you have any idea of the cause of this behaviour, was my coworker right?


texts: How to handle preprocessing (StandardScaler, LabelEncoder) when using data generator to train?

texts:
So, I have a dataset that is too big to load into memory all at once. Therefore I want to use a generator to load batches of data to train on.

In this scenario, how do I go about performing scaling of the features using LabelEncoder + StandardScaler from scikitlearn?

Some more context:

I have 10million+ samples of data with 23 features and 1 label column in a database.

My setup used to be (when it was ~3 million samples) to load in pandas with sql, perform some more feature extractions, use LabelEncoder on some features, do train/test split and then use StandardScaler on the training features. And then fit my keras model.

However this workflow is no longer possible on my machines because of the amount of data. (MemoryErrors.)

I'm looking into using keras.utils.Sequence to load batches of data instead of everything in memory at once, this way i would only need to have the complete list of indexes, and one full batch in memory at a time.

However how would I go about LabelEncoding and more importantly: How would I go about feature scaling in this scenario? And given the context, is this a correct approach?


texts: Can pinball loss be used to construct a prediction interval?

texts:
I'm modeling some time series data ($\{y_t\}_t$) and would like to construct a model that is able to return not just a single-value prediction $\hat{y_t}$, but an interval $C_t=(\hat{y}_{t, lower}, \hat{y}_{t, upper})$ such that $y_t \in C_t$ with some probability.

Now, I've learned about the pinball loss:
$$L(q, z)=\cases{qz, & z >= 0 \\ (q-1)z,& z<0}$$
If I understand it correct, $q \in (0,1)$ is the quantile that I want to predict, and $z$ is the difference between the actual value $y$ and what my model predicted $\hat{y}$. If the model is trained to optimize this loss for some $q$, it will return me the estimate $\hat{y}^{(q)}_t$ such that $P(y_t < \hat{y}^{(q)}_t)=q$. Is this interpretation correct?

Then, can I simply train two models, say one for $q=0.05$ and the other for $q=0.95$ in order to get the estimates of the intervals that contain the actual values with the probability $0.95-0.05=0.9$?


texts: Analyzing Sentiments of Financial News related to a Company

texts:
I'm trying to build a model which gives me the sentiments of the Financial News related to a company and I want to predict the stock price accordingly. But the major problem that I'm facing is understanding the news for the counterpart.

Let's say I have a news headline as "Total is under pressure and CEO has confirmed that they are planning to sell stakes soon". My model will always give negative sentiment which is correct, but this might actually be a good news for Shell or Exxon, lets say. But how do I tell my model that it is actually a good news for Shell.

Also is there any good process to understand which news relate to which companies and how I can calculate the sentiment accordingly. Maybe a good labelled data-set or pre-trained architecture which might help me out?

P.S. Most important, is there any labelled dataset or any other pre-trained architecture which I can use to calculate the sentiments of financial news?


texts: AUC and classification report in Logistic regression in python

texts:
I have been trying to implement logistic regression in python. Basically the code works and it gives the accuracy of the predictive model at a level of 91% but for some reason the AUC score is 0.5 which is basically the worst possible score because it means that the model is completely random. Also the classification report returns error: "UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)". Does anyone know what should I change so it works properly?

 import numpy as np
 import pandas as pd
 from sklearn.cross_validation import train_test_split
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import accuracy_score
 from sklearn.preprocessing import StandardScaler
 from sklearn.metrics import roc_auc_score
 from sklearn.metrics import classification_report

 data_file = pd.read_csv('loan.csv', delimiter=',')

 # variable preprocessing

 data_file['loan_status'] = np.where(data_file['loan_status'].isin(['Fully
 Paid', 'Current']), 1, 0)
 loan_stat=data_file['loan_status']
 loan_stat=loan_stat.astype(np.float64)

 m = {
    'n/a': 0,
    '< 1 year': 0,
    '1 year': 1,
    '2 years': 2,
    '3 years': 3,
    '4 years': 4,
    '5 years': 5,
    '6 years': 6,
    '7 years': 7,
    '8 years': 8,
    '9 years': 9,
    '10+ years': 10
 }
 emp_length=data_file.emp_length.map(m)
 emp_length.astype(np.float64)

 annual_inc=data_file['annual_inc']
 delinq_2yrs=data_file['delinq_2yrs']
 dti=data_file['dti']
 loan_amnt=data_file['loan_amnt']
 installment=data_file['installment']
 int_rate=data_file['int_rate']
 total_acc=data_file['total_acc']
 open_acc=data_file['open_acc']
 pub_rec=data_file['pub_rec']
 acc_now_delinq=data_file['acc_now_delinq']

 #variables combined into one dataframe

 X=pd.DataFrame()

 X['annua_inc']=annual_inc
 X['delinq_2yrs']=delinq_2yrs
 X['dti']=dti
 X['emp_length']=emp_length
 X['loan_amnt']=loan_amnt
 X['installment']=installment
 X['int_rate']=int_rate
 X['total_acc']=total_acc
 X['open_acc']=open_acc
 X['pub_rec']=pub_rec
 X['acc_now_delinq']=acc_now_delinq
 X['loan_stat']=loan_stat

 X=X.dropna(axis=0)
 y=X['loan_stat']
 X=X.drop(['loan_stat'], axis=1)

 scaler=StandardScaler()
 X=scaler.fit_transform(X)

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
 random_state=42)

 model=LogisticRegression(penalty='l2', C=1)
 model.fit(X_train, y_train)
 score=accuracy_score(y_test, model.predict(X_test))
 roc=roc_auc_score(y_test, model.predict(X_test))
 cr=classification_report(y_test, model.predict(X_test))

Here is the link to the data: https://www.kaggle.com/wendykan/lending-club-loan-data/downloads/lending-club-loan-data.zip


texts:
My question would be about backpropagation and understanding the terms feedforward NN vs backpropagation. I have two questions really:

If I understand correctly, even a feedforward network updates its weights (via a delta rule for example). Isn't this also backpropagation? You have some random weights, run the data through the network, then cross-validate it with the desired output, then update the rules. This is backpropagation, right? So what's the difference between FFW NN and RNN? If you can't backpropagate on the other hand, how do you update the weights in a FFW NN?
I've seen NN architectures looking like this:

Basically all the neurons are being fed the same data, right? OK, the weights are randomized and thus different in the beginning, but how do you make sure Temp. Value #1 will be different from Temp. Value #2, if you use the same update rule?

Thank you!


texts: Why do we determine the values of λ in regularization as ln λ, such as ln λ=-18 instead of for example λ=0.3?

texts:
I'm studying Pattern Recognition and Machine Learning by Christopher Bishop. What I realized is, he defines values of λ as ln λ. For example:

  We see that, for a value of lnλ = −18, the over-fitting has been suppressed and we
  now obtain a much closer representation of the underlying function sin(2πx). If,
  however, we use too large a value for λ then we again obtain a poor fit, as shown in
  Figure 1.7 for lnλ = 0

What is the reason for this? Why he doesn't just use λ?


texts: Looking for language and framework for data munging/wrangling

texts: I have an extensive series of projects that require importing data from many different types of different sources (websites, web apis, sensors, legacy text files, etc). Is there a good framework (preferably open source) that has been built bottom up to tackle this type of problem? Preferably one leveraging an appropriate language (python, most likely) and already having an extensive plugin database.


texts:
I have a dataset that includes a ratio(y-axis) and tenure(x-axis). My goal is to take the training set and establish a learning curve by tenure (1 to 15 years) which looks like this:

Actual Group Average:

[array([0.38056266, 0.45929261, 0.53092972, 0.53245087, 0.55000385, 0.53092972,  0.6007738 , 0.54705655, 0.6120135 , 0.72657781, 0.48381324,
       0.64294081, 0.6181171 , 0.67747932, 0.5790196 , 0.58390105])

Given new data, let's say someone tenured for 3 years, is 0.64, previous year 0.53. I would need to predict the area under the curve for 5 years, or in year 8. For someone that is 5 years tenured then predict for year 10, and so on.

After I have the areas for the actual group curve above and the predicted curve for individual, I would take the difference of the two to rate for potential.

My question is, given tenures 1 through 15, how would I find the auc in 5 year intervals for both the predicted and actual group?


texts: Generating Synthetic Image to improve the performance of classifier

texts:
I need some suggestion from experts. For my project work, I have been learning about Generative Adversarial Network.

I am trying to make a classifier (say CNN) to train with original CIFAR-10 dataset and then evaluate the performance.

After that, Generating more data of all classes using GAN and retrain the model and compare the performance.

But I don't know how feasible will the work be. How complex will be the work and also I want to know if the idea is acceptable as a project?


texts: Does it make sense to expand word embeddings so that each array index is a feature input or should the embedding itself be a model input?

texts:
If you are building a DNN, say, with two layers, and you want to use embeddings as one of your feature inputs, what's the best way to input the embedding?

I'm trying to understand if I should break the embeddings up so that every array value in the embedding becomes its own input feature to the model or whether the embedding should be kept in array form.

I've been following AirBnB's model for inspiration.

I'm trying to predict a binary classification in the final layer.


texts: Can somebody explain me the the following parameters of Keras LSTM layer

texts:
keras.layers.LSTM(units,stateful=False,unroll=False)

What units,stateful and unroll represents here??


texts:
I am reading "A Course in Machine Learning" and, in chapter 2, the author says:

"For most models, there will be associated parameters. These are
the things that we use the data to decide on. Parameters in a decision
tree include: the specific questions we asked, the order in which we
asked them, and the classification decisions at the leaves."

My question is about the first sentence. Is there any model in machine learning that does not have parameters? I can't think of any. For sure there are models without hyperparameters (for instance, the linear model does not contain any hyperparameter, but it still contains 2 parameters, the slope and the y-intercept).

If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?


texts:
This web link is to a site that talks about forecasting building electricity, like a time series regression concept.
In the article they talk about the NN architecture as:

the architecture of this neural network can be written as 120:7:24

Is an MLP type NN? What I also dont understand is if they account for times series methods to forecast/predict. For example I thought for time series applications a sliding window concept needs to be used Vs a typical regression problem that does not have any element of time. Any tips greatly appreciated!


texts: How to find out what portions of an image is helping CNN to classify it

texts:
I am working on an image classification problem using Transfer learning. Right now, I am getting an accuracy of 75% on train data and 67% in test data. Now I want to understand what portions/parts of the image are being utilized for classification graphically.
Are there any packages that can return a copy of one of the input images with markings on the most useful pixels or parts of the image?


texts:
Framework I used: pix2pix tensorflow

I am trying to train pix2pix to correct contrast on an image. However, I am getting blurred and grainy images. There is lot of black noise in the image(examples below). Is it because of the loss?

Note: I have made no change to pix2pix-tensorflow and used it as is.
The losses I see at the end of my training are:

discriminator loss: around 0.7
generator_loss_GAN: around 2.2
generator_loss_L1: around 0.16

Sample outputs: (Input-Output-Target)

Another example:

Loss information from tensorboard:

My questions are:

Am I first of all using correctly for my usecase?
How can I make sure there is no loss of data in the image at all and only the light amount changes in the target picture?
If GAN isn't right way, can you let me know how to achieve this?


texts: How to interpret weird loss/accuracy behavior in cross validation

texts:
I'm using transfer learning to classify binary medical MRI brain images. I used 5-fold cross validation, SGD optimizer and scheduler with different learning rate with 1e-4 and 1e-5 to get the following results. The learning curve and the accuracy have weird behavior and I don't know how to interpret and improve. Please help me indicate the problems and the things to try on.
The optimizer and scheduler settings I used:
params = [
            { 'params': parameters['base_parameters'], 'lr': sets.learning_rate },
            { 'params': parameters['new_parameters'], 'lr': sets.learning_rate }
            ]

optimizer = torch.optim.SGD(params=params, momentum=0.9, weight_decay=1e-4)
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)

Averaged learning curve and accuracy results with lr = 1e-4(average over 5 folds):

Averaged learning curve and accuracy results with lr = 1e-5:

So I have a few questions regarding the results.

Will the result be different if I change from SGD to Adams with the same learning rate?
In lr = 1e-4 situation, the model predicts single class mostly. As I searched in the forum, people said with lower learning might save the issue and indeed with  lr = 1e-5, the model gives more diverse results (still poor accuracy). However, the train/validation accuracy stops improving and yet fluctuating after certain epochs. What might be the reason for this?
The loss  curve stops decreasing around 2.5, but in normal situations, isn't the loss value lower than 1? Why is my loss value stops at 2.5?
The data augmentation method I used was randomly select the brain regions during training (i.e. some regions might be ignored). Will this cause the accuracy not improving as shown in the curve? Or should I use other data augmentation method (e.g. train on the whole brain with rotations)?

It might be complicated questions to answer, but i would be so appreciated if anyone could help me with this. Thank you!!


texts: How to identify clusters after multiple runs?

texts:
Suppose I run an unsupervised clustering algorithm. After multiple runs, I find clusters and would like to know if the same cluster was found more than once.

For example:

I can figure out A-orange, B-green and C-blue are probably the same because their centroids are close together.

However, take the following example:

How can I programmatically figure out that A-blue and C-orange are the same? That A-orange and C-blue are the same? But B-orange and B-blue are not similar to any in A and C?


texts: different results with MEKA vs Scikit-learn!

texts:
I have a multilabel data which I've trained using different classifiers with MEKA (multilabel version of WEKA) and the evaluation results (e.g. accuracy) that MEKA gives me are different from those I get from the same classifiers of scikit-learn.

I am using Binary Relevance method (which is both in MEKA and scikit-multilearn), because of my data being multilabel. And for instance, I am using SVM which is SMO in MEKA (and WEKA) and linearSVC in sklearn.

Now the accuracy I get from 10-fold cross-validation with BR>SMO in MEKA is about 50% where I get something around 20% accuracy from BinaryRelevanvce>LinearSVC. Other evaluation measures (e.g. f1-score, hamming loss etc.) don't match as well.

I tried to give all the parameters I could find in the input of LinearSVC, the same value as those of SMO but there's no luck. My Python code would be like this:

from skmultilearn.problem_transform import BinaryRelevance
from sklearn.svm import LinearSVC

my_classifier = BinaryRelevance(
                classifier=LinearSVC(C=1, random_state=1, tol=0.001, max_iter=10000, dual=False),
                require_dense=[True, True])

I must add that using BR>SMO in MEKA to train and evaluate the model would take almost an hour on my data, but sklearn's linearSVC (with skmultilern's BinaryRelevance) takes less than 10 minutes on the same data (hence the lower accuracy).

And by the way, I'm using accuracy_score method in sklearn.metrics for accuracy.

So my question is what am I missing? How can I get the same accuracy from SVM in MEKA and scikit-learn?


texts: How to read the predicted label of a Neural Netowork with Cross Entropy Loss? Pytorch

texts:
I am using a neural network to predict the quality of the Red Wine dataset, available on UCI machine Learning, using Pytorch, and Cross Entropy Loss as loss function.
This is my code:
input_size = len(input_columns)
hidden_size = 12
output_size = 6 #because there are 6 classes

#Loss function
loss_fn = F.cross_entropy

class WineQuality(nn.Module):
    def __init__(self):
        super().__init__()
        # input to hidden layer
        self.linear1 = nn.Linear(input_size, hidden_size)
        # hidden layer and output
        self.linear2 = nn.Linear(hidden_size, output_size)

    def forward(self, xb):
        out = self.linear1(xb)
        out = F.relu(out)
        out = self.linear2(out)
        return out

    def training_step(self, batch):
        inputs, targets = batch
        # Generate predictions
        out = self(inputs)
        # Calcuate loss
        loss = loss_fn(out,torch.argmax(targets, dim=1))
        return loss

    def validation_step(self, batch):
        inputs, targets = batch
        # Generate predictions
        out = self(inputs)
        # Calculate loss
        loss = loss_fn(out, torch.argmax(targets, dim=1))
        return {'val_loss': loss.detach()}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        return {'val_loss': epoch_loss.item()}

    def epoch_end(self, epoch, result, num_epochs):
        # Print result every 100th epoch
        if (epoch+1) % 100 == 0 or epoch == num_epochs-1:
            print("Epoch [{}], val_loss: {:.4f}".format(epoch+1, result['val_loss']))

model = WineQuality()

def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result, epochs)
        history.append(result)
    return history

loss_value = evaluate(model, valid_dl)

#model=WineQuality()
epochs = 1000
lr = 1e-5
history = fit(epochs, lr, model, train_loader, val_loader)

I can see that the model is good and that the loss decreases. The problem is when I have to do a prediction on an example:
def predict_single(input, target, model):
    inputs = input.unsqueeze(0)
    predictions = model(inputs)
    prediction = predictions[0].detach()
    print("Input:", input)
    print("Target:", target)
    print("Prediction:", prediction)
    return prediction

input, target = val_df[1]
prediction = predict_single(input, target, model)

This returns:
Input: tensor([0.8705, 0.3900, 2.1000, 0.0650, 4.1206, 3.3000, 0.5300, 0.2610])
Target: tensor([6.])
Prediction: tensor([ 3.6465,  0.2800, -0.4561, -1.6733, -0.6519, -0.1650])

I want to see to what are associated these logits, in the sense that I know that the highest logit is associated to the predicted class, but I want to see that class. I also applied softmax to rescale these values in a probability:
prediction = F.softmax(prediction)
print(prediction)
output = model(input.unsqueeze(0))
_,pred = output.max(1)
print(pred)

And the output is the following:
tensor([0.3296, 0.1361, 0.1339, 0.1324, 0.1335, 0.1346])
tensor([0])

I don't know what is that tensor([0]). I expect my predicted label, a value like 6.1 if the target is 6. But I am not able to obtain this.


texts:
I have some time-series data, which I need to use to predict a binary label for a given time-stamp. I was initially doing it using a Logistic Regression Model but I later figured that a time-series based problem could be better solved using Hidden Markov Models. I might also save on time since this prediction is to be made for ~200,000 time-stamps.

I have been able to obtain the next n hidden states using the hmmlearn implementation, but what I really need are 0/1 labels.

How can I (if possible) use HMM as a binary classifier?


texts: What is the proper train data format in LSTM?

texts:
I want to train a model to detect wrong word using in sentence.

I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is wrong or correct .
If a sentence length is  10,  I want LSTM learn 2 words surrounding current word ( so each word become 5 word) . So I need transform the sentence to [10, 5] .
But the first word(doesn't have left words) and  right word (doesn't have right words) , so I need pad them.
Then use word2vec transform this sentence to  [10, 5, embedding_size] , same as other sentences. (i.e each word transform to 5 word embeddings)
trainning (feed model by [5, embedding_size] * bacth_size )

I saw a lot LSTM input format and finally work out above flow.
Is my steps right ?

Actually I am still confusing with LSTM input format.

I have seen

Some pad all sentence to same length
Some split sentence into word blocks with same size (like I do above).
others ....

I am trying to use code to generate error samples, and mix with correct samples for lstm training.

For example:

corrects  = [
    ['How', 'are', 'you', '!'],          # [1, 1, 1, 1]
    ['Fine', ',', 'thank', 'you', '.'],   #  [1, 1, 1, 1, 1]
    ['Do', 'you', 'have', 'meal', '?'],   #  [1, 1, 1, 1, 1]
    ...
]

wrongs  = [
    ['How', 'were', 'you', '!'],            # [1, 0, 1, 1]
    ['Find', ',', 'thank', 'you', '.'],    # [0, 1, 1, 1, 1]
    ['Did', 'you', 'have', 'meal', '.'],    #  [0, 1, 1, 1, 0]
    ...
]

I come up with several thoughts

Because lstm can learn current word's network weight with pre words. So directly pass sentence to lstm model, don't transform from [sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).
Previous sentence's last word may be affacted by latter sentence's first few word, so It better chain all sentence . (How to BiLSTM in this situation ?)
Use feature like [sentence_len, step_len, embedding_size], doesn't need BiLSTM any more ? I think it still help here.

There are too many thoughts in my brain, hard to make it clear. So I post question here.


texts: How to prevent/tell if Decision Tree is overfitting?

texts:
In SKLearn's documentation on Decision Trees, they say we should pay special attention not to overfit the tree. How can we do this? I am aware that using random forests may prevent it, but how do I generally tell if it's overfitting? Can you tell by an accuracy score?

Is, for example, a 0.99 accuracy score an indicator of overfitting? Would in this case 0.95 mean not overfitting? What are some best practices in addition to a balanced input to avoid overfitting (especially with using SKLearn)?


texts: Question on ANOVA and Correlation/Association

texts:
I've been working on examining statistical relationships between variable:

Pearsons, Spearman's for continuous variables
Kendall's Tau, Cramer's V for ordinal/nominal variables.

I know there's many more ways. Recently I read about ANOVA and hypothesis testing. It seems similar to measuring correlation and association. In fact, I can't tell if it is just another way of doing the same thing, or if it is something entirely different. Most explanations of ANOVA seem a bit more complicated than most explanations of correlation or association.

For example, I know that Pearson's R is a measure of covariance scaled by standard deviation. And ANOVA stands for Analysis Of Variance. So it appears to me that it's the same sort of thing. But I can't tell 100% for sure.

Will someone please shed some light on this technique, what it is used for, and how it contrasts with measuring correlation?


texts: Batching data for LSTMs vs fully connected models

texts:
I've implemented an LSTM auto-encoder. It trained well, and does what I want it to so far. But, I think I've misunderstood something fundamental about lstms.
In a simple dense network whose input layer is size 7, I can feed a tensor of shape (batch_size, 7) where batch_size = len(data) / 7 and get the expected results. I successfully perform inference on the entire dataset at once.
I thought it would be the same with my LSTM autoencoder, but it only seems to work if I feed the data one entry at a time in a for-loop.
Dense reconstruction using of for-loop: OR a single "batch":

LSTM reconstruction using for-loop:

LSTM using a single "batch":

I know that LSTMs are sequential by nature, but I was under the impression that you can feed batches to the LSTM and some fancy optimization of for-loops occurs. Did I miss something conceptually?


texts: How to Train on plain text paragraphs and return keyphrases? is that even possible?

texts:
I am working on keyphrase extraction. Right now, I was able to create some features and run the candidate phrases along with the features for training a machine learning model for classification using random forest.

Now, out of curiosity, I would like to try out deep learning. I would like to remove the layer of feature extraction manually and then have it figure out the features by itself. Then generate a model by just passing some text documents and the relative key phrases (1/0 whether correct or incorrect) for each document. My question, does any training model accept plain text instead of floating point values? If not, how do I achieve the same thing by converting the sentences and keyphrases into floating point values and then pass through the trained model?

I tried creating a model using Keras Sequential model (sample given):

model = Sequential()
model.add(Dense(18, input_dim=14, init='uniform', activation='relu'))
model.add(Dense(14, init='uniform', activation='relu'))
model.add(Dense(1, init='uniform', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
model.fit(X, Y, epochs=300, batch_size=10)


texts:
I am trying to run GAIL in AI2THOR. I have found three implementation of GAIL available:

https://github.com/openai/imitation
https://github.com/openai/baselines/tree/master/baselines/gail
https://github.com/andrewliao11/gail-tf

I think all of them can be used for any Open AI gym environment. The AI2THOR environment I'm using has a wrapper to interface with gym：https://github.com/TheMTank/cups-rl.
What am I supposed to do to run one of the GAIL implementation in this AI2THOR? I'm totally new to kind of issue so any suggestions/guidelines would be appreciated. Thanks.


texts: How does CNN doing bounding box regression and what do features and weights represent for?

texts:
I knew that, in the house price logistical regression problem, the weights and features represent the "importance" of factor or coefficients of feature variables respectively, then minimize LSR loss can get the value of coefficients, and question are:

How does CNN doing bounding box regression?

I actually did a lot of googling to find an intuitive explanation, but no luck.

What do features and weights represent for in BBR?

I think it couldn't be $T$, $L$, $W$, and $H$ because these absolute values will vary a lot due to distance/scale and perspective difference,but the ratio of $\frac{W}{H}$ is a reasonable feature (for me to understand) due to it is a relative value.


texts: how to design sigma(std) vector to have values above > 0 in VAE autoencoder

texts: I am trying to understand how to design mu and sigma vector in VAE autoencoder, as we know std value can not be larger than 0. If I use linear function, the value can be lower than zero, so I think Relu is the only activation function to apply in order to get sigma (Std) vector larger than zero. Would this be correct understanding?


texts: Propensity model with Only Positive Data

texts:
Is it possible to build a propensity model (i.e., the likelihood that a user will buy an item) using only positive values.
For example, I have a bunch of data about Customers (people that bought stuff) and Users (people that haven't bought stuff yet)
I want to get the likelihood that a User becomes a Customer.
It seems that the only way to do so is to train a model using the data of Customers, therefore using only Positive values.


texts:
I'm reading "An Introduction to Statistical Learning: With Applications in R".
In the Paragraph 2.2.2 The Bias-Variance Trade-Off, the authors say:

I'm not able to understand why the bias tends to initially decrease faster than the variance increases. Can you help me ?
Both rigorous and intuitive explanations are greatly appreciated


texts: Changing duplicate a line hotkey in Jupyter notebook

texts: To duplicate a line in Jupyter-notebook on a Mac, the hotkey is cmd + shift + D. The problem is I use Google Chrome, and this is the same hotkey to bookmark a webpage, which takes priority over the hotkey in Jupyter-notebook. Has anyone struggled with this before, and found a workaround? I would probably prefer to use this hotkey for duplicating a line because it's the same hotkey I use in IDEs and text editors.


texts: Can `k=1` be a good choice for K neighbors classification?

texts:
Running sklearn.KNeighborsClassifier() on Kaggle's Leaf Classification sample (set of 99 species, 10 specimen each), with defaults kNN parameters and a grid search optimization (using sklearn.model_selection.GridSearchCV), I find the best accuracy is achieved for k=1.

Although this is a technically valid choice of k, I feel using a single neighbor means the model wasn't able to find a strong relationship between neighbors.

Am I right, or is k=1 sometimes a good choice, and for which kind of problems ?

Note that using neg_log_loss as scoring parameter leads to an optimal number of k=5 neighbors.  I am still struggling to understand how this is an improvement since the accuracy is smaller then.

Update : I was able to find more information using the "1NN" and "1-NN" keywords, which suggest this is indeed a good choice in some situations.


texts: Evaluating a model with different target class distributions between training and testing data

texts:
I'm having a bit of an argument about when class imbalances matter when training a classifier, so I was hoping to get some help on understanding a specific concept.
Say I have a problem where I want to take some inputs $X$ and predict some binary target class $Y$. My training/validation sets have an equal class distribution between positive and negative cases. I train a classifier, and then go to evaluate the model on a holdout test set.
In this test set, the target class distribution is now heavily skewed (think something like $95\%-5\%$). There is no change in relationship between inputs and outputs, meaning no covariate shift or concept drift. What kinds of differences should I expect to see in reported metrics, and is this an invalid way to evaluate a model's performance?
My initial thought was that I would probably expect to see changes in a metric like accuracy, but not necessarily in something like precision. If there are no per-class distributional changes in $X$, then my model's ability to identify the minority class should remain mostly the same. But the more I think about it, the less sure I am, and I would love if somebody could help me understand this better.


texts: Concrete Dropout for Recurrent Neural Networks (Keras)

texts:
I would like to use the Concrete Dropout Framework from GAL in application to recurrent neural networks. There is a great paper about it and the implementation can be found on the website (Thank you so much Yarin!):

https://github.com/yaringal/ConcreteDropout

He implements a Keras Layer called Concrete Dropout, which can be applied to arbitrairy layers. Nevertheless all his examples are feed forward networks, so here is my question:

Can I use it in a straight forward manner to regularize RNNs?

Example Code in Keras for this would be:

model.add(ConcreteDropout(GRU(32)))


texts: When optimizing the MSE, the correlation between prediction and target increases?

texts:
After optimizing the MSE (mean squared error) in a regression task, how is the change in Pearson correlation coeficient between target vector and the prediction?

Is any behaviour possible?
Or is sure that it becomes larger or lowers?


texts: Ordinal Integer variable vs Continuous Integer variable

texts:
I am working on titanic dataset. I have one feature Pclass which I understand is an ordinal variable having values 1,2 and 3. I have created a new feature cabin_int from feature Cabin, which is essentially the number of cabins alloted to a passenger. So, it has values like 0,1,2,3 and 4. Now this new feature is not ordinal, it is just a continuous variable taking only integer values.

My question is how does Machine Learning algorithm understands the difference between the two, if I pass these two columns (Pclass and cabin_int) as they are during training of my model?

If some more feature engineering needs to be done, please tell.


texts: Bug in single layer Adaline Neuron implementation

texts:
I am trying to implement a single layer Adaline neuron, with the following mathematical foundation:
The cost function is defined as:

The weight update is defined as:

inserting the partial derivative the final weight update becomes:

My Python implementation looks as follows:
class Adaline:
    n_iter = 0
    eta = 0

    weights_ = []

    def __init__(self, eta=0.01, n_iter=50):
        self.eta_ = eta
        self.n_iter = n_iter

    def fit(self, X, y, seed=123456):
        np.random.seed(seed)
        self.weights_ = np.random.normal(0, 0.1, len(X[0]) + 1)

        for iter in range(self.n_iter):
            weights_update = np.full(len(self.weights_), 0, dtype="float64")
            cost = 0

            for x, y_actual in zip(X, y):
                y_predicted = np.dot(self.weights_[1:], x) + self.weights_[0]
                update = (y_actual - y_predicted)

                cost += update ** 2

                weights_update += np.multiply(np.full(len(self.weights_), update),
                                              np.concatenate([[1.0], x]))

            cost = 0.5 * cost
            weights_update *= self.eta_
            self.weights_ = np.add(self.weights_, weights_update)

            print("iter (" + str(iter) + ")\n J =", cost, "\n wΔ =", weights_update, "\n w =", self.weights_)

        return self

    def predict(self, x):
        z = np.dot(self.weights_[1:], x) + self.weights_[0]
        return 1 if z >= 0 else -1

I train the neuron with widely known the iris dataset:
df = pd.read_csv("iris.data", header=None, encoding='utf-8')
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values

adaline = Adaline(0.01, 3).fit(X, y)

The following output shows something doesn't work. The cost ("J") keeps on increasing all the time, as do the weights, which of course is undesired:
iter (0)
 J = 89.6784555395353
 wΔ = [0.53973585 3.54224292 3.27735598]
 w = [0.58664708 3.51395659 3.12645013]
iter (1)
 J = 42892.97766680374
 wΔ = [ -28.75940385 -160.64752234  -90.03319066]
 w = [ -28.17275677 -157.13356575  -86.90674053]
iter (2)
 J = 66918619.409713484
 wΔ = [1136.57758637 6347.70919273 3552.29203057]
 w = [1108.40482961 6190.57562699 3465.38529004]

The bug seems to be in my fit function, but I can't find it.
Thanks for the help!


texts: Setting sparse=True in Scikit Learn OneHotEncoder does not reduce memory usage

texts:
I have a dataset that consists of 85 feature columns and 13195 rows. Approximately 50 of these features are categorical features which I encoded using OneHotEncoder. I was reading this article about sparse data sets and was intrigued to see how changing the value of the sparse parameter when defining a OneHotEncoder object may reduce memory usage for my dataset.
Before applying OneHotEncoding to categorical features in my dataset, I have a memory usage of 9.394 MB. I found this by running this code:
    BYTES_TO_MB_DIV = 0.000001
    def print_memory_usage_of_data_frame(df):
        mem = round(df.memory_usage().sum() * BYTES_TO_MB_DIV, 3)
        print("Memory usage is " + str(mem) + " MB")

    print_memory_usage_of_data_frame(dataset)

Setting OneHotEncoder spare=True:
    from sklearn.compose import ColumnTransformer
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline

    numeric_transformer = Pipeline(steps=[
        ('knnImputer', KNNImputer(n_neighbors=2, weights="uniform")),
        ('scaler', StandardScaler())])

    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=True))])

    preprocessor = ColumnTransformer(transformers=[
        ('num', numeric_transformer, selector(dtype_exclude="object")),
        ('cat', categorical_transformer, selector(dtype_include="object"))
    ])

    Z = pd.DataFrame(preprocessor.fit_transform(X))
    print_memory_usage_of_data_frame(Z)

Memory usage is 25.755 MB
Then running the same code above but setting spare=False like so:
    OneHotEncoder(handle_unknown='ignore', sparse=False)

Memory usage is 25.755 MB
According to the linked article, which used the sparse option in pandas get_dummies, this should result in reduced memory storage, is this not the same for Scikit Learn's OneHotEncoder?


texts: Threshold Value for Random Forest Classifier

texts: Is there any way to set a threshold value that can be set for voting results for Random Forest Classifier in sklearn package in order to prevent any misclassification for incoming test data.


texts: Multi-class Classification with Sigmoid Activation at Output Layer

texts:
In multi-class classification (mutually exclusive classes) using Neural Networks (ANN), it is generally advised that we encode our target labels as one-hot, use a softmax layer as the output with the number of nodes at the output layer being equal to the number of classes in the problem, and using categorical cross entropy as loss function. However, I was wondering, is it possible to use sigmoid activation functions with binary cross entropy as loss function, AND less number of nodes at the output layer instead? Here's an example:
Suppose the problem has 4 classes. We use only 2 nodes at the output layer, with sigmoid activations. The way we encode the target labels are then: (0,0) for class 1, (0,1) for class 2, (1,0) for class 3, and (1,1) for class 4 (like using binary codes). So if the input is of class 3, then we expect our network to output (1,0).
I have read posts similar to this, How does Sigmoid activation work in multi-class classification problems and https://glassboxmedicine.com/2019/05/26/classification-sigmoid-vs-softmax/ but in these cases, they are saying why we shouldn't use sigmoid with number of nodes in the output layer being equal to the number of classes, whereas what I'm proposing here is to use less. Note that the above encoding also works if we have say 3 classes (we thus only use 3 labels, e.g. (0,0), (1,0) and (1,1)).
Would the above implementation work as good or even better than softmax? And any reasons why it should or shouldn't work better?


texts: cost sensitive loss function in lightbm with individual cost

texts:
i am looking for a cost sensitive function that will have weights according to individual row feature (like amount)
this way i can penalize more FN which has large amount vs. low dollar amount.
took the raw solution from the excellent article : https://maxhalford.github.io/blog/lightgbm-focal-loss/
and incorporated it into the objective function (correctly ?)  , but i wonder how should i do the same for the evaluation function.
here is the code:
class FocalLoss_with_amt:

def __init__(self, gamma, alpha=None):
    self.alpha = alpha
    self.gamma = gamma

def at(self, y):
    if self.alpha is None:
        return np.ones_like(y)
    return np.where(y, self.alpha, 1 - self.alpha)

def pt(self, y, p):
    p = np.clip(p, 1e-15, 1 - 1e-15)
    return np.where(y, p, 1 - p)

def __call__(self, y_true, y_pred):
    at = self.at(y_true)
    pt = self.pt(y_true, y_pred)
    return -at * (1 - pt) ** self.gamma * np.log(pt)

def grad(self, y_true, y_pred):
    y =  2 * y_true - 1  # {0, 1} -> {-1, 1}
    at = self.at(y_true)
    pt = self.pt(y_true, y_pred)
    g = self.gamma
    return at * y * (1 - pt) ** g * (g * pt * np.log(pt) + pt - 1)

def hess(self, y_true, y_pred):
    y =  2 * y_true - 1  # {0, 1} -> {-1, 1}
    at = self.at(y_true)
    pt = self.pt(y_true, y_pred)
    g = self.gamma

    u = at * y * (1 - pt) ** g
    du = -at * y * g * (1 - pt) ** (g - 1)
    v = g * pt * np.log(pt) + pt - 1
    dv = g * np.log(pt) + g + 1

    return (du * v + u * dv) * y * (pt * (1 - pt))

def init_score(self, y_true):
    res = optimize.minimize_scalar(
        lambda p: self(y_true, p).sum(),
        bounds=(0, 1),
        method='bounded'
    )
    p = res.x
    log_odds = np.log(p / (1 - p))
    return log_odds

def lgb_obj(self, preds, train_data):
    amt = train_data.get_weight()
    y = train_data.get_label()
    p = special.expit(preds)
    return amt*self.grad(y, p), amt*self.hess(y, p)

def lgb_eval(self, preds, train_data):
    amt = train_data.get_weight()
    y = train_data.get_label()
    p = special.expit(preds)
    is_higher_better = False
    return 'focal_loss', self(y, p).mean(), is_higher_better

fl_amt = FocalLoss_with_amt(alpha=0.27242, gamma=2.478478446)

d_train.set_weight(weight=np.log(X_train['amount']))
d_val.set_weight(weight=np.log(X_val['amount']))

model_amt=lgb.train(best, d_train, 10000, valid_sets=[d_val], fobj=fl_amt.lgb_obj,
                   feval=fl_amt.lgb_eval,early_stopping_rounds=40, verbose_eval=500)


texts: How can we use Neural Networks for Decision Making intead of Bayesian networks or Desicion Trees?

texts: I am working on Decision Making in Self driving cars and I am wondering how I can use Neural networks (is there any type) ? that can repleace or mimic the bayesian networks or Decision Tree for Decision Making process ?


texts: Training anomaly detection on text string

texts:
I have text string stored in a column, resulting after pre-processing of web traffic data. Now I want to apply anomaly detection on it and for that purpose I applied countvectorizer and tf-idf_vectorizer

Data:

0,   Mozilla Windows U Windows NT enUS AppleWebKit...
1,   Mozilla Windows NT rv Gecko Firefox
2,   Mozilla iPod U CPU iPhone OS like Mac OS X en...
3,   Mozilla Windows NT rv Gecko Firefox
4,   Mozilla compatible MSIE Windows NT NET CLR ...

Now tf-idf vectorized data looks as below, which is a 8000x1500 data:

Suggestion on how can I apply anomaly detection models on it.

Any Idea on how to apply following techniques will be highly appreciated.


texts: (Newbie) Decision Tree Classifier Splitting precedure

texts:
I have a dataset with 4 categorical features (Cholesterol, Systolic Blood pressure, diastolic blood pressure, and smoking rate). I use a decision tree classifier to find the probability of stroke. I am trying to verify my understanding of the splitting procedure done by Python Sklearn. Since it is a binary tree, there are three possible ways to split the first feature which is either to group categories {0 and 1 to a leaf, 2 to another leaf} or {0 and 2, 1}, or {0, 1 and 2}. What I know (please correct me here) is that the chosen split is the one with the highest information gain.

I have calculated the information gain for each of the three grouping scenarios:

{0 + 1 , 2} --> 0.17

{0 + 2 , 1} --> 0.18

{1 + 2 , 0} --> 0.004

However, sklearn's decision tree chose the first scenario instead of the third (please check the picture).

Can anyone please help clarify the reason for selecting the first scenario? is there a priority for splits that results in pure nodes. thus selecting such a scenario although it has less information gain?

I have added the frequency of each class/feature/category so it would be easy to calculate the Gini index


texts: How to implement gridsearchCV for onevsrestclassifier of LogisticRegression classifier?

texts:
parameters = [{'C': [10**-2, 10**-1, 10**0,10**1, 10**2, 10**3]}]

model_tunning = GridSearchCV(OneVsRestClassifier(LogisticRegression(penalty='l1')), param_grid=parameters,scoring="f1")
model_tunning.fit(x_train_multilabel, y_train)

ValueError                                Traceback (most recent call last)
<ipython-input-38-5d5850fe8978> in <module>()
  2
  3 model_tunning = GridSearchCV(OneVsRestClassifier(LogisticRegression(penalty='l1')), param_grid=parameters,scoring="f1")
----> 4 model_tunning.fit(x_train_multilabel, y_train)

ValueError: Invalid parameter C for estimator OneVsRestClassifier(estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
      intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
      penalty='l1', random_state=None, solver='liblinear', tol=0.0001,
      verbose=0, warm_start=False),
      n_jobs=1). Check the list of available parameters with `estimator.get_params().keys()


texts: How to measure the correlation of different algorithms

texts: In stacked generalization, several algorithms are trained on the training set (i.e. at layer 1) and their predictions are then stacked using a layer 2 model. In many documentations, it is said that it is better that the layer 1 algorithms should be of low correlation. How can one compute this correlation between algorithms ?


texts: Can GAN generate very happy facial expression

texts:
I have some dataset ${(x1,y1), (x2,y2)...(xn,yn)}$, where, $x$ is the picture of a facial expression，while $y$ is the fraction corresponding to their degree of the happiness (happy laugh: $y$ close to $100$; sad cry : $y$ close to $0$). The range of the $y$ is $0$~$100$.

In my dataset, I didn’t have the sample with $y$ value greater $80$，but I want to get a very happy facial expression. In other words，I want to get a facial expression with very high y value $(y>95)$.

Can GAN (Generative Adversarial Networks) implement this? If not，is there any model that can implement this goal?


texts: Why is the difference of datetime = zero in the following dataframe?


texts: Difference between indicator column and categorical identity column in tensorflow

texts:
I am learning Tensorflow and came across different feature columns used in Tensorflow . Out of these types, two are categorical_identity_column and indicator_column. Both have been defined in the same way. As far as I understand, both convert categorical column to one-hot encoded column.

So my question is what is the difference between the two? When to use one and when to use the other?


texts: Does using your test set ultimately burn your data set in case of failure?

texts:
Given a data set I want to train a machine learning algorithm on. The data is split into training, validation, and test data.
I now successfully trained my algorithm to work well with the training data and validating using the validation set is also promising. However, when applying the test data the model underperformes.
I am now stuck with two options:

Throw everything away and start new with the same data set. This however has been likened to p-hacking.
Throw the data set away as it is now burned. This could kill my project or be really expensive as I need to recollect data, this might even be impossible.

Is my data set ultimately burned when applying the trained model unsuccessfully on my test set?
Related Bonus: Is there some form of Bonferroni Correction I could apply to keep reusing the data set in case I would have burned the data set?


texts:
I am familiar with the LSTM unit (memory cell, forget gate, output gate etc) however I am struggling to see how this links to the LSTM implementation in Keras.

In Keras the input data structure for X is: (nb_samples, timesteps, input_dim).

Suppose that the shape of X is: (1000, timesteps = 10, 40).

1) Does this mean that the LSTM cells will only consider ‘batches’ of 10 previous time steps ?

2) Or is the output from LSTM cells passed between these sets of 10 timesteps I.e could you capture long term dependencies 50 timesteps out?


texts: Unsupervised Anomaly Detection on system metrics like memory, cpu, io, net, etc

texts:
In all the examples that I can see online, people have used a labelled dataset. I however am stuck trying to construct a model to perform anomaly detection on unlabelled dataset (unsupervised anomaly detection).

  Problem Statement: To separate the anomalous points from the normal ones. The features available with us are mem_usage, mem_total, cpu_usage_idle, cpu_usage_iowait, etc

Using the anomaly detection, I plan to label the dataset saying whether anomalous or not. A step further would be aggregate each row based on the server the data belongs to and find if the server is healthy, problematic. Even further would be predicting when a server might face a similar problem and thus suggest ways to mitigate the problem.

Any kind of resources (papers, code blogs, videos) are really welcome. I am just starting out and am open to trying out new stuff as well.