texts:
I know this may sound nooby, but how do I save a Deep Q-Learning agent's progress? I mean when I close at i.e. episode 500 when my agent is trained and I restart (in my case a pygame) my agent is going to load its data and it will continue with its progress ( does not start again very stupid :) ). How can it be done?

If you need more details/clarification, feel free to ask! :)

Thanks for any answer! I really do appreciate it!


texts: Find threshold in rate to determine reason for lost customer

texts:
I'm not sure if anybody will be able to help with this, or even if I'll be able to explain it well but I am stuck so here goes....

I have a set of customers, some are lost, some are still active. We have a model which looks through the data and assigns reasons for why they became lost. One of the reasons to determine is if them not using our software caused them to become lost - we have software to automate their orders but they can ignore this and process orders manually.

I have created 2 variables; software usage rate and software usage vs their country average (as a ratio).

My aim is to determine the point, in either or both of these, which is basically the threshold/marker to say the software was an issue. I am stuck as there are in essence no results - we know if a customer is lost, but we can only speculate/model on the reasons.

All I've been doing so far is looking at the distributions of lost and active manually.

Any help would be great.

Thanks


texts: GANs (generative adversarial networks) possible for text as well?

texts:
Are GANs (generative adversarial networks) good just for images or can they be used for text as well?
Like training a network to generate meaningful text from a summary.
UPD - quotes from the GAN inventor Ian Goodfellow.

GANs have not been applied to NLP because GANs are only defined for
real-valued data. (2016) source
It is not a fundamentally flawed idea. It should be possible to do at least one of the following... (2017) source


texts: Why are the ANN training and validation accuracy graphs not smooth?

texts:
I am currently training an ANN using Keras (Python3), and I am slowly optimizing the model's architecture and came across something I have not seen before.
The graph of the training and validation accuracy seems a bit odd. The graph appears 'step-like' in the sense that it is not a smooth curve, but different.
Model:
model = tf.keras.models.Sequential()

model.add(Dense(7,activation='relu'))
model.add(Dense(10,activation='relu'))
model.add(Dense(10,activation='relu'))
model.add(Dense(1,activation='sigmoid'))

model.compile(optimizer=tf.keras.optimizers.Adam(0.0001),
              loss='binary_crossentropy',
             metrics=['accuracy', 'mse', 'mae'])

history = model.fit(
    X_train.values,
    y_train,
    epochs=1000,
    validation_split = 0.2,
    verbose=1,
    )

My two questions are:
(1) what is this indicative of, and
(2) how do i fix this assuming it is problematic?


texts: kNN - what happens if more than K observation have the same distance to the centroid of the cluster

texts:
EDIT It was pointed out in the Answers-section that I am confusing k-means and kNN. Indeed I was thinking about kNN but wrote k-means since I'm still new to this topic and confuse the terms quite often. So here is the changed question.

I was looking at kNN today and something struck me as odd or - to be more precise - something that I was unable to find information about namely the following situation.

Imagine that we pick kNN for some dataset. I want to remain as general as possible, thus $k$ will not be specified here. Further we select, at some point, an observation where the number of neighbors that fulfill the requirement to be in the neighbourhood are actually more than the specified $k$.

What criterion/criteria should be applied here if we are restricted to use the specific K and thus cannot alter the structure of the neighborhood (number of neighbors). Which observations will be left out and why? Also is this a problem that occurs often, or is it something of an anomaly?


texts:
I am working on predicting half-hourly UK electricity prices with prophet. I have two other time series: gas prices and initial national demand out-turn. So, after merging all the data-sets together based on datetime index, I used gas prices and initial national demand as regressors.

When I use only gas as regressor, I get better prediction results. However, If I add load demand, the model gives worse results. I am a bit confused cuz both regressors are positively correlated and have impact on electricity prices.

Also, one important detail is that the load demand has very clear pattern compared to electricity and gas prices series. So, from the prediction graph, it seems like prophet model is following that pattern that is easier to follow, which in this case is load demand. Anyhow, I might be wrong somewhere.

Some of the code:

#Prophet:
model_P = Prophet(weekly_seasonality=True,yearly_seasonality=False, daily_seasonality=True)
model_P.add_seasonality(name="monthly", period=30.5, fourier_order=5)
model_P.add_regressor('Gas')
model_P.add_regressor('INDO')
model_P.fit(pd.DataFrame({'ds': train.index, 'y':train, 'Gas': train_gas, 'INDO': train_indo}))

def populate_indo_val(dt):
current_date=str(dt.date())
return df[current_date]['INDO'][0]

def populate_gas_val(dt):
current_date=str(dt.date())
return df[current_date]['Gas'][0]

future_dates = model_P.make_future_dataframe(periods=test.shape[0],freq='30T')
future_dates['INDO']=future_dates['ds'].apply(populate_indo_val)
future_dates['Gas']=future_dates['ds'].apply(populate_gas_val)
results_P = model_P.predict(future_dates)
forecast_P = results_P.set_index('ds').yhat

I am also attaching the forecasting model graphs. When only gas is used as regressor and the second graph that depicts the model where only initial load demand is used for prediction.


texts:
I was going through a paper comparing glove and word2vec. I came across the pound notation shown below. What does it mean when used like this?

The link for paper is here


texts: How to extract all the information from a midi file (monophonic and polyphonic) and then vectorize them to feed into a Neural Network?

texts:
I have tried Music21,Midicsv and all the packages regarding midi in python but can't yield a result yet.In midicsv i can't figure out that how should approach to vectorize the csv file.

Does anyone have any idea. I will be grateful for any kind of help.

To know more about this issue please check the following links:

Issue on music21

Issue on Midicsv

Please help me to solve this problem.


texts:
When training my CNN model, based on the random initialization of weights, i get the prediction results. In other words, with the same training and test data i get different results every time when i run the code. When tracking the loss, i can know if the result would be acceptable or not. Based on this, i want to know if there is a way to stop the training if the loss begins by a value superior to a desired one in order to re-run it. The min_delta of the EarlyStopping does not treat this case.
Thanks in advance


texts: How to find the various matrix sizes in designing a CNN

texts:
I am trying to understand CNN especially the maths and working mechanism using Matlab as the coding language. I have few confusion regarding the concept and the associated programming and will be immensely grateful for an intuitive answer.

Below is the structure of my CNN for 5 classes. I could calculate only the output structure of the first Conv layer and stuck on determining the number of parameters i.e., number of neurons?

The output for the first convolution layer that I could calculate: In the first layer an input of size [50 50 2] is convolved
with a set of M_1 5-dimensional filters applied over all the input channels. The first 2 D convolutional layer is composed of M_1 = 20 filters of size [5x5x 1] having the step size (stride) for traversing the input vertically and horizontally as 1 creating a feature map of size {(h-f_h+1) x (w - f_w +1)x 1x M_1} = (50-5+1)x(50-5+1)x20 = [46x46x 20] So we have 20 channels.


texts: Training LSTM Recurrent Network in TensorFlow

texts:
I have trained RNN's before "by hand" using basic tools like Numpy or BLAS, but I am having trouble getting a simple RNN to converge in TF. Full Code

I tried standard things like adjusting the learning rate, the momentum, and adding noise to the gradient, but I am concerned that I don't understand what TF is doing underneath.

In a couple tutorials, it appeared that in TF you train the networks like Echo State Networks where the hidden state is evolved on random interconnections to train up a reservoir, then you use a linear (or non-linear) transformation to map the hidden state onto your labels.

I examined the gradients that Tensorflow uses to train, and it appears that it is training the internal state transition matrices as one would expect if using "Back-Propagation Through Time", which I can't tell if it is trying to use.

Can you help me understand what I am doing/understanding incorrectly with regards to training RNN's in TF?

cell_layer = tf.contrib.rnn.BasicLSTMCell(state_size, state_is_tuple=True)
cell = tf.contrib.rnn.MultiRNNCell([cell_layer]*1)
x, y_ = tf.placeholder(tf.float32, shape=(batch_size,1)), tf.placeholder(tf.float32, shape=(batch_size,1))
outputs, states = tf.nn.dynamic_rnn(cell, tf.expand_dims(x, -1), dtype=tf.float32)# init_state)

W = tf.Variable(tf.random_uniform((state_size, 1), -1.0, 1.0))
b = tf.Variable(tf.random_uniform((1, 1), -1.0, 1.0))
y = tf.matmul(tf.reshape(outputs, (-1, state_size)), W) + b

loss = tf.reduce_mean(tf.square(y - y_))
opt = tf.train.MomentumOptimizer(1e-3, 0.9)
grad = [(g + tf.random_uniform(v.shape, -1.0, 1.0) * tf.reduce_mean(tf.abs(v))*noise, v)
        for g, v in opt.compute_gradients(loss)]
train = opt.apply_gradients(grad)

EDIT: Loss plot

I am less concerned about the loss than I am about the fact that there appear to not be any dynamics in between iterations. It appears to converge to the average. Below are some of the later losses and a part of the sequence to be predicted.

Loss:  11.9126
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9092
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.918
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Loss:  11.9274
Actual:  bcdefghijklabcdefghijklabcdefghijklabcde
Pred:    ffffffffffffffffffffffffffffffffffffffff

Note

>>> s = 'bcdefghijklabcdefghijklabcdefghijklabcde'
>>> n = map(ord, s)
>>> chr(sum(n)/len(n))
'f'


texts: Jupyter notebook kernel dies when reading CSV file on M1 Mac

texts:
Odd Jupyter notebook problem on M1 mac here. Using these steps I was able to get Jupyter Notebooks to run in the first place. It worked fine executing the following simple code:
print('test')
import pandas as pd from pandas import read_csv

It also allowed me to create a basic list for example:
test = [1,2,3]

However, when I try to upload a small dataset (1.7mb) on the other hand using:
data = read_csv("dar_ward_comparison.csv", index_col=0)

I get a "The kernel appears to have died. It will restart automatically." within about 10 seconds.
Does anyone have any suggestions on what might be the root cause?


texts: Scoring function for transformers (BERT etc)

texts:
While using BERT / transformers for NLP tasks, a major problem faced by us was to detect if the answer returned by model is correct or not, or what is the confidence level of the answer. The confidence returned by BERT was average of start token and end token confidence, was very high but still the answer was wrong.
Does anyone know how to evaluate or reject the answer returned by BERT or transformer based models. Any method or scoring function or technique to safely rely on?


texts: How many augmentated data points for each training image?

texts:
What are some useful rules of thumb for picking the number of augmenters per training image?

I realize this is a hyperparameter I can vary and test: I'm just trying to get a sense for reasonable ranges/choices based on people's experience with different network architecture and other particulars.

My particulars are that I have a relatively simple object detection task, with about 1100 annotated images, a single category. I am doing the transfer learning on a faster-rcnn with resnet-101 backbone.

I do an 80/20 train/test split, and even when I augment just 5x for each training image [h/v flips, small rotations, brightness changes, gaussian blur, motion blur, scale], the network is learning fairly well given all my other parameters.

For my final step I am planning to just add more augmented images per training image. I had always though 100 was a good number but I now realizes this is sort of arbitrary, and I picked it up just from some random example I used once from the internet (!). Even with 10 augmenters per image, I would have over 8k training images for the category, which should be enough, no?

Note a few people have asked about how many images there should be per class for training, but they don't take augmentation into account.

Related questions

https://stackoverflow.com/questions/55356982/how-many-imagesminimum-should-be-there-in-each-classes-for-training-yolo
How many images per class are sufficient for training a CNN


texts: Is there a way to Label/Annotate My Bubble Plot (Scatter plot with a z-axis) on matplotlib?

texts:
I would like to label my bubble plot, with the labels being INSIDE the bubbles. Something like this.

Is There any way to do that?

What I've tried:

plt.annotate(temp.index, (temp.gre_total, temp.ugrad_gpa))
plt.show()

But that throws an error. I think that would only work if I looped through every value? I'm not very sure.


texts: How should I treat these non-English documents in the NLP task?

texts:
So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a machine learning model.

Now the question is: How should these non-English documents be treated?

Should I exclude them from the final corpus and from training the model?
or should I manually translate them (Requesting natives from each language to translate it for me) and include them in the final corpus?
or should I use Google translate/DeepL to translate these non-English documents into English and then include them in the final corpus?

Each document in the corpus under question is not larger than 500 words each.

Any help or hint will be appreciated.


texts: What does it mean for the training data to be generated by a probability distribution over datasets

texts:
I was reading the Deep Learning book and came across the following para (page 109, second para):

  The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{\text{data}}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

Can somebody please explain to me the meaning of this paragraph?

On page 122 the last paragraph, it also gives an example

  a set of samples $\{x(1), \dots, x(m) \}$ that are independently and identically distributed according to a Bernoulli distribution with mean $\theta$.

What does this mean?


texts:
I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model.
I do the following:

from transformers import BertModel
hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

where

token_ids with Shape : [1, 4, 47]
attn_mask with Shape : [1, 4, 47]
seg_ids with Shape : [1, 4, 47]

but I have an error :

TypeError                                 Traceback (most recent call last)
<ipython-input-74-5fa632122cc7> in <module>()
      1 from transformers import BertModel
----> 2 hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

TypeError: __init__() got an unexpected keyword argument 'attention_mask'

How can I fix this error??

and how can I find the word embedding and the similarity of a given word using this model?

I could not find any tutorials or similar codes for this task.


texts: Classification Model showing different accuracy for SAME data?

texts:
This is my first post here, so kindly pardon any commonplace errors.

So, i have been training an XGBoost multi-class model on Google Colab. I am using a balanced dataset, with 31000 rows, where each class has 1000 rows.

My routine procedure is to train a model, get some metrics like Accuracy, ROC score and Pickle the model.

The problem is, when i load the pickled model in another Google Colab model with the same dataset, containing the exact same test, train splits as before, i get wildly different values of accuracy and ROC

I am ensuring the exact same data splits by using the random_state variable. Also in the XGBoost classifier, i am using fixed values of random_state and seed, to ensure there is no randomness in the results.

I will attempt to exemplify my problem.

For instance, i trained a model in notebook 1. I got accuracy value of 83.0806 % and ROC score of 91.3732 %

Happy with the results, i pickle the trained model for future use.

I tried to test the reproducibility of my work by opening a new Google Colab notebook (notebook 2 )  and loading the exact same dataset, with all the pre-processing steps and data splits same as before.

I then load the pickled model, use the predict function;

loaded_model.predict(x_test)

However, this time, i get very different values of accuracy & ROC score. I get an accuracy of 95.8870 % and  a ROC score of 97.8383 %

Please note that this is the exact same dataset as used in notebook 1

can someone please tell me why this is happening ? is this an issue in Google Colab ?

Kindly help me out. I will be grateful for any guidance someone can provide in this regard


texts: Using neural networks to recognize digits in a scene

texts:
Neural nets are widely used (as example for the MNIST dataset). Using neural networks and convolutional neural networks, in particular, we can get over 99% accuracy. However, the MNIST dataset is a dataset where each image only has one digit cropped and centered so that it is always in the same position.

If I were to create a neural network to recognize digits where the digits may not be in the center (and possibly there may be other objects or digits in the scene), how do I go about developing it?

I need to detect the digits and their position.


texts:
I have a large data set with over 100k samples and I want to  predict a continuous target feature from 4 other continuous features using Scikit Learn.
For this project, I would like to visualize and analyze the data using both 1 dimensional and two dimensional histograms. I know how to plot histograms and I know what a histogram means/displays mathematically but how can I make good use of it in order to analyze my data?

One thing that comes to mind is that I could spot regions with outliers, but this doesn't seem so useful/efficient (correct me if I'm wrong).

So what are useful ways to use histograms for analyzing Machine Learning data?

Thanks


texts: Is it possible to confine regression output to some conditions in an ANN?

texts:
I have a NN that accepts amount spent as input, and predicts the number of clothing items bought as a multi-target regression problem (see: How to count categorization instances in a NN?).

Each clothing item however has a fixed value, and it would make no sense for the NN to predict a sequence of purchased items which has a different value to the input budget. Is there any way I can convey this information to the NN to either force it to confine its output to match the dollar amount of the input, or to assist it in its training?


texts: How to give names/labels to topics in LDA

texts:
I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused.

How can I use the information produced by LDA to automatically generate names to the topics generated?


texts:
While reading the paper "Backpropagation Applied to Handwritten Zip Code Recognition", one
of the original papers attributed to the development of convolutional neural networks,
on page 3, section 3.1 the authors write (emphasis mine):
The remainder of the recognition is entirely performed by a multilayer network. All of the connections in the network are adaptive, although heavily constrained, and are trained using backpropagation. This is in contrast with earlier work (Denker et al. 1989) where the first few layers of connections were hand-chosen constants implemented on a neural-network chip. The input of the network is a 16 by 16 normalized image. The output is composed of 10 units (one per class) and uses place coding.
What does place coding mean? A Google search of this term yielded nothing conclusive. Any insights appreciated.


texts: How to predict multiple time points in future?

texts:
Let's say I have a database of customer's purchase history. So, my data has below info
a) customer demographics such as age, gender, country etc.
b) customer order history such as order_id, order_date, order_amount etc.
Now, I would like to predict what and when will be his next 3 purchases?
For ex: customer X has high likelihood to purchase a wallet within 1 month, trimmer within 3 months and mobile phone within 8 months.
Am not sure whether it is possible to implement this in ML. Can you guys guide me on how to formulate this problem? and how to prepare my dataset to be suitable for a ML model?
I think what I am trying to do is multi step foreasting but not just using a time series but with a data (continuous, categorical) which has chronology of events occurred (based on his purchase history we can know the order of events).
Can experts here provide me the list of steps and techniques to follow? So, I can learn accordingly and be aware of the tasks involved in this pipeline?


texts: Graphs for representing performance of machine learning classifiers

texts: Can any please guide about how many types of the graph other than ROC can be plotted to represent the performance of the machine learning classifier?


texts:
I've written a simple neural network that can predict XOR gate function. I think I've used the math correctly, but the loss doesn't go down and remains near 0.6. Can anyone help me find the reason why?

import numpy as np
import matplotlib as plt

train_X = np.array([[0,0],[0,1],[1,0],[1,1]]).T
train_Y = np.array([[0,1,1,0]])
test_X = np.array([[0,0],[0,1],[1,0],[1,1]]).T
test_Y = np.array([[0,1,1,0]])

learning_rate = 0.1
S = 5

def sigmoid(z):
    return 1/(1+np.exp(-z))

def sigmoid_derivative(z):
    return sigmoid(z)*(1-sigmoid(z))

S0, S1, S2 = 2, 5, 1
m = 4

w1 = np.random.randn(S1, S0) * 0.01
b1 = np.zeros((S1, 1))
w2 = np.random.randn(S2, S1) * 0.01
b2 = np.zeros((S2, 1))

for i in range(1000000):
    Z1 = np.dot(w1, train_X) + b1
    A1 = sigmoid(Z1)
    Z2 = np.dot(w2, A1) + b2
    A2 = sigmoid(Z2)

    J = np.sum(-train_Y * np.log(A2) + (train_Y-1) * np.log(1-A2)) / m

    dZ2 = A2 - train_Y
    dW2 = np.dot(dZ2, A1.T) / m
    dB2 = np.sum(dZ2, axis = 1, keepdims = True) / m
    dZ1 = np.dot(w2.T, dZ2) * sigmoid_derivative(Z1)
    dW1 = np.dot(dZ1, train_X.T) / m
    dB1 = np.sum(dZ1, axis = 1, keepdims = True) / m

    w1 = w1 - dW1 * 0.03
    w2 = w2 - dW2 * 0.03
    b1 = b1 - dB1 * 0.03
    b2 = b2 - dB2 * 0.03

    print(J)


texts:
I have been trying to understand the backpropagation for a while now.
I have came across two variants of it.

In the Andrew Ng class the derivatives of the weights of hidden layers are calculated using the error signal that is distributed back to the hidden node.
In Geoffrey Hinton class the derivatives of the weights of hidden layers are calculated using the derivatives of the next layer that are already computed and from my knowledge of calculus that makes more sense.

Can someone explain how the first variant works?


texts: Feature Extraction and Vector Space Model

texts:
I have a dataset of reviews and I want to extract the features along with their opinion words in the reviews.

Is it possible to extract features from my data using any Vector Space Model?(TF-IDF, Word2Vec, etc.)

Edit for sample data:

"Teacher", "Subject", "Feedback"
"Dr. Tayal","Discrete Mathematics","He has very good subject knowledge. He didn't take all the lectures. He teaches and explains concepts very well."
"Ms Vibha","OOPS","Subject knowledge is poor, but she is regular with classes. She is unnecassrily strict and does not teach the subject nicely."


texts: Understanding XG Boost Training (Multi class classification)

texts:
I have been working with XG boost for classification (multi class classification : 6 classes)
I use 5 fold CV to train and validate my model.

Please refer to the paramters, which i had used in my model.

 params    = {"objective": 'multi:softprob', "eta": 0.1, "max_depth": 7,
               "min_child_weight": 4,"silent": 1, "subsample": 0.8,
               "colsample_bytree": 0.8, "num_class" : 6, "gamma" : 0,
               "eval_metric" : 'merror', "seed": 0}

I plotted the training and testing error for each fold in a 5 fold CV.

Questions:

What can I understand/interpret from the training & test loss graph ?
Training error reduces to zero, but the testing error reduces over a period & become idle.

I am not sure whether the model is overfitting ?
How can I reduces the error b/w training & testing, only through hyperparam tunning or
this is how the XG Boost model works ?


texts: Bug in sentiment analysis and classification for unlabeled text

texts:
I'm working on the transcript of Trump and Biden's debate and want to analyze the sentences and classify negative, positive, or neutral comments, but I ran into one problem. I used both TextBlob and the transformers pipeline to analyze the sentiment but unfortunately in both ways, there are some very disastrous flaws!
For example,I found that TextBlob recognizes -0.70 polarity in "fewer people are dying every day" (negative comment)
or the transformers pipeline recognizes "The audience here in the hall has promised to remain silent." as a negative comment with 0.99 percent certainty!
Why do you think it's happening?
Is there any way we can prevent this?
Is there any way better than this for analyzing the sentiment of unlabeled text?
Also, I'm not comfortable with sentences like "Oh, Really?!" being classified as neutral. It's more of a sarcastic or negative comment I think.
Here's my colab notebook, I've added one "Problem" markdown where I've observed these examples.


texts: Which value of output should be taken in multiple sessions of training Neural Network

texts: Suppose I am using Neural Network for a 2 class classification. After training the network with the training set, I want to predict the class label of a dataset with no class label. Now with retraining, the same dataset gives different result. For example in one training session, a sample gave the output of it belonging to class 1 while in the other session it gave the output of it belonging to class 2. Then which value should be taken as the correct one?


texts: Should we keep all channels when doing image classification?

texts:
I am discovering the world of image recognition and now trying to build an image classifier. The set of images I have have the shape (101,101,3) which means that it has 3 channels.

If I'm not mistaken the channels corresponds to the Red, Green and Blue channels (Please correct me if I am wrong).

In which classification tasks should we keep the three channels ? I've seen people converting the Image to a grayscale so that the images shape will be (101,101,1). What difference does it make when you do so ?

I am aware that when you keep the 3 channels the computation needs grow with it, here I am more looking for an answer about the intuition behind keeping the 3 channels, move to grayscale or add new custom channels.


texts:
I hope this question is okay for the forum. I want to ask for your experiance with Python editors.

Currently, I use VS-Code to work with Python. However, in R Studio I really appreciate that it holds data frames in the memory and makes it easy to view/inspect dataframes and other items. I'm "closer to the data" in R Studio. Also line-by-line/blockwise execution of code is really helpful.

So my question: Is there anything like R Studio for Python (preferably non-browser based)? Can someone recommend an editor for Python with similar features like R Studio (especially wrt data handling)?


texts: I am trying to create a ranking system for recommending books to an user. Let's suppose we have some subjects of books like 'A', 'B', 'C', 'D' and from the past behaviour, it is observed that the user is more inclined towards subjects 'A' and 'B'. My dataset has a format of binary classification dataset i.e. it contains specifications of books and a labels specifying whether the user read it or not (1/0). Based on this, I assign probabilities of the user liking a book, using a model like RandomForestClassifier. The problem is, based on the past behaviour, the books with subject 'A' and 'B' always tend to get higher probability scores. Is there any way to add some 'discoverability' to the model so that it assigns high scores to even books of other subjects?


texts:
I am dealing with an unbalanced binary classification problem. The problem is so unbalanced (2:98) and hard to predict that I am interested in probability of the positive outcome instead of trying to predict the actual binary output. Depending on the model used this require either to calibrate the model in probabilities (tree based models) or transforming scores into probabilities using some spline (NN).
But in the end, for all practical matters I use buckets of probabilities. With 2% being the average I use predefined buckets like these : [0,0.5%],[0.5%,1%],[1%,2%],[2%,5%],[5%,10%],[10%,25%],[25%,50%],[50%,100%]. To attribute a bucket to an instance, the simple way to go is to just put the instance in the bucket that match the predicted probability. I am aware this might not be optimal (removing information on the output, not using the optimal buckets ... etc.) but let's say those buckets are an expert-based constraint (or enforced by law in some cases).
I have some doubts on the necessity to go trough individual probability to predict a bucket. I also have the feeling that directly predicting buckets instead of probabilities could help remove a bit of the model instability I am observing (basically different random initialisation for NN might yield very different individual output - I am usually painfully dealing with this trough regularisation).
To get a general solution I was trying to find some loss function that would match this ordinal/multi-class/calibrated in probabilities problem, but couldn't find anything. So I am asking you: is there a general approach to predict given buckets of probabilities ? (Feel free to give model specific approaches instead / or give an explanation on why this might be a bad idea.)


texts: When using Partial Least Squares (PLS), does the response distribution need to be symmetric?

texts:
I've been working through the exercises in the Applied Predictive Modeling book.  The solution guide for Chapter 6 exercise 2 states that a log transformation is required to transform the response variable.  https://github.com/topepo/APM_Exercises/blob/master/Ch_06.pdf

From the linked pdf:

"Before pre-processing and modeling the data, let’s examine the distribution of the permeability response (left-hand plot in Figure 4). Clearly the distribution is not symmetric, but is skewed- to-the-right. When the response has this kind of distribution, an appropriate next step is to log- transform the data. We then should verify if the transformation induced a symmetric distribution (right-hand plot in Figure 4). For these data, the log-transformation did induce an approximately symmetric distribution and we will model the log-transformed response."

Nowhere in the original textbook can I find a requirement to transform the response variable if it's not symmetric when using PLS.  Is there a reference someone could share that outlines this requirement if it exists?


texts:
I'm develloping an object detection models to detect lesions in video record from operation.
A problem observed during evaluation is that, given that an operation is specifique to a patient, if a lesion is correctly detected, it has a good chance to be correctly detected during the whole video. The opposite is also true, a non-detected lesion will likely not be detected during the whole video.
The video are from real case, so they duration are not balanced, some video are longer than other.
Currently, I'm using the MaP frame-wise , but I believe it is not adequate with the issue I mention abose: some patients with longer video have more "weight" than the other.
An idea would be to calculate the MaP patient-wise (or video wise) then average it ?
Is there a standart approach ?


texts: Python OneHotEncoder Using Many Dummy Variables or better practice?

texts:
I am building a neural network and am at the point of using OneHotEncoder on many independent(categorical) variables.  I would like to know if I am approaching this properly with dummy variables or if since all of my variables require dummy variables there may be a better way.

df
    UserName    Token                       ThreadID    ChildEXE
0   TAG     TokenElevationTypeDefault (1)   20788       splunk-MonitorNoHandle.exe
1   TAG     TokenElevationTypeDefault (1)   19088       splunk-optimize.exe
2   TAG     TokenElevationTypeDefault (1)   2840        net.exe
807 User    TokenElevationTypeFull (2)      18740       E2CheckFileSync.exe
808 User    TokenElevationTypeFull (2)      18740       E2check.exe
809 User    TokenElevationTypeFull (2)      18740       E2check.exe
811 Local   TokenElevationTypeFull (2)      18740       sc.exe

ParentEXE           ChildFilePath               ParentFilePath
splunkd.exe         C:\Program Files\Splunk\bin C:\Program Files\Splunk\bin 0
splunkd.exe         C:\Program Files\Splunk\bin C:\Program Files\Splunk\bin 0
dagent.exe          C:\Windows\System32         C:\Program Files\Dagent 0
wscript.exe         \Device\Mup\sysvol          C:\Windows  1
E2CheckFileSync.exe C:\Util                     \Device\Mup\sysvol\ 1
cmd.exe             C:\Windows\SysWOW64         C:\Util\E2Check 1
cmd.exe             C:\Windows                  C:\Windows\SysWOW64 1

DependentVariable
0
0
0
1
1
1
1

I import the data and using the LabelEncoder on the independent variables

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

#IMPORT DATA
#Matrix x of features
X = df.iloc[:, 0:7].values
#Dependent variable
y = df.iloc[:, 7].values

#Encoding Independent Variable
#Need a label encoder for every categorical variable
#Converts categorical into number - set correct index of column
#Encode "UserName"
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Encode "Token"
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])
#Encode "ChildEXE"
labelencoder_X_3 = LabelEncoder()
X[:, 3] = labelencoder_X_3.fit_transform(X[:, 3])
#Encode "ParentEXE"
labelencoder_X_4 = LabelEncoder()
X[:, 4] = labelencoder_X_4.fit_transform(X[:, 4])
#Encode "ChildFilePath"
labelencoder_X_5 = LabelEncoder()
X[:, 5] = labelencoder_X_5.fit_transform(X[:, 5])
#Encode "ParentFilePath"
labelencoder_X_6 = LabelEncoder()
X[:, 6] = labelencoder_X_6.fit_transform(X[:, 6])

This gives me the following array:

X
array([[2, 0, 20788, ..., 46, 31, 24],
       [2, 0, 19088, ..., 46, 31, 24],
       [2, 0, 2840, ..., 27, 42, 15],
       ...,
       [2, 0, 20148, ..., 17, 40, 32],
       [2, 0, 20148, ..., 47, 23, 0],
       [2, 0, 3176, ..., 48, 42, 32]], dtype=object)

Now for all of the independent variables I have to create dummy variables:

Should I use:

onehotencoder = OneHotEncoder(categorical_features = [0, 1, 2, 3, 4, 5, 6])
X = onehotencoder.fit_transform(X).toarray()

Which gives me:

X
array([[0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],
       ...,
       [0., 0., 1., ..., 1., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],
       [0., 0., 1., ..., 1., 0., 0.]])

Or is there a better way to approach this this?


texts:
I'm working with an unbalanced classification problem, in which the target variable contains:

np.bincount(y_train)
array([151953,  13273])

i.e. 151953  zeroes and 13273 ones.

To deal with this I'm using XGBoost's weight parameter when defining the DMatrix:

dtrain = xgb.DMatrix(data=x_train,
                     label=y_train,
                     weight=weights)

For the weights I've been using:

bc = np.bincount(y_train)
n_samples = bc.sum()
n_classes = len(bc)
weights = n_samples / (n_classes * bc)
w = weights[y_train.values]

Where weightsis array([0.54367469, 6.22413923]), and with the last line of code I'm just indexing it using the binary values in y_train. This seems like the correct approach to define the weights, since it represents the actual ratio between the amount of values of one class vs the other. However this seems to be favoring the minoritary class, which can be seen by inspecting the confusion matrix:

array([[18881, 19195],
       [  657,  2574]])

So just by trying out different weight values, I've realized that with a fairly close weight ratio, specifically array([1, 7]), the results seem much more reasonable:

array([[23020, 15056],
       [  837,  2394]])

So my question is:

Why using the actual weights of each class yielding poor metrics?
Which is the right way to set the weights for an unbalanced problem?


texts:
So, I want to create a Player Profile Radar Chart something like this:

Not only the scale of each variable different, but also I want a reversed scale for some statistics like the 'dispossessed' stat, where less actually means good.

One solution for the variable scale for each statistic maybe is setting a benchmark and then calculating a score on a scale of 100?

But, How do I display the actual numbers on the chart then? Also, how do I get the reversed scale for some of the statistics.

Currently working in Excel. What is the most powerful tool to create a complex chart like this?


texts: What's wrong with my gradient descent? (simple, one variable)

texts:
# calculating the cost function
theta = np.array([[0], [0]])
def h(X, theta):
    return np.matmul(X, theta)

def cost(X, y, theta, m):
    return 1/(2* m) * np.sum((h(X, theta)-y)**2)

print(cost(X, y, theta, m))
# 1-variable gradient decent
cost_vals = []
def grad_decent(X, y, theta, m, alpha):
    theta1 = theta[0, 0]
    theta2 = theta[1, 0]
    for i in range(0, iter):
        cost_vals.append(cost(X, y, np.array([[theta1], [theta2]]), m))
        theta1 = theta1 - alpha/m * np.sum((h(X, theta)-y) * X[:, 0])
        theta2 = theta2 - alpha/m * np.sum((h(X, theta)-y) * X[:, 1])
        # print(f"Iteration {i}: ", theta1, ", ", theta2)

    return np.array([[theta1], [theta2]])

As one can see I'm doing gradient descent with only one feature, i.e. I want to fit a line of the form theta1 + theta2 * x to my data. I know that for such small feature sizes one would definitely use the normal equation, but I wanted to implement it with gradient descent. My learning rate alpha is 0.01 and I'm following the machine learning course of andrew ng on coursera. However, instead of converging, my cost values increase so there is some mistake in the algorithm which I can't find. X is of the shape 97x2 where the first column is just ones since there is only one feature but I want to fit a polynomial with two coefficients.


texts: How to calculate the prediction interval?

texts:
I am recently engaged in a project of predicting the future blood glucose values of the patient. I was able to get a lower RMSE and MAE by using Gradient Boosting Algorithm. But although the error is low the predicted and actual values have a significant difference. For example, in the below image the blue line is actual while the orange is the predicted(using Gradient Boosting algorithm).

With the above prediction it's clear that it's not that accurate. So I wanted to predict the prediction interval with the upper bound of 0.9 and lower bound of 0.1, such that I will be able to tell that their next glucose values will lie within a specific range. I saw in an article that they have used gradient boosting algorithm to predict the intervals with the quantile loss function. The code I typed for predicting the prediction intervals is as follows,
LOWER_ALPHA = 0.1
UPPER_ALPHA = 0.9
lower_model = ensemble.GradientBoostingRegressor(loss="quantile",
                                    alpha=LOWER_ALPHA)

upper_model = ensemble.GradientBoostingRegressor(loss="quantile",
                                    alpha=UPPER_ALPHA)
lower_model.fit(X_train, y_train)
upper_model.fit(X_train, y_train)
predictions = pd.DataFrame(y_test)
predictions['lower'] = lower_model.predict(X_test)
predictions['upper'] = upper_model.predict(X_test)
plt.figure(figsize = (20,20))
plt.plot(predictions['lower'])
plt.plot(predicted)
plt.plot(predictions['upper'])

What I got as a result is,

But I feel like I have done something wrong. Because the gap between the upper bound and lower bound is very high and for that type of prediction we don't need machine learning. For example if we look at the first point the upper bound lies somewhere around 215 and lower bound lies around 60, So as a result I what I can tell is that the next value is gonna lie within 60-215 range. What have I done wrong? And can anyone suggest a better way of calculating the prediction interval? Thanks in advance.


texts:
Does anyone know a libarary for performing coreference resolution on German texts?

As far as I know, OpenNLP and Stanford NLP are not able to perform coreference resolution for German Texts.

The only tool that I know is CorZu which is a python library.


texts: Multi-Step Forecast for Multivariate Time Series (LSTM) Keras

texts:
I have been trying to understand how to build LSTM model for multivariate time series forecast using Keras but I am still unsure how to present the data in the correct shape.

My Dataset:
•   5 cities.
•   Each with 2 features. Temperature and humidity.
•   Daily records of the last 10 weeks(Monday, Tuesday, …, Sunday)

What I want to do:

Given Monday’s record of the new week of a city, I'd like to forecast the Temperature and humidity  for the remaining 6 days of that city. I.e. Multi-Step Forecast for Multivariate Time Series. Output shape(6,2)

How I have started off:

For each city, the input shape [(num_samples, num_time_steps, num_features) ] would be  (10, 7, 2).

For 10 weeks, I will have five unique samples(5-cities) with the same shape  (70, 2). So if I stack all vertically I will have (350, 2) or 3D shape (50,7,2).
Then create a supervised series with lag 1, I will have a shape(244, 4)

# Split train/test data.
train on 7-weeks. So input_shape= 5*(7,7,2) = (35,7,2)
test on 3-weeks. . So input_shape= 5*(3,7,2) = (15,7, 2)

What I am confused about:

The above layout seems to disregard the unique nature of each sample. I looked at this but still, a bit confused on how to transform it to a regression model.

I want the network to train each city's data separately as in this pic
I would appreciate any suggestion.
Thanks


texts: Convert Lasagne to Keras code (CNN -> LSTM)

texts:
I would like to convert this Lasagne code:

et = {}
net['input'] = lasagne.layers.InputLayer((100, 1, 24, 113))
net['conv1/5x1'] = lasagne.layers.Conv2DLayer(net['input'], 64, (5, 1))
net['shuff'] = lasagne.layers.DimshuffleLayer(net['conv1/5x1'], (0, 2, 1, 3))
net['lstm1'] = lasagne.layers.LSTMLayer(net['shuff'], 128)

in Keras code. Currently I came up with this:

multi_input = Input(shape=(1, 24, 113), name='multi_input')
y = Conv2D(64, (5, 1), activation='relu', data_format='channels_first')(multi_input)
y = LSTM(128)(y)

But I get the error: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4

I know the difficult part is to connect an LSTM to the output of Conv2D. Maybe using TimeDistributed?


texts: Consider for a dataset with many transactions each with items sold. Now, consider that we are given minimum support level and confidence level and that we have constructed the tables L1 and L2 each having many item sets that are all higher than the minimum support level. Consider that we can further have many three combinations items sets, however, none of them is higher than the minimum support level, therefore L3 is empty, then can we generate association rules from L2 ? or we just cant generate any rules since L3 is empty?


texts: Ideas for model to predict wind speed in a local kitesurfing spot

texts:
We have a sensor in a local kite spot that captures real time wind speed, we've found that we can somewhat predict if there's going to be "good" wind by looking at different forecast models and weather conditions.
I think it should be easy to train a model that takes input from different forecast models as well as real time data to predict with higher accuracy the speed in the spot.
Do you think this would be too crazy? if not how would you suggest I approach building this, having no experience with machine learning at all


texts: Error in FUN(newX[, i], ...) : argument "Iclo" is missing, with no default


texts:
Is there a way to reload all attributes after having removed someones without reopening the data file ?

Any help please ?


texts:
Data
For simplicity I propose the following scheme:

I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous:

             Gene_1    Gene_2    Gene_3
Patient_1    220.43    12,959    12,311
Patient_2    270.27    12,870    13,234

             miRNA_1   miRNA_2    miRNA_3
Patient_1    220.43    12,959     12,311
Patient_2    270.27    12,870     13,234

I must draw correlations all the genes against all the miRNAs (using either Pearson, Kendall or Spearman correlation, it doesn't matter) establishing as final result the following structure:

Gene     miRNA      Correlation  P-value        Adjusted P-value
Gen_1    miRNA_1    0,959        0.00311        0.00014
Gen_1    miRNA_2    -0,039       0.00311        0.00014
Gen_1    miRNA_3    -0,344       0.00311        0.00014
Gen_2    miRNA_1    0,1333       0.00311        0.00014
Gen_2    miRNA_2    0,877        0.00311        0.00014
...

Problem
The result of the Cross Join (all against all) can result in a DataFrame with billions of rows. To give a dimension of the required space, leaving aside the columns of the gene and the miRNAs and considering a small result of 300 million rows would need 300000000 * (16 bits of the correlation in floating pt + 64 bits of p-value + 64 bits of adjusted p-value) = 5 GB approx.
In order to optimize the memory usage, I do the computation in batches, the problem is with the adjusted p-values since I use the method of Benjamini & Hochberg (1995) with the function p.adjust of R (using a Python wrapper) that requires the complete p-value array, which makes me run out of memory.
Is there any way to compute, either from another library or another similar statistical method, the p-value adjusted in batches?
I already tried the FastLSU technique to be able to filter out some p-values that are not significant, but as I understood it, when I get the adjusted p-value for each row I need to know all the total p-values. If I could get for each row the adjusted p-value my problem would be solved since I could download the results of the batches to disk and goodbye to complications.
If someone could shed some light on the subject I would be very grateful


texts: Loading Street View House Numbers Dataset in Python

texts:
I am trying to work with the Street View House Numbers Dataset in Python. To load the .mat file digitStruct.mat in python I am using H5py

import h5py
labels_file = './sv/train/digitStruct.mat'
f = h5py.File(labels_file)
struct= f.values()
names = struct[1].values()
print(names[1][1].value)

I get [<HDF5 object reference>] but I need to know the actual string


texts: Test and Control analysis to measure the impact of Change in sales-rep for territories

texts:
I hope you all are doing well.

Before I proceed with my problem statement, a few terminologies for reference -

Territory = Sales Territory - Think of it like a county/region
assigned to a particular rep and no overlap of area/customers between
2 Reps
Rep/ Sales Rep = Sales Representative who visits customers to convert
sales
Calls = Number of times a customer is visited by the rep in a month
Goal Attainment = % of Target achieved for the month - e.g. if Target
was 500 units and total sales were 600, Attainment is 120%

I am working on a statistical/data-science problem and would like to get some thoughts on how to approach the hypothesis testing.

We have some attributes at Territory level, viz. - Total Sales, New to Brand Sales, Total Calls (by Reps) and % Goal Attainment, all rolled-up at Territory-Month level. I have data for 2-years, Jan 2018 to Jan 2020.

Now the problem I want to solve is to do a test & control analyses to see if the Sales Rep change has any impact on the territory performance (sales) or not. The Test group would be a set of 30 Territories who have undergone a Rep change in the last 2 years (Rep changed at least 6 months ago, i.e. no later than July 2019) and the control group having similar territories without any change in Sales Rep for last 2 years.

I want to get some thoughts on how to find a matching control pair for each test territory. I have a list of 107 Territories with 30 having a rep change (basically test group) and remaining 77 available to form a control group. Since Sales are my target variable, I'm thinking of creating a composite score on normalized Calls and Goal Attainment and calculate distance from Mean (or Mean-Squared value) and pair the test territories with control ones having the least distance from mean for the pair.

After my test and control group is formed, I want to conduct hypothesis testing, the null hypothesis being Rep Change doesn't impact sales - for this, I'm planning to use two-tailed t-test (for n<= 30) at 95% significance level. I would really appreciate your thoughts on this approach and If I could do anything else for robust testing.


texts: How to turn classification tree results into a GIS map

texts:
I'm new-ish to machine learning, so this could be a silly question. Apologies if so.

The idea is is to predict groundwater occurrence based on a regression tree.
This is my conceptual model:

Target variable: well success rate (the spatial location of those wells that did/didn't find groundwater). This is spatially-distributed ground-truth data in point format with spatial coordinates and so on. There are like 1200 data points.
Explanatory variables: a bunch of indirect indicators (topography, landforms, rainfall, geology, etc). These are raster layers. I simply grab the pixel value of each one that corresponds to the location of the wells in the target dataset.

I have managed to develop and train a regression tree in Spyder which predicts groundwater occurrence with reasonable accuracy. Now I'd like to use turn the results into a groundwater occurrence map in a GIS environment (QGIS, to be specific).

If regression trees produced "coefficients" I could simply multiply them by each pixel value in each explanatory variable layer to obtain a prediction for each pixel, and with that develop the map. However, I understand this is not the case because DTs use if/else logic instead of coefficients.

So I'm stuck at this point. Since the model works I feel I'm kinda close, but I wouldn't have a clue as to how to proceed.

So how would you guys go about this?


texts: Remote sensing image data storage in hadoop hdfs

texts: I am new to hadoop environment. anyone kindly please help me how to store the remote sensing image data in hadoop hdfs? thanks in advance


texts: How many items does apriori algorithm consider while working on data set of say 100 product?

texts:
For finding supermarket product association rules it is said that apriori is suited to identify frequent itemsets. From here then we can mine association rules.
Given 100 products, and their transactions, we need to consider itemset of one item, then 2nd item,... upto the 100th item. Then consider 1st and 2nd item, 2nd and 3rd,... and so on..
I am reading that this is a computationally complicated problem and hence we have theapriori algorithm that reduces the computational complexity. Does apriori algorithm start with one item in the set and then till how many items does it process in the set?


texts: Formula to calculate confidence value in Adaboost

texts:
I am coding an AdaBoostClassifier with the two class variant of SAMME algorithm. Here is the code.

def I(flag):
    return 1 if flag else 0

def sign(x):
    return abs(x)/x if x!=0 else 1

AdaBoost Class

class AdaBoost:

    def __init__(self,n_estimators=50):
        self.n_estimators = n_estimators
        self.models = [None]*n_estimators

    def fit(self,X,y):

        X = np.float64(X)
        N = len(y)
        w = np.array([1/N for i in range(N)])

        for m in range(self.n_estimators):

            Gm = DecisionTreeClassifier(max_depth=1)\
                        .fit(X,y,sample_weight=w).predict

            errM = sum([w[i]*I(y[i]!=Gm(X[i].reshape(1,-1))) \
                        for i in range(N)])/sum(w)

            '''Confidence Value'''
            #BetaM = (1/2)*(np.log((1-errM)/errM))
            BetaM = np.log((1-errM)/errM)

            w = [w[i]*np.exp(BetaM*I(y[i]!=Gm(X[i].reshape(1,-1))))\
                     for i in range(N)]

            self.models[m] = (BetaM,Gm)

    def predict(self,X):

        y = 0
        for m in range(self.n_estimators):
            BetaM,Gm = self.models[m]
            y += BetaM*Gm(X)
        signA = np.vectorize(sign)
        y = np.where(signA(y)==-1,-1,1)
        return y

The much I know the formula for confidence is

The much I read, the actual minima occurs when c=1/2 but for any value of c the classifier should produce the same result. But when I am coding the class the output for c = 1 and c = (1/2) are coming different. Moreover if I am not multiplying anything ie. c=1 then the output of my classifier is better and produces identical results with the sklearn implementation of AdaBoost Classifier.

So why multiplying 1/2 is giving bad results?


texts: asymmetric cost function for deep neural network binary classifier

texts:
I am building a deep neural network based binary classifier, with single output. The loss function I actually want to minimize is
$$ \mathcal L(\hat y,y) =
\begin{cases}
0,  & \text{if $\hat y$ = 0} \\
1,  & \text{if $\hat y$ = 1 & $y$ = 0} \\
\gamma \approx -0.7 ,  & \text{if $\hat y$ = 1 & $y$ = 1}
\end{cases}
$$
where $y \in \{0;1\}$ is sample's label, $\hat y \in \{0;1\}$ - classifier's output and $\gamma$ - a hyperparameter. I think this is a case of assymmetric loss. (One can see it as betting: no reward when not betting, stake 1\$, payout 1.7\$ if bet wins)

From what I know by now, this loss function is probably not suited well for backpropagation and gradient descent. Question: Is there a better-suited formulation?

The often used cross-entropy loss doesn't allow for trade-off tuning between precision and recall. LINEX and LINLIN are assymetric by design purpose, but I coundn't find an example of a deep NN trained with them. An alternative could be leaving the loss function as it is and resorting to SPSA, but I'd like to keep it simple if possible.

Edit: I came up with
$$ \mathcal L(\hat y,y) = - (\gamma \hat y)^{y}(-\hat y)^{1-y}$$
At the moment, I have no clue if it is going to work for NN learning. I'm concerned (maybe unnecessarily) that, without logarithm, it is not convex regarding NN weights. (Last layer has sigmoid activation.) Here is the log loss shown for comparison.
$$ \mathcal L(\hat y,y) = - (y\log \hat y + (1-y)\log(1-\hat y)) = -\log[\hat y^y (1-\hat y)^{1-y}]$$


texts: I have a dataset born to solve a classification problem. Due to the imbalances of the Y, i choose to move to an anomaly detection task. Should I use the Y i have inside the anomaly detection model as a features? Is it an overfitting Risk?


texts: TypeError: 'tuple' object cannot be interpreted as an integer

texts:
I have an image (480,640,3) and I am trying to resize it to (150,150,1). Since my Keras model has been trained on images of shape (150,150,1), unfortunately I cannot seem to resize it using the PIL library. Any suggestions on how to fix this?

This is the code I am using right now:

The error is caused on the line image5.resize()

    stream.seek(0)
    data = numpy.fromstring(stream.getvalue() , dtype = numpy.uint8)
    image5 = cv.imdecode(data , 1)
    print(image5.shape)
    #cv.imwrite('uhhu.png',image5)
    img = image5.resize((150,150,1 ), Image.ANTIALIAS)
    cv.imwrite('hhh.png',img)
    x = img_to_array(img)
    x = x.reshape((1,) + x.shape)
    x = x/255
    x = numpy.array(x)
    print(x.shape)
    #score = loaded_model.predict(img)
    #print(score)


texts:
What is the correct strategy of using Grid Search? Am I understand correctly that to use correctly Grid Search I should:

Give Grid Search initial parameters that have wide range. For example if parameter alpha usually ranges from 0 to 100 I should give list of parameters like [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100] and if parameter beta usually ranges from 0.1 to 0.0001 give list of parameters [0.1, 0.01, 0.001, 0.0001]?

Now we picked the parameters 'alpha': [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100], 'beta':[0.1, 0.01, 0.001, 0.0001] and we run Grid Search with them.

We got best parameters from Grid Search: 'alpha': 50, 'beta':0.0001

What is the next step here? How should we run Grid Search again with:
Variant 1:
'alpha': [0, 10, 20, 30, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 70, 80, 90, 100], 'beta':[0.1, 0.01, 0.001, 0.0001, 0.00001, 0000001]

OR
Variant 2
 'alpha': [42, 44, 46, 48, 50, 52, 54, 56, 58], 'beta':[0.0001, 0.00001, 0000001]

Is variant 2 good variant? I thought that variant 1 is better because the best combination of parameters may be 'alpha': 30, 'beta':0.00001. Is it possible? Or I should remove all parameters that does not perform well in first Grid Search as in variant 2?

Repeat step 3 until found parameters that are giving best score.

So my question is:

Is choosing parameters in step 1 correct way of doing that?
Which variant of step 3 should I use?
How does that strategy looks overall? Is it good strategy or anything should be changed?


texts:
I work with python and images of watches (examples: watch_1, watch_2, watch_3). My aim is to take a photo of a random watch and then find the most similar watches to it in my database. Obviously, one main feature which distinguishes the watches are their shape (square, rectangular, round, oval) but there are also other ones.

For now, I am just running a PCA and a KNN on rgb images of watches to find the most similar ones among them. My source code is the following:

import cv2
import numpy as np
import os
from glob import glob
from sklearn.decomposition import PCA
from sklearn import neighbors
from sklearn import preprocessing

data = []

# Read images from file
for filename in glob('Watches/*.jpg'):

    img = cv2.imread(filename)
    height, width = img.shape[:2]
    img = np.array(img)

    # Check that all my images are of the same resolution
    if height == 529 and width == 940:

        # Reshape each image so that it is stored in one line
        img = np.concatenate(img, axis=0)
        img = np.concatenate(img, axis=0)
        data.append(img)

# Normalise data
data = np.array(data)
Norm = preprocessing.Normalizer()
Norm.fit(data)
data = Norm.transform(data)

# PCA model
pca = PCA(0.95)
pca.fit(data)
data = pca.transform(data)

# K-Nearest neighbours
knn = neighbors.NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric='minkowski').fit(data)
distances, indices = knn.kneighbors(data)

print(indices)

However when I try to run this script for more than 1500 rgb images then I get a MemoryError at the point where the data are processed by the PCA method.

Is this normal for a pc with 24GB RAM and 3.6GHz Intel Core CPU without any discrete GPU?

How can I overcome this?

Shall I use another method like Incremental PCA (or a deep learning algorithm) or simply shall I buy a discrete GPU?


texts: Looking for a classification (?) algorithm for linearly separable but unlabeled data points

texts:
I have a dataset that is linearly separable with two lines - something like that:

Now I'am looking for the right kind of algorithm to do what I guess a SVM would do with labeled data - find the margins or decision boundaries for each class (three in this case). I tried spectral clustering and Gaussian mixture, but those don't seem to work.
Origin of the data: edges from tracked bounding boxes for cars on a three lane road. Thanks!

Edit: K-Means is apparently not really working for this kind of distribution:

I also tried out Ismor's suggestion to do some transformation before k-means, which results in:

The output here is very sensitive to the setting of the origin for y0, I couldn't get it right...


texts: Do generative model produce varying outputs for same input

texts:
I am new to data sciences. I believe the generative model generate responses on-the-fly for a valid user input. Is it correct to assume that such models would generate different responses for the same question?
For e.g: if we trained the model on say medical data. Now if user 1 asked "what is fever" and user 2 asked the same question, could be that user 1 and 2 will receive different answers? if this is so then how to circumvent this problem?
thanks in advance


texts: how to decide categorical variables for prediction

texts:
I have a dataset that contains weekly sales for stores and categories. It looks like this:

I would like to apply gradient boosting method to predict weekly sales. My question is: Should I create dummy variables for categories(1 to 7 which indicates product type) and stores(1 to 11)?


texts:
How can 2-layer networks be used to classify more than two categories?
Can it be done just by adding more units/nodes into the existing layers?


texts: is it better to use embedding with attributes in collaborative filtering or content-based approach?

texts:
I have a dataset with different text documents, a set of users who each read a different document, some historical info such as their reading speed, and other attributes related to the texts and users.
I want to predict their speed on a new set of documents. The new documents will not be the ones used in the test before, and different users tend to read different text in a non-repeated way.
Could this be treated as collaborative filtering with speed viewed as ratings? if so how to use the text/doc embedding as input? The user - text id or user - text embedding will barely form a matrix given the unique pairing, though user - text attribute could form a matrix.
Or is it better to treat it as a content-based approach? or still other approaches? What would you advise? Thanks.


texts: When forecasting time series, how does one incorporate the test data back into the model after training?

texts:
When you build a classification or regression model, you typically split the data into a train data set and a test data set. The test data is a randomly selected subset of the overall data.

Once you are done with the training, you discard the test data, and apply the model you built to new unknown data.

But in the case of time series, this wouldn't work: You can't just randomly choose any subset of the data for your test data set, since the data is ordered. The test data set has to be composed of later observations of the data, while the train data set is made up of earlier observations of the data.

Say for example that you have data for 11 months of the year, and you want to forecast the values for December. You would train your model with the data from January through September, and then test it on the data from October and November.

But after you've successfully trained your model, what do you do with the test data?

On one hand, it doesn't make sense to forecast the values for December using a model built with Jan-Sep data. Such a model would miss any important trends that occurred in Oct and Nov.

On the other if you bring the data from Oct and Nov back into the model, the parameters of the model will change, and so we're no longer sure that we are going to get the same accuracy that we got when we trained the model with just the Jan-Sep data.

There seems to be a dilemma here.

So how is this problem approached when using machine learning models, especially non parametric models like Neural Networks for time series forecasting? Do they incorporate the test data into the model or do they discard it?


texts:
I'm implementing a sequence-2-sequence model with RNN-VAE architecture, and I use an attention mechanism. I have problem in the decoder part.
I'm struggling with this error:   IndexError: list index out of range
When I run this code:

decoder_inputs = Input(shape=(len_target,))
decoder_emb = Embedding(input_dim=vocab_out_size, output_dim=embedding_dim)
decoder_lstm = LSTM(units=units, return_sequences=True, return_state=True)
decoder_lstm_out, _, _ = decoder_lstm(decoder_emb(decoder_inputs),
             initial_state=encoder_states)

print("enc_outputs", encoder_outputs.shape) # ==> (?,256)
print("decoder_lstm_out", decoder_lstm_out.shape)# ==> (?,12,256)
print("zzzzzz", z.shape) # ==> (?,256)

attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([z,z], decoder_lstm_out)

The error is raised at the last line, and the traceback given:

Traceback (most recent call last):
     File "malek_tuto.py", line 197, in <module>
           attn_out, attn_states = attn_layer([z,z], decoder_lstm_out)
     File "C:\Users\lightland\Anaconda3\lib\site-
       packages\tensorflow\python\keras\engine\base_layer.py", line 728, in
       __call__    self.build(input_shapes)
     File "D:\PFE\Contribution\modele\layers\attention.py", line 24, in
        build shape=tf.TensorShape((input_shape[0][3], input_shape[0][3])),
     File "C:\Users\lightland\Anaconda3\lib\site-
       packages\tensorflow\python\framework\tensor_shape.py", line 615, in
         __getitem__   return self._dims[key]
     IndexError: list index out of range

in AttentionLayer class , build function id defined by:

 def build(self, input_shape):

      assert isinstance(input_shape, list)
      print("hhhhhhhhhh",input_shape)
      print("jjknkjnjk")
      # Create a trainable weight variable for this layer.

       self.W_a = self.add_weight(name='W_a',
                               shape=tf.TensorShape((input_shape[0][2],
                                input_shape[0][2])),
                               initializer='uniform',
                               trainable=True)
       self.U_a = self.add_weight(name='U_a',
                               shape=tf.TensorShape((input_shape[1][2],
                               input_shape[0][2])),
                               initializer='uniform',
                               trainable=True)
       self.V_a = self.add_weight(name='V_a',
                               shape=tf.TensorShape((input_shape[0][2], 1)),
                               initializer='uniform',
                               trainable=True)

       super(AttentionLayer, self).build(input_shape)

If someone can help me I'll be so thankful, I cannot understant where the problem is, and how to resolve it.

Thanks in advance


texts: How to remove rows from a data frame that are identical to other df?

texts:
I have two data frames.
So how do I remove rows that have identical email addresses in df2 from df1?

>>df1
   First  Last  Email
0 Adam   Smith  email@email.com
1 John   Brown  email2@email.com
2 Joe    Max    email3@email.com
3 Will   Bill   email4@email.com

>>df2
  First  Last   Email
0 Adam   Smith  email@email.com
1 John   Brown  email2@email.com


texts: How to create a multiple layer perceptron with layers of specific sizes in keras?

texts:
I am trying to recreate a model based on its description. The model is described as a "10 layers of size 500 ... 10". From my understanding, the size refers to the number of hidden units, which I could customize using model.add(dense)

My code looks like this:

model = Sequential()
model.add(Conv2D(32, (3,3), activation='relu', input_shape=input_shape)
model.add(Dense(500, activation = 'relu'))
...
model.add(Dense(10, activation = 'softmax'))

However, it is yielding a very low accuracy rate (about 20% lower than what I was supposed to get).

Am I doing something wrong?


texts: Simple Q-learning neural network using numpy

texts:
import numpy as np
from numpy import exp, array, random, dot

R = np.matrix([[-1, -1, -1, -1,1, -1],        # for correct action the
reward is 1 and for wrong action it's -1
           [-1, -1, -1, 1, -1, 1],
           [-1, -1, -1, 1, -1, -1],
           [-1, 1, 1, -1, 1, -1],
           [-1, 1, 1, -1, -1, 1],
           [-1, 1, -1, -1, 1, 1]])

Q = np.matrix(np.zeros([6, 6]))  # Q matrix

gamma = 0.99  # Gamma (learning parameter).
lr = 0.1  # learning rate
initial_state = 1  # Initial state.
w = 2*np.random.random((6,1)) - 1  # random weights for each action

def sigmoid(x):
    return 1 / (1 + exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

def available_actions(state):  # This function returns all available actions
in the state given as an argument
    current_state_row = R[state,]
    return current_state_row

def sample_next_action(available_actions_range):  # This function chooses at
random which action to be performed within the rangeof all the available
actions.
    next_action = int(np.random.choice(available_actions_range, 1))
    print("next_action: ", next_action)
    return next_action

def update(action, gamma,predict):  # This function updates the Q matrix
according to the path selected

    Q[current_state, action] = action + gamma * predict
    return Q[current_state, action]

for i in range(1000): ## Training
    current_state = np.random.randint(0, int(Q.shape[0]))
    available_act = np.array(available_actions(current_state))
    action = np.max(available_act)

    Qout = np.multiply(available_act,w) # forward propagation # multiplying
    the selected action with the weights)
    predict = sigmoid(np.max(Qout))
    Qtarget = update(action, gamma, predict)  # target q values(rewards)
    loss = np.sum(Qtarget - Qout) # backward prop # calculating the error
     between predicted and target q values
    adjustment = np.multiply(action ,loss , Qout)
    w = w + adjustment # adjusting weights
    Q += Qout

print("Trained network:")
print("--------")
print("weights : ", w)
np.set_printoptions(precision=2,suppress=True)
print(Q / np.max(Q) * 100)  # Normalize the "trained" Q matrix

I am a newbie into neural networks and reinforcement learning. I am trying to create a neural network (for learning purpose) which contains a single neuron with 6 inputs and 1 output. The problems that I am facing are as follows -

I am not sure if the way I am trying to predict the Q value through my Q network is correct ? and I am unsure as the calculation that I am doing to find out the target Q value is correct.
When I run the code the weights become 'Nan' after a few iterations and this leads the Q matrix to have all the Nan values.


texts:
I have a fairly simple game in which I wish to use Q-learning to train an agent, but I have some questions regarding state representation. I'm new to RL so bare with me:

If you have a game where you and your enemy each have 3 players, each player has health and a location on a 10 x 10 grid. My understanding is that you would need to enumerate all possible states for this game in order to successfully implement Q-learning. How would this be done? If we forget about health for a second and we look at position alone, we have 6 players, each of which could be in any of the 100 locations so our state space for position alone would be 100^6. This seems like an exceptionally large matrix to be working with, and we haven'y even added a parameter for health. What am I missing? is this the point at which you look for alternative state representations?

Some guidance or links to relevant articles would be greatly appreciated. Thanks!


texts:
So I understand what LSTM units are. But I have trouble understanding the implementation / function in Keras framework.

Let's say, I add a layer

model.add(LSTM(5, ...))

does this mean, I have

five different LSTM units, that are independent, or
one LSTM unit, that is unfolded into five time steps?

I assume that the second answer is right, because else a lot of models I saw would not make much sense.

When I have as first hidden layer a LSTM layer with size 5, does this mean my input should be a time series with 5 time steps?

Please correct me if I'm wrong. I am confused because I thought one LSTM would use its own state in the next iteration.

What is the correct meaning of LSTM in Keras?


texts: Cost of greater than 1, is there an error?

texts:
I'm computing cost in the following way:

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(y, y_)
cost = tf.reduce_mean(cross_entropy);

For the first cost, I am getting 0.693147, which is to be expected on a binary classification when parameters/weights are initialized to 0.

I am using one_hot labels.

However, after completing a training epoch using stochastic gradient descent I am finding a cost of greater than 1.

Is this to be expected?


texts: Validation loss and validation accuracy stay the same in NN model

texts:
I am trying to train a keras NN regression model for music emotion prediction from audio features. (I am a beginner in NN and I am doing this as study project.)
I have 193 features for training/prediction and it should predict valence and arousal values.
I have prepared a NN model with 5 layers:
model = Sequential()
model.add(Dense(100, activation='elu', input_dim=193))
model.add(Dense(200, activation='elu'))
model.add(Dense(200, activation='elu'))
model.add(Dense(100, activation='elu'))
model.add(Dense(  2, activation='elu'))

And this is my loss and optimizer metrics:
model.compile( loss = "mean_squared_error", optimizer = 'RMSprop', metrics=['accuracy'] )

When I try to train this model, I get this graph for loss and validation:
So the model is trained and reaches accuracy of >0.9 on training data, but on test data accuracy wont fall, but it stays on ~0.5.

I don't know how to interpret this graph. I don't think this is overfitting, because validation accuracy wont fall, but it stays the same.
How can I try fix this?
Update:
I tried to add dropout and regularization and it worked in a way that now I clearly see that I have a problem with over-fitting. But now I am stuck again.
I can not make my model to decrease validation loss. It always stops at about 0.3 validation loss.
I tried changing my model architecture, data preprocessing, optimizer function, and nothing helped.


texts: Machine learning terminology (method, model, architecture, task, objective)

texts:
I got a bit confused about the usage of machine learning terminology in books / papers / discussions that seems somehow not completely consistent to me. Therefore, I want to know if you would agree with my understanding of the following terms (if not I would appreciate any suggestion for improvement). I come from a NLP background, so I hope it's ok to use this as example.
We have machine learning methods (e.g. BERT) that define a way to solve a problem / multiple problems by using a trained model. The model has an architecture (e.g LSTM) that defines its "building blocks" and a way how it is trained to be useful to solve the problem. This way is either referred to as a task that the model has to perform or as an objective that is used to train the model (e.g. language model objective). The model typically learns from this task or objective by using a loss function and backpropagation to adapt its values.


texts:
Is it correct to say that, in the Bias-Variance trade-off:

the bias error represents the ability of the model to possibly map the trend of the training set ?

and the variance represents how much the model is prone to variations if the dataset changes ?


texts: How do I approach a classification problem where one of the classes is defined by 'not any of the others'

texts:
Suppose that I am interested in three classes $c_1$, $c_2$, $c_3$. But my dataset actually contains several more real classes $(c_j)_{j=4}^n$.

The obvious answer is to define a new class $\hat c_4$ that refers to all classes $c_j$, $j>3$ but I suspect this is not a good idea since the samples in $\hat c_4$ will be rare and not very similar to each other.

To visualize what I'm trying to say, suppose I have the following two variable space and the classes $c_1$, $c_2$, $c_3$, $\hat c_4= \bigcup_{j=4}^n c_j$ are depicted in red, til, green and black respectively. This is how I suspect my data would look like.

Is there any standard way to approach this problem? What would be the most efficient classifier and why?


texts:
I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.

1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood that, for each word, we generate one training sample for each context word.

So, if we have the sentence "cat sat on the mat" we will have samples:

(cat, sat)
(sat, cat)
(sat, on)

And so on.

Then you train your network which will have the dimensions.

Input: $(V,1)$
Weights 1: $(d,V)$
Hidden layer: $(d,1)$
Weights 2: $(V,d)$
Output layer: $(V,1)$

Ok, so these dimensions match and we are good. For one given input, given that the weights are the same, we always have the same estimation in the output layer.

In this definition we have symmetric samples (for example, (cat, sat) and (sat, cat)) and saying that we use center words as inputs and context words as outputs is meaningless since they are interchangeable?

2 - Watching Stanford's NLP with Deep Learning class (38:54), it seems like for the same center word we can get different outputs:

The numbers in the red circles that I drew should be the same from what I understand. I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.

3- I saw in other places (couldn't find reference now) another formulation: for each input word, we represent the context as a vector with ones for the context words and zeros elsewhere. So, in this case, instead of one training example for each context word, we would have one training example for each center word.


texts: not getting point predictions from sagemaker clarify

texts: I wanted to create shap values for my predictions in sagemaker. I found out that I can use "clarify" functionality in sagemaker to get shap values. However, I want to get point predictions as well. The bias config in clarify can include a prediction function but it seems its just for the train data. I wonder how I can include the predict function in the shap.config part of that so it gives me both individual predictions as well as SHAP values.


texts: Combining multiple neural networks with different activation functions

texts:
I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the same range values.

P = (P1 + P2 + P3)/3

Where 0 < P1 < 1, -1 < P2 < 1, 0 < P3 < 1

Can I directly average the predictions or I need to perform a normalization to have all prediction fall into the same interval ?


texts: How do you evaluate ML model already deployed in production?

texts:
so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production.

But because people change, data and many other factors change as well. And performance of our model eventually will decrease. So then it needs to be replaced with the new, better model.

What are the common techniques, model stability tests, model performance tests, metrics after deployment? How to decide when to replace current model with newer one?


texts: How to specify steps_per_epoch and validation steps on infinite dataset?

texts:
I have a huge csv dataset with a size of 200 GB. I'm using CsvDataset to make dataset generator for loading data from the disk while training the model. I want all the data to be passed on each epoch. So, what should I pass in the parameters steps_per_epoch and validation steps.
Here is my Keras model using the data_set.

    training_csvs =  sorted(str(p) for p in pathlib.Path('.').glob("path-to-data/Train_DS/*/*.csv"))

    training_csvs

    training_dataset=tf.data.experimental.CsvDataset(
        training_csvs,
        record_defaults=defaults,
        compression_type=None,
        buffer_size=None,
        header=True,
        field_delim=',',
        # use_quote_delim=True,
        # na_value="",
        select_cols=selected_indices
        )

    print(type(training_dataset))
    for features in training_dataset.take(1):
        print("Training samples before mapping")
        print(features)

    validate_ds = training_dataset.map(preprocess).take(10).batch(100).repeat()
    train_ds = training_dataset.map(preprocess).skip(10).take(90).batch(100).repeat()

    model = tf.keras.Sequential([
        layers.Dense(256,activation='elu'),
        layers.Dense(128,activation='elu'),
        layers.Dense(64,activation='elu'),
        layers.Dense(1,activation='sigmoid')
        ])
    history = model.compile(optimizer='adam', loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
                            metrics=['accuracy'])

    model.fit(train_ds,
        validation_data=validate_ds,
        validation_steps=20,  # I think it's wrong, on each epoch only 20 batches would be used...
        steps_per_epoch= 20,
        epochs=20,
        verbose=1
        )


texts:
I used the Neural Network Toolbox in matlab to train my data. I used four training algorithms, Scaled Conjugate Gradient (SCG), Gradient Descent with momentum and adaptive learning back-propagation (GDX), Resilient Back Propagation (RBP) and Broyden-Fletcher-Goldfarb-Shanno quasi-Newton back propagation (BFG). I have fixed the seeds at different points and obtained the accuracy. This is what I get:

The first column contains the size of the feature set. I have added features and increased the size of feature set to analyse the performance. Initially I have ranked the features and then taken the top 8 feature as one set, the top 16 feature as the next set and so on. The first number before the '-' , is the performance of the algorithm on the training set, the second number after the '-' is the accuracy of the testing set. The train and test set has been divided into 60 and 20 respectively. The other 20 is the validation set. The learning algorithms each have been run with the same seed values to fix the accuracy.  I have fixed the seed to obtain each of the results btw, Like I have used rng(1), rng(10), rng(158), rng(250) and averaged the results to obtain one single pair of train-test accuracy, and I have done this for each pairs. As you can see I am getting the same accuracy for all feature set size for each of the individual training algorithm. The same data shows perturbation in SVM chen I change the set size. What does this mean?


texts: What does all zero mean decrease in accuracy (MDA/permutation importance) signify?

texts:
I have a model I've trained on ~3400 features with ~500 samples (40:20:20 train:test:val) that I've calculated MDA on using the eli5 package. However, all the features are zero when I calculate this with the train or test dataset.
My dataset is a fairly sparse dataset with mostly zeros and I'm trying to classify disease. Here is a sample:

The model is a random forest that is predicting disease class (0 or 1). the dataset is imbalanced with appx (90% 0 and 10% 1)
My model accuracies are as follows:

train: 0.9923469387755102

test: 1.0

val: 0.9764705882352941

confusion matrix for test
[[76  0]
[ 0  8]]

these are my gini importances:

MDAs are all zero for testing and training. Why is that and what does it mean for the generalizability of the model?


texts: Tensorflow Deep learning network not utilizing GPU?

texts:
I have a Nvidia GeForce GT 755M (PC), which I heard should be at least functional for running deep learning models. But when I train my model (DCGAN) and check the task manager process info (Win 10) I see close to 100% CPU utilization, and very little GPU activity (1-5%). That amount of GPU activity seems to be the constant even if I'm not running a model...so my concern is that I'm not using my graphics card at all, and that I would get better performance if I did.

Furthermore, when I run the following code it indicates that no GPU was found:

import tensorflow as tf
import warnings
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))

So my questions are the following:

Is my GPU something that should improve performance vs using the CPU?
Why isn't tensorflow finding the GPU?
How can I make Tensorflow utilize the GPU?

I am just beginning to learn about the hardware aspect of ML so any advice and any recommended beginners tutorials for getting basic understanding/intuition would be much appreciated!

Screenshot of component utilization while training a convolutional NN:

Thanks!


texts: Team Collaboration using Jupyter Notebooks?

texts:
Is there an alternative to JupyterHub that allows sharing of notebooks across team members. We need the following requirements:

Be able to run the server on a HPC.
Multiple user login.
Version control with Git or similar service.
Sharing of notebooks between team members.


texts:
I have been trying to find the correlation b/w the following type of temporal data for quite some time.

DataSet: A A B A B C B C B A B C X Y X Y Z X Y X Z Y Z A B A B C B (The actual dataset has 1000's of unique points)

I have to find some sort of similarity value b/w these points (my points I mean 'A', 'B', 'X', etc). For example similarity b/w 'A' and 'B' in above dataset is high so similarity('A', 'B') should return me a high value and similarity('A', 'X') should return a low value.

I am trying to calculate the similarity so that I can cluster the similar points. (For above dataset, possible good clusters are {A, B, C}, {X, Y, Z})

Till now I have tried the following approaches:

I tried to find segments of data whose correlation = len(segment)/unique(points in segment) value is high. If the value is high than that means in this segment the same type of points are coming back and forth and therefore highly correlated. The problem with this correlation formula is that it is not robust enough. For example Dataset is: X Y X X X X X Y X. Now the correlation value for subsegment with starting index = 3 and ending index = 7 is higher (correlation = 5/1) than the correlation value of the whole Dataset i.e 1 to end (correlation = 9/2). Now due to such problems I have to do a lot of manual settings, like select a segment having more unique points if correlation diff b/w those segments is less than some manually selected value (tradeoff b/w correlation value and unique points), and this does not work when I have 1000's of unique points.
Word2Vec: I ran this algorithm on the whole data as a single sentence (i.e merged the data in a single sentence separated by space). I don't exactly understand this algorithm but it seems to not work on a single sentence. It gives > 0.99 similarity b/w almost all points. I don't know why is this not working since it is supposed to learn the similarity b/w words based on how they are occurring in a sentence.

How should I approach this clustering problem? Can I tweak word2vec to fit for my problem? A better mathematical formula? Any other way?

Please correct me if I am doing something wrong.


texts: Data Scientists | Sectors , Departments and Profiles

texts:
I'm searching a lot about what people call Data Scientist. From my brief search, I've came with the conclusion that Data Scientists' are probably what you may call the future evolution of a Business Intelligence Professional, by focusing important topics such has machine learning, big data, etc.

In this search It's being difficult to define what may be the most important sectors, departments (inside companies) and main profiles to this subject.

Can someone help me on understanding what is happening to the business world around this subject?

Summary:

Most important sectors? (for example, is it the retail and bank sector? There are more?)
Most important departments? (are the majority of Data Scientists working on IT departments? Risk departments? Marketing? I have no idea.)
Most important topics about the profile of a Data Scientist (what are the imperative skills to have?)


texts:
I've recently read about maxout in slides of a lecture and in the paper.

Is maxout the same as max pooling?


texts: How to represent a set of sets as a vector

texts:
I'm pretty new to machine learning.

I know I can represent a set of discrete values as a vector of 0/1 values. For instance, in the set of features {a, b, c, d, e}, the subset containing {a, c} can be represented as [1, 0, 1, 0, 0] and the subset containing {c, d, e} can be represented as [0, 0, 1, 1, 1], meaning I have as many dimensions as elements, which is workable when you have a finite (and small) number of elements.

But now, for a clustering task, I want to represent sets of sets, like, for instance, representing the set {{a, c}, {c, d, e}}. How can I do that? Here, the basic 0/1 approach won't work, as I'll have 2^n possible combinations. What is the workaround, if any?

Edit: here is the transcription as a less abstract, more business problem. I want to find clusters of people according to the trips they made. A trip consists in a set of cities visited, and a set of transportation used. For instance, people might have visited cities in the set {Rabat, Alger, Marrakech, Tunis, Hammamet} with transportation such as {car, plane, train}. A trip could be {Rabat, Marrakech, plane} or {Alger, Marrakech, Tunis, car, train}. Note the order in which cities were visited, or the order in which vehicles were used, is not considered.

An example of the items I want to find clusters of could be a person having made those two trips, represented as p1 = {{Rabat, Marrakech, plane}, {Alger, Marrakech, Tunis, car, train}}.


texts: Reinforcement Learning using PPO2 in openai gym retro, mario not learning the clear the easy episode

texts:
I am training mario game in retro using ppo2 baselines for some time. I have tried level3 and level1 too. But even after full training when I play using saved checkpoints, the mario is not able to finish the level. Mostly it falls into the hole or does not even reach half of first easy level.
Do I need to write reward scripts? Or tune some more hyperparameters?
If someone has experience using gym, retro or training such games, then they can suggest.

The code for training:

python -m baselines.run --alg=ppo2 --env=SuperMarioBros-Nes --gamestate=Level1-1.state --num_timesteps=1e7

The code for running mario using saved checkpoint:

python -m baselines.run --alg=ppo2 --env=SuperMarioBros-Nes --gamestate=Level1-1.state --num_timesteps=1e7 --load_path=C:\Users\SANDEE~1.BHU\AppData\Local\Temp\openai-2019-05-09-18-26-28-156556\checkpoints\08300 --play --num_env=1


texts: strange behavior for validation dataset loss and accuracy

texts:
I'm training an LSTM model for a binary classification and using validation_split = 0.2 (in Keras), I see the following curves for the loss and accuracies of the train and test datasets. Here are two questions:

Why for epoch < 10, the loss of the test dataset is much smaller than that of the training dataset and also its accuracies much higher?

At epoch around 80 the curves for the train and test dataset losses converge whereas their accuracies don't. Does it mean that the model is still underfitting?


texts: Plotting Stacked Histogram for Time-series data

texts:
Given the dataset:

timestamp                     item       itemcount
2019-03-18 07:40:08.759        A          10
2019-03-18 08:40:08.759        B          5
..................................................
2019-05-20 07:40:08.759        D          4
2019-05-21 07:40:08.759        E          8

I want to plot stacked histogram like:

where the x-axis should be the date and y axis the itemcount and stack will be each item. I want the graph with subplots for every month.

I am new here so will be happy to get any feedback on my mistakes. Thank you.

here's one sample code i found online which plots the same graph in the figure above.

  # Import Data
df = pd.read_csv("https://github.com/selva86/datasets/raw/master/mpg_ggplot2.csv")

# Prepare data
x_var = 'manufacturer'
groupby_var = 'class'
df_agg = df.loc[:, [x_var, groupby_var]].groupby(groupby_var)
vals = [df[x_var].values.tolist() for i, df in df_agg]

# Draw
plt.figure(figsize=(16,9), dpi= 80)
colors = [plt.cm.Spectral(i/float(len(vals)-1)) for i in range(len(vals))]
n, bins, patches = plt.hist(vals, df[x_var].unique().__len__(), stacked=True, density=False, color=colors[:len(vals)])

# Decoration
plt.legend({group:col for group, col in zip(np.unique(df[groupby_var]).tolist(), colors[:len(vals)])})
plt.title(f"Stacked Histogram of ${x_var}$ colored by ${groupby_var}$", fontsize=22)
plt.xlabel(x_var)
plt.ylabel("Frequency")
plt.ylim(0, 40)
plt.xticks(ticks=bins, labels=np.unique(df[x_var]).tolist(), rotation=90, horizontalalignment='left')
plt.show()


texts:
I've been working on a project where it consists of a dataset of profiles (50k rows) along with their position, age group and hobbies (200 columns). These features (except for the position) are graded from 0 to 10. A value near 0 means that they have no interest in that particular hobby or they are not part of the age group and a value near 10 means that they are more likely to be into this hobby(or in the age group etc...)
Some of the profiles are not graded for some hobbies. So, I decided to put a 0 so as they are not into this hobby. I have also deleted the columns that had more than 80% of empty values.
The profiles have 2 binary categories. I want to distinguish these categories with a training model. So I applied PCA to diminue the dimensions and then Logistic Regression for the classification. I got an accuracy of 0.68. So I was wondering if I may have preprocessed the data wrongly or could the model be improved in some way that I missed ? I also tried DecisionTreeClassifier and tried to prune the decision tree but still the same accuracy so that made me think maybe it comes from earlier steps?


texts: Time series data: How I measure influence of new product sales on existing product sales (statistically)?

texts:
Here my goal is…

Find Product 5 (New Product) is really influencing other product sales (product 1 to 4) or not?
If it is influencing other product sales, how much?

New to R and tried several related posts but didn’t find exact answer to my question. I love R and learning every day something new which helping us in taking data driven decisions.

My sample Dataset is like below (Week and Product 1 to Product 5 Sales per each week)
Here my new product is Product 5 and launched on Week 5.

Week    Product-1   Product-2   Product-3   Product-4   Product-5
1   2   4   5   5
2   4   4   6   4
3   4   4   6   5
4   4   4   6   6   4
5   4   6   5   3   5
6   2   7   6   4   3
7   3   8   7   5   6
8   2   9   9   3   6

Here my questions are

What is the best process or model to show the influence of product 5 (statistically)?
Do I need to run co-integration tests before I run correlation?
Example some of these products are never be correlated with Product 5
(example: growth in cockle growth vs. growth in electricity demand)
How I know correlation vs. causation in this mix?
Since my new
product launched on week 5, where I can start my correlations? Is it
from week 5 or from earlier weeks?
Do I need to test for stationarity first? and bring the data to stationary?