texts: Measurement error in mixture density network input

texts:
I am fairly new to keras / tensorflow but it looks like a mixture density network can solve my problem.
I’d like to predict a continuous variable using, in a simple case, 3 features. All features and the target are chemical measurements, and the instrument that measures them outputs a value plus a known standard deviation with an approximately normal distribution - ie the uncertainty in the measurement. This uncertainty is not uniform across each sample or feature.
I’d like to add the measurement uncertainty ( both the value and stdev) into the input of my MDN to see the affect of different measurement uncertainties and the propagation of error through the MDN on the resulting prediction.
But I am at a bit of a loss on how exactly to do this. I’d like to use keras plus tensorflow v2 but I’m open to other frameworks.
My idea was that the input would a 2d array of measured values (samples, features) and the measurement uncertainty of the same shape.
Is this possible?


texts: Difference between tf.keras.backend.one_hot and keras.utils.to_categorical

texts: I'm working on a classification project and need to do one hot encoding on my data set. I'm just wondering what is the difference between tf.keras.backend.one_hot and keras.utils.to_categorical, and is one of them preferred over the other?


texts: How to determine what algorithm to apply in the following dataset (included)

texts:
I have a dataset that I will be using to build a classifier on. Below I have plotted the First and the Second Principal Component of the data using sklearn.preprocessing.PCA. Since the two different classes are not well separated a linear classifier will not work here.

My question is which classifier would be best for this scenario.

My research brought be to KNN. But My intuition says the class ratio is highly imbalanced a large value of k in KNN would always tend towards the larger class count. It will be a nightmare to train it on SVM since therw are to many observations in the dataset and it will take too long.


texts: Computing spectral gap of p-laplacian, p > 2

texts: I'm looking for code allowing computation of the spectral gap of a graph p-laplacian with p > 2, i.e. the second largest eigenvalue. See http://www.ml.uni-saarland.de/code/pSpectralClustering/pSpectralClustering.htm for the concept (the code provided there only works for p \le 2). Alternatively, I'd like to know if this nonlinear optimization can be performed using some off-the-shelf methods, like straightforward gradient descent etc.


texts: machine learning on athlete performances to predict the time in a future race

texts:
Example Data

I have a dataset (in R as a data frame) of race results for athletes.

athlete racedistance time    location tracktype       date    coach
A          100       10.0       UK     typeA       2014-01-01  carlos
A          200       20.0       US     typeB       2014-02-01  carla
A          100        9.5      AUS     typeC       2014-03-01  chris
B          100       11.0       UK     typeA       2014-01-01  carla
B          200       21.0       US     typeB       2014-02-01  carlos
B          400       61.0      AUS     typeC       2014-03-01  carla
B          100       10.5      GER     typeA       2014-04-01  clive
C          100        9.5       UK     typeA       2014-01-01  clive
C          200       21.5       US     typeB       2014-02-01  chris

Question

Is there an appropriate machine learning algorithm or method that can use the previous results of each athlete as a feature, when trying to predict the time for an athlete in a future race?

For example, athlete A has three races, with one month rest between them. In the third race he performs slightly better than the first race over the same distance.

Can an algorithm learn that the second race had an effect on the athlete, which meant he performed better in the third race?

From what I've read on the subject and the training examples I've completed it would appear that each 'row' of data should be independent, is this the case for all ML algorithms? Is there another prediction technique I should be considering to solve this type of problem?


texts: Why are my evaluation stats giving weird results when training a neural network?

texts:
I am training an RNN which uses LSTM layers on stock data. Upon training and getting the evaluation stats, I get the same stats almost every time, which are very bad. Here is what my stats look like:
   Column    MSE            MAE            RMSE           RSE            PC             R^2
col_0     2.14646e+04    9.68327e+01    1.46508e+02    1.77569e+00    NaN            -7.75692e-01

First of all, is there any reason why the Pearson Correlation is Not a Number? Occasionally, I get a real number for that, but it is within .02 to .06. Secondly, is my data being messed up the reason that none of the other stats will change? I twist and tweak the hyperparameters of my model, but everything besides the PC stat stays EXACTLY the same. What can I do to fix this? Thanks for the help in advance.


texts: Using standard ML models for modeling a derivative when the data set only contains function values

texts:
I want to model a process that is a function of time, $X(t)$. I have a data set which corresponds to a coarse sampling of the function values $X$ at different time points, $t_1, t_2, ...$. I have a analytic-derived model for the derivative $\frac{dX}{dt}$ = $F(X(t),G(t))$ which has a couple tunable parameters in the (known) functions $F$ and $G$. I've been able to tune these models with the following process:

Chose a set of parameters for the functions $F$ and $G$.
Do a forward-Euler method from the earliest time in my data set to the ending point in my data set, generating the $X(t)$ which corresponds to the above parameters in $F$ and $G$.
Use this $X(t)$, along with my measured data points to generate a model error.
Use the above modeled error to go back to (1), iterating and doing an optimization over the parameters in the expression for $\frac{dX}{dt}$ to minimize the error.

Note that in the above I have to deal with the function itself and not the derivative because my measured data is on a coarser time grid than the derivative needs to be computed. For example, my derivative changes meaningfully on a scale of $\Delta t$, but my data set might only have points every $5\Delta t$ to $10 \Delta t$. I cannot take the function values themselves from the data set and create a derivative myself because of this discrepancy in the time scales.

Now this process works pretty well, and I'm able to achieve a pretty good error. However, I'm wondering if there's a better way to model the derivative $\frac{dX}{dt}$ than this analytic model. For example, would a neural network or a regression tree be able to do a better job?

My question is, how would I do this in a computationally efficient way? Let's take a neural network as an example. I'm used to having inputs and their associated outputs when training a NN with standard packages. But the problem is, if I want to model the derivative, I don't have the derivative as an output to train on. I only know the function values and have only been able to compute a error to minimize on through the forward Euler integration. Of course I could treat the NN parameters as I did with the parameters above, and do a brute force minimization on them, but I obviously lose a lot of the efficiency of backprop in doing so (or, if I was doing this with a decision tree, would lose the logic of how regression trees are normally optimized).

Is there a better way for me to do this? Are there methods that exist to handle this problem? If anyone could point me in a fruitful direction it would be much appreciated.


texts: Why doesn't AlphaGo expect its opponent to play the best possible moves?

texts:
In the game won by Lee Sedol, AlphaGo was apparently surprised by a brilliant and unexpected move from Lee Sedol.
After analysing the logs, the Deep Mind CEO said that AlphaGo had evaluated a 1/10000 probability for that specific move to be played by Lee Sedol.

What I don't understand here is : whatever the probability for a good move to be played, why taking the risk ? Why not instead expecting the opponent to always plays the best moves ?

Of course it's always possible that you miss the best move your opponent could play when playing Monte Carlo to evaluate his possibilities, but here it seems that the move was founded. If AlphaGo knew that its strategy could be counter by such a move, why not choosing another strategy, where the worst case scenario would be less "worse".


texts: Is there a difference if input nodes have discrete or range value

texts:
If you have some input nodes containing fruit values like {apple=0, pear=1, oranges=2} vs temperature values like {5, 10, 30, 50}.
Is there any difference on how you set up the neural network to learn the output?
I'm guessing on the first case the neural network only learn about the input you used to train. For example if you then try with banana the result will be something random.
But in the case of temperature if I input 35 I would expect a result similar to 30.
Also, I'm guessing for the first case would need much more data to learn than the second case. So wondering if have to take different consideration on each case.


texts:
I'm looking for a Python library that helps me identify the similarity between two words or sentences.

I will be doing Audio to Text conversion which will result in an English dictionary or non dictionary word(s) ( This could be a Person or Company name) After that, I need to compare it to a known word or words.

Example:

1) Text to audio result: Thanks for calling America Expansion
will be compared to American Express.

Both sentences are somehow similar but not the same.

Looks like I may need to look into how many chars they share. Any ideas will be great. Looks a functionality like Google search "did you mean" feature.


texts: Integration of Sentiment analysis in CRM

texts:
What is the process for integrating sentiment analysis in a CRM? What I am searching for is a system which analyzes the customer comments or reviews using the CRM and finds out the customer sentiment on the services provided by the system or company or a product.

I have done a sentiment analyzer which takes text and shows the sentiment of the text. Now I want to integrate the above-mentioned sentiment analyzer to a CRM, how can I do that?


texts: Working with multimodal dataset, does sufficient audio data means sufficient visual data?

texts: I am working with a multimodal dataset, it's a video dataset with a total video 10400. For the audio data I have it turned into a log melspectrogram and the spectrogram have the following shape (timesteps, num_features) = (190, 96) and then I used the resnet18 architecture and it worked. Now for the visual data, it has the following shape (num_frames, height, width) = (75, 100, 50) however when I tried using resnet18 on it the validation loss was stuck while the training loss keeps going down, so I guess it must have overfit? The same thing happened even when I tried with initial number of filters of 1. And I have tried doubling the amount of data with horizontal flip yet the same thing still happens. Does resnet18 really is too powerful for the task? But if it does why does it work with the audio and not the visual?


texts: Questions about adding metadata to a CNN using keras

texts:
I have a convolutional neural network and would like to include some metadata. My metadata is in a multiple csv files that correspond to each class and it contains a bunch of geometric properties (about 8 numerical measurements), specifically revolving around size and volume that would help classification of similar looking images but have varying size and volume. I am currently using Keras to build my models.
What I am unsure of is where and how to add metadata into my model? Do I need to combine all csv's, do I keep them separate similar to to training images file folders or something else?
What happens to the metadata when the I shuffle the images?
What happens to the metadata when I augment the images?


texts: How to deal with a modular set of features? Input data from intermittent sensors

texts:
I want to set up a ML problem with input data from sensors. The issue is that the sensors are not active all the time, and the standard methods to deal with missing values (delete row, impute, predict…) do not apply, because it is not exactly missing data.
All the sensors provide the same variables, which I use as input features. So my input features are of the form "Sensor N Feature M", and my columns are (S1F1, S1F2, …, S2F1, …, SnFm). The rows are labelled by the different logging times.
Now the issue is that, for any row, only some subset of the sensors is available. This is just how the system is supposed to work: there is something moving, and sensors come and go when they are visible. If I remove the rows with NA, I am left with nothing, because all the sensors are not visible at some point.
I know that this is a very imprecise question, but what I need is some "modular" way of dealing with the input features, a way to set up a "modular" ML problem that takes all the available features for every time (row). Are there techniques to deal with this kind of scenario? I do not even know how to look for this in the literature. Any keywords or references pointing in the right direction would be very helplful.


texts: How to use a Multinomial Naive Bayes Classifier on different sets of data?

texts:
I am working on a sentiment analysis project involving tweets. I used a Kaggle dataset to train my model for sentiment analysis and want to use that trained model to predict the sentiment on an entirely separate group of tweets that are not included in the dataset.
The code is as follows (aside from some cleaning steps):
tweets = pd.read_csv("tweets.csv")
sentiment = pd.read_csv("sentimenttraining.csv")

#counter
def sentiment_counting(input):
    counter = CountVectorizer(strip_accents='unicode', stop_words=['and','the', 'but'])
    counts = counter.fit_transform(input)
    return counts

counts = sentiment_counting(sentiment["text"])

text_train, text_test, sentiment_train, sentiment_test = train_test_split(counts, sentiment["sentiment"], random_state = 100)

#sentiment
sentiment_model = MultinomialNB(alpha = 0.9)
sentiment_model.fit(text_train, sentiment_train)

score = sentiment_model.score(text_test, sentiment_test)
print(score)

tweet_counts = sentiment_counting(tweets["text"])

predictions = sentiment_model.predict(tweet_counts)

However, in the final line (asking for predictions) I get an error--"dimension mismatch"--because the dimension is entirely different. Of course the dimensions are different--this is a different set of tweets. How can I fix this problem?


texts: How to calculate the elapsed time of a flag status per day?

texts:
I'd like to figure out the elapsed time between flag status changes.

Simplified example: a person can only be sad or happy. I'd like to know how long each mood was active until it changed.

I'm using common storage and a script to loop the records, which is very expensive. I'm looking for a general best practice. I'm open to use any technology in combination or as a replacement. I'm new to data science and probably miss the right terms. Related advices are very appreciated.

Index: user_mood_log

 |    changed_at    |  mood  | user |
 ------------------------------
 | yyyy-mm-dd 12:00 | sad    | John |
 | yyyy-mm-dd 15:00 | happy  | John |
 | yyyy-mm-dd 18:00 | happy  | John |
 |        ...       |   ...  |  ... |

Index: user_mood_log_daily

 |    date    |  sad  | happy | user |
 -------------------------------------
 | 2019-05-22 |   3   |   21  | John |
 | 2019-05-21 |   5   |   19  | John |
 |    ...     |  ...  |   ... |  ... |

Challenges:

The last record of yesterday (or even older) needs to be considered. Means I can't simply group by day.
A mood might not change for several days.
Duplicated mood logs are possible.

Any advice? Thanks in advance!


texts: How does fusing operations lower accuracy for machine learning models?

texts: In this talk the speaker Sachin Joglekar mentions that it's important to consider tradeoffs when choosing delegates for optimizing Tensorflow Lite. One of the tradeoffs he mentions at 10:14 is that accuracy is lower when many operations are fused. Why would this be the case?


texts:
I have a question regarding CNN.
I do understand how they work on the surface. Very simply put, they are D-NN for images.
I'll use this example as a reference for this question. In the example, they are not initializing weights and biases anywhere. They are using tf.layers.conv2d() function in the example.
Let's focus only on 2 of the function's arguments:

filters: specify the number of filters to apply
kernel_size: specifies the size of each filter

Questions:

We are not defining these filters. Does tensorflow define them for us or in general if I were to use this for a different set of images how does this work?

Does this filter include both weights and biases? If not, then what exactly does the GradientDescentOptimizer() defined in the above example update after each training step?

I understood the code and understand how the entire process works. I also understand convolution and how it works. But, I'm trying to apply these concepts and implement my own code using tensorflow and CNN and I'm kind of stuck here.


texts: Error using imread function in matlab code

texts:
I was doing some sort of speech recognition pattern for recognition of sound but I stuck at imread error in matlab, any advice what to change in picture below you can find code and error.

Code

Error

x=audioread ('one.wav');
x1=audioread ('two.wav');
x2=audioread ('three.wav');
x3=audioread ('four.wav');

figure, plot (x);
imwrite (x, 'one.jpg');
figure, plot (x1);
imwrite (x1, 'two.jpg');
figure, plot (x2);
imwrite (x2, 'three.jpg');
figure, plot (x3);
imwrite (x3, 'four.jpg');

AKA = imread ('one.jpg');
AKB = imread ('two.jpg');
AKC = imread ('three.jpg');
AKD = imread ('four.jpg');

AA = rgb2grey(AKA);
AB = rgb2grey(AKB);
AC = rgb2grey(AKC);
AD = rgb2grey(AKD);

aa = imcrop (AA,[200,66,50,300]);
ab = imcrop (AB,[200,66,50,300]);
ac = imcrop (AC,[200,66,50,300]);
ad = imcrop (AD,[200,66,50,300]);

figure, imshow (aa);
figure, imshow (ab);
figure, imshow (ac);
figure, imshow (ad);

p = [aa(75:15:210,:);ab(75:15:210,:);ac(75:15:210,:);ad(75:15:210,:)];
E = double (p');
T = [10 10 10 10 10 10 10 10 10 10 ...
 20 20 20 20 20 20 20 20 20 20 ...
 30 30 30 30 30 30 30 30 30 30 ...
 40 40 40 40 40 40 40 40 40 40 ];


texts: How do I implement my loss function in Keras/Tensorflow, when it seems to have different parameters to the default ones?

texts:
So, I'm a university student studying Data Science, and after my previous question about TensorFlow got literally zero answers on Stack Overflow, I figured I'd post this one here instead.
I need to construct a Siamese network for classifying whether or not two characters are members of the same alphabet. Looking online, I've found this tutorial on Towards Data Science, but it uses a default loss function, while I need to code my own.
The problem is, the loss function I was instructed to write and use seems to have a different set of parameters than the ones Tensorflow expects. In essence, I need to calculate a function with the parameters of the two image vector representations generated by the neural networks, a margin, and a parameter that indicates whether or not they're members of the same alphabet. If they're members of the same alphabet, I calculate the distance between the vectors, square it, then halve it. If they're members of different alphabets, I take the maximum of 0 and the margin minus the distance between the vectors, then square it and halve it.
However, Tensorflow seems to expect loss functions to have two parameters, lists of the true values and the predicted values, each prepended with the value of the batch size. This doesn't seem compatible with the contrastive loss function that I'm required to use. How can I manage to get Tensorflow to use my contrastive loss function instead of one of the default ones?
I really need an answer as soon as possible, because the assignment is due by Sunday.


texts:
I am using GTZAN dataset to make a CNN and classify by musical genres.
I'm getting very good results except Val. loss (See Image)

I am processing the audio files using Librosa, obtaining the spectogram and then using the power_to_db function.
This is my CNN Model:
class CNNModel(object):

def __init__(self, config, X):
    self.filters = 32 # number of convolutional filters to use
    self.pool_size = (2, 2)  # size of pooling area for max pooling
    self.kernel_size = (3, 3)  # convolution kernel size
    self.nb_layers = 4
    self.input_shape = (128, 625, 1) # cambiar por x.shape

def build_model(self, nb_classes):

    model = Sequential()
    model.add(
            Conv2D(
                self.filters,
                self.kernel_size,
                padding ='same',
                input_shape = self.input_shape))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))

    model.add(
            Conv2D(
                self.filters,
                self.kernel_size))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size = self.pool_size))
    model.add(Dropout(0.25))

    model.add(
        Conv2D(
            self.filters + 32,
            self.kernel_size,
            padding ='same'))
    model.add(Activation('relu'))

    model.add(
        Conv2D(
            self.filters + 32,
            self.kernel_size,
            padding ='same'))
    model.add(MaxPooling2D(pool_size = self.pool_size))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(nb_classes))
    model.add(Activation("softmax")) #mirar

    return model

I leave the link of my github in case you want to see the whole code.
Every song is (128, 625) Shape, I used MinMaxScale to Scale the data.
This is my loss function and my optimizer
loss = losses.categorical_crossentropy,
optimizer = optimizers.SGD(lr=0.001, momentum=0, decay=1e-5, nesterov=True)

I have read about overfitting and that seems to be the cause but I do not know how to solve it at the code level.
Update 1: With Dropout(0.9) I get this results:

Thank you


texts: What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

texts: I am analyzing a portfolio of about 225 stocks and have gotten data for each of them based on their "Price/Earnings ratio", "Return on Assets", and "Earnings per share growth". I would like to cluster these stocks based on their attributes into 3 or 4 groups. However, there are substantial outliers in the data set. Instead of removing them altogether I would like to keep them in. What ML algorithm would be best suited for this? I have been told that K Means would not work so well since the outliers would skew the centroids of a particular cluster. Any and all thoughts welcome!


texts:
I am using Keras library to build a CNN model. I want to use data augmentation for training data.

Should I use rescale parameters for data augmentation?

ImageDataGenerator(rescale=1./255)


texts:
I was reading a paper on traffic flow optimization using Multi-Agent Q learning. the paper proposes the following method:
Deploy a Reinforcement learning controller at each intersection with traffic lights.

first the Q value equation is:

$Q^{t}(s,a) = (1- \alpha )Q^{t-1}(s,a) +  \alpha (R{t} +  \gamma max_{a}(Q^{t-1}(s,a))$

second the state is:
the sum of vehicle queues lengths at the current intersection and one hop intersections

third the action space is:

here the actions represent the possible moves of vehicles at the intersection.

forth the reward at time t is

$R^{t} = -( w_1\sum q_{current intersection} + w_2\sum q_{neighbors} )$

where the q refers to vehicle queues lengths, w1 and w2 are constants.

fifth there is the algorithm in the image below where it acquires the action required for maximizing the Q value

What I am trying to understand is, the reward calculation does not take an action as a parameter. How does it choose an action properly.

I am a newbie to reinforcement learning, so please if you find my question naive, cosedire referring me to a proper textbook.
Thanks


texts: Distplot 'a' parameter, does not make logical sense to me, can anyone explain?

texts:
Most sns plot parameters have names that make somewhat logical sense,
For example scatterplot has the x attribute to choose what to plot on the x-axis, and y for y-axis. Similarly kdeplot uses the parameter data.
But as far as I can see, distplot uses the parameter 'a' and only that to specify what data to plot (I tried substituting 'a' for 'data' but that gave me an error)
Could someone tell my why was this choice made for the parameter name, as this seems very illogical from where I stand, and has forced me to correct my code more than a few times.


texts: How to Obtain Output of Intermediate Model in Keras

texts:
I'm creating a neural architecture using the functional API as follows:

x2 = layer1(x1, name='layer1')
x3 = layer2(x2, name='layer2')
m1 = Model(x1, x3)
x5 = layer3(x4, name='layer3')
x6 = TimeDistributed(m1)(x5)
m2 = Model(x5, x4)

I've trained m2 (with m1 getting implicitly trained). I can access the output of the layers of m2 (say layer3) by creating an intermediate model:

layer_name = 'layer3'
intermediate_layer_model = Model(input=model.input, output=model.get_layer(layer_name).output)
intermediate_output = intermediate_layer_model.predict(input)

Since m1 is a nested model, I'm not able to access the layers of m1 through m2. How do I obtain the output x3 (layer2's output) when you use the trained m2 to predict on a new data point.


texts: What happens if you add a constant value to all input data points to neural networks?

texts:
I have a somewhat basic question about neural networks. What would be the effect on the performance of a neural network if we add a constant value to all data points? For example, suppose you have temperature data in degree Celcius which is not an SI unit. Will it make any difference to the model if your input is in degree Celcius or Kelvin? If yes, then in what way?

I have a dataset where one feature values are in nanometers for example 35nm, 42nm, etc. If the answer to the above question is No then you can simply divide all of them by 100 to reduce them between 0 and 1 and it shouldn't matter to the model? Does it make more sense to divide it by 100 or the maximum value? What do you do if your input is limited by neither 100 nor the maximum value?

Thanks for your help in advance.


texts: Does recall has different interpretation when comes to classification and information retrieval

texts:
Recall Definition

In terms of classification

  The recall is defined as no of positive instances that are correctly detected by the classifier.

$$
TP = \frac{TP}{(TP+FN)}
$$

In terms of Information retrieval

  Recall is defined as No of relevant documents that are retrieved.

 Let p = No of relevant documents that are retrieved   and
     q = Total number of relevant documents

 Formula  = p/q

Do both the definition work on any common understanding?. I still can't get my head through this. If both are different, why are they using the same name recall?


texts: Predicting which apps users may be interested in

texts:
I am building a mobile app that can predict what apps users may be interested in downloading from the play store, based on what apps the user has already installed on their device and how much time they have spent on these apps. Also, there is the option to scroll through the top apps in the play store and you can "favourite" any apps that you find interesting/want to download.

Based on this data, I would like to make predictions and notify users of any new popular apps that are released in the future, however since the predictions will be user specific, I am unsure if the dataset will be too small for k-means clustering (Group the apps into genres and find most popular genres). 35 is the average number of apps installed on user's smartphones, and if you include any "favourited apps" the total dataset could be around 50.

Perhaps there is another more suitable technique I could apply to make accurate predictions but I am unsure where to go at the moment.


texts: Determining threshold in an area with very few samples of positive label

texts:
I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding the wrong sample carries a heavy penalty. I have concluded I want to optimize something like the following:

n_discards - n_false_discards * penalty

Where I expect penalty to be around 5000.

Now, it's easy enough to look at my validation data (around 100k samples) and look at where this value is maximized. However, the threshold that this returns is quite unstable. The high penalty means my threshold should be in an area with very few false discards, but the absence of these makes it very difficult for me to accurately estimate where the threshold should be, exactly. This leads to a lot of threshold variance between folds, and more importantly, between what the validation would suggest and what's right for the test set.

I feel like there must be a better way than going on a sample-by-sample basis, but I haven't found the right way. I've considered trying to fit distributions from scipy.stats and using weighted cdf's to optimize the above function, but I haven't been able to get these fits to work very well. I've looked into sklearn.isotonic, and I've tried some heuristics myself (linear probability-based interpolation in between the false discards to approximate expected number of false discards at that probability). So far I'm not really satisfied with any outcome.

In case it's relevant: the model I use is xgboost.

I feel like this should be a rather common dilemma, but I haven't been able to find any robust methodology to deal with this. What is a good approach for dealing with this issue?


texts: Export dataset with predicted target - Python

texts:
I've this code (part of predictive model):

training_features, test_features, training_target, test_target, = train_test_split(df.drop(['target'], axis=1),
                                              df['target'],
                                              test_size = .3,
                                              random_state=12)
    coef_gini = DecisionTreeClassifier(criterion = "gini", random_state = 100, max_depth = 3, min_samples_leaf = 5)
    coef_gini.fit(training_features, training_target)

    y_pred = coef_gini.predict(test_features)
    y_pred

How can I export my dataset with the predictive value in order to create a confusion matrix to determine the False Positive value?

Thanks!


texts: Seeking advice to conceptualize the completed machine learning product and leave the Jupyter Notebook behind

texts:
I have a well functioning sports betting model that is my obsession, but it exists only inside a jupyter notebook. I have a long ETL function that pulls and arranges the data from two separate APIs, and prepares them for the model which I run daily.
I'm seeking advice on which technologies to use for the database (I'm currently just loading, adding to, and re-saving CSV files through pandas) , how that connects to my ETL function, and which cloud service to prop it up on.
The end goal is a fully automated model for which I can create a simple UI (Dash or Flask?) that spits out daily predictions and displays historical results.
Thanks so much.


texts: Is there a term for measuring error on a second prediction based on the first's?

texts:
I have created a dataset which contains six values per row which may be the target value. Two rows for example:

1   1   1   1   0   7742    8901    7505    8714    8263    -1  29748   9394    60
1   -1  1   0   0   5658    6817    5420    -1  6178    7309    31833   6630    65

Where columns 6 through 11 of the 14 may be the target with its value shifted into the target column and its value replaced with -1.

The aim of this isn't to generate a great all round model but a model which when it makes a prediction it's maximum error is within a (beyond a reasonable doubt) range, not probably accurate.

I figure the approach to this is to skip initial prediction error and make a set of secondary predictions where it is assumed the initial predictions are accurate. The second set of predictions is then compared to the known value and only if its error is within a certain amount, in my case 0.03%, then track the absolute maximum error.

The final error reported in the absolute maximum error from the second round of predictions without ever questioning the quality of the initial predictions directly.

The end result so far has been models which where an initial prediction becomes the target and the second prediction results in an error of less than 0.03% then the first prediction has an absolute error of less than 3%. Roughly 10% of records survive.

Is there a term that describes this process?

Thanks


texts: Does a post-doc in Mathematics help getting a Data Science job?

texts:
I am a graduate student in Mathematics, and I have been recently offered a post-doc at a good university. While it is not a top school in the most strict sense of the word, it is one of the best positions in my field of research.

However, I have recently started wondering whether academia is really the right choice for me, and I am considering learning data science with the intention of transitioning towards industry.

The question is, assuming I was to take that step and decide to start studying towards leaving academia, would it be better for me to take the post-doc, or perhaps stay one more year in graduate school?

The obvious advantage of taking the post-doc is that it would give me ample time of financial stability to learn new things. However, I am mostly interested in an answer from the point of view of starting a career in data science: is a post-doc in Mathematics something that would be considered a worthwhile addition to one's resume, or does that perhaps make one look a little bit too "academic" and actually impede one's ability to get an industry job?

The advantage of staying for one more year in graduate school would be perhaps be that since it is a top American program, I would have access to plenty of resources that I am not sure would be available at a good European university. Moreover, it seems to me that there are more good opporunities in data science in the US rather than abroad. Thus, if my goal was to get a job here, I wonder if it wouldn't be easier to just not leave the US in the first place..? I am not a US citizen, and so work authorization is something I have to consider.


texts: does accuracy and performance depends on whole dataset or fraction of that data mimics the whole?

texts:
MNIST dataset is handwritten images which contains 60000 training and 10000 testing samples , can I use only a fraction of dataset for instance 10000 for training and 4000 for testing. I want to know that whether my results will be comparable to other claimed results. Like I used random forest classifier on 10k training and 4k testing and got accuracy of 93 %. would this result be equivalent when I use 60k training and 10k testing .

and in some published papers I read that they only used 40 images per class and evaluated their accuracy and performance using them does it conforms to the standards that use 60k as training and 10k as testing.
Or we can say that the choice of selection of training and testing data is dependent on the method used to classify


texts: Updating weights python for REINFORCE policy gradient method

texts:
All,

I am trying to implement REINFORCE(williams) algorithm. This is a policy gradient reinforcement learning algorithm.

I am using python, and hope to use keras.

The pseduocode I am using is as follows:

from the initial state, run a simulation of the policy, till end.
the actions taken . are generated from (in my scenario) a NN.
We update the weights by: learning rate * derivative(log policy) * V
We are using V as an unbiased sample of Q.

V is defined as the average reward being at a particular state.

In keras, we can build a neural network, and then update the weights by taking weights + (log of actions * V)? Is this correct?

How would I manually update the weights, I appreciate that you can extract the weights from keras sequential models, but there is also a derivative function.

Maybe I have misunderstood the algorithm, any thoughts would be greatly appreciated.


texts: The Difference between One Hot Encoding and LabelEncoder?

texts:
I am working on a ML problem to predict house prices and Zip Code is one feature which will be useful. I am also trying to use Random Forest Regressor to predict the log of the price.

However, should I use One Hot Encoding or Label Encoder for Zip Code? Because I have about 2000 Zip Codes in my dataset and performing One Hot Encoding will expand the columns significantly.

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?


texts: On which data evaluate an object detection model ? (similar or real life data ?)

texts:
I'm training an object detection model (SSD300) to detect and classify body poses in thermal images.

Even I have more than 2k different poses, the background does not change much (I have only 5 different points of view).

I trained my model on these images (70% for the training and 30% for validation).

Now, I want to evaluate the model on an unbiased dataset.

Should I keep images of my dataset for this purpose or should I use a real life dataset ?

(A good solution would be to have a real life training set, but I don't have)

I tried both, but as expected, I have an mAP=0.9 when evaluated on similar pictures and mAP=0.5 when evaluated on completely different images.

Bonus question: is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")

Precision-Recall ? (but I have to choose a pose classification threshold...)


texts: Error using twitter R package's userTimlien

texts:
I am using twitteR package to retrievie timeline data. My request looks as follows:

tweets <- try(userTimeline(user , n=50),silent=TRUE)

and this worked quite well for a time, but now I receive this error message:

Error in function (type, msg, asError = TRUE)  : easy handle already used in multi handle

In a related question on Stackoverflow one answer is to use Rcurl directly but this does not seem to work with twitteR package. Anybody got an idea on this?


texts: How do I handle with my Keras CNN overfitting

texts:
In my CNN, I have 700 images of class 0, 700 images of class 1, and 72 validation images.

My code:

visible = Input(shape=(256,256,3))
conv1 = Conv2D(16, kernel_size=(3,3), activation='relu', strides=(1, 1))(visible)
conv2 = Conv2D(32, kernel_size=(3,3), activation='relu', strides=(1, 1))(conv1)
bat1 = BatchNormalization()(conv2)
conv3 = ZeroPadding2D(padding=(1, 1))(bat1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv3)
drop1 = Dropout(0.30)(pool1)

conv4 = Conv2D(32, kernel_size=(3,3), activation='relu', padding='valid', kernel_regularizer=regularizers.l2(0.01))(drop1)
conv5 = Conv2D(64, kernel_size=(3,3), activation='relu', padding='valid', kernel_regularizer=regularizers.l2(0.01))(conv4)
bat2 = BatchNormalization()(conv5)
pool2 = MaxPooling2D(pool_size=(1, 1))(bat2)
drop1 = Dropout(0.30)(pool2)

conv6 = Conv2D(128, kernel_size=(3,3), activation='relu', padding='valid', kernel_regularizer=regularizers.l2(0.01))(pool2)
conv7 = Conv2D(128, kernel_size=(2,2), activation='relu', strides=(1, 1), padding='valid')(conv6)
bat3 = BatchNormalization()(conv7)
pool3 = MaxPooling2D(pool_size=(1, 1))(bat3)
drop1 = Dropout(0.30)(pool3)

flat = Flatten()(pool3)
drop4 = Dropout(0.50)(flat)

output = Dense(1, activation='sigmoid')(drop4)
model = Model(inputs=visible, outputs=output)

opt = optimizers.adam(lr=0.001, decay=0.0)

model.compile(optimizer= opt, loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)
test, lt = ReadImages(TEST_DIR)

data = np.array(data)
labels = np.array(labels)

perm = np.random.permutation(len(data))
data = data[perm]
labels = labels[perm]
#model.fit(data, labels, epochs=8, validation_data = (np.array(test), np.array(lt)))

aug = ImageDataGenerator(rotation_range=20, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,
    horizontal_flip=True)

# train the network
model.fit_generator(aug.flow(data, labels, batch_size=32),
    validation_data=(np.array(test), np.array(lt)), steps_per_epoch=len(data) // 32,
    epochs=7)

model.save('model.h5')

It returns these numbers:

Epoch 1/7
43/43 [==============================] - 1004s 23s/step - loss: 1.8090 - acc: 0.9724 - val_loss: 1.7871 - val_acc: 0.9861
Epoch 2/7
43/43 [==============================] - 1003s 23s/step - loss: 1.8449 - acc: 0.9801 - val_loss: 1.4828 - val_acc: 1.0000
Epoch 3/7
43/43 [==============================] - 1092s 25s/step - loss: 1.5704 - acc: 0.9920 - val_loss: 1.3985 - val_acc: 1.0000
Epoch 4/7
43/43 [==============================] - 1062s 25s/step - loss: 1.5219 - acc: 0.9898 - val_loss: 1.3167 - val_acc: 1.0000
Epoch 5/7
43/43 [==============================] - 990s 23s/step - loss: 2.5744 - acc: 0.9222 - val_loss: 2.9347 - val_acc: 0.9028
Epoch 6/7
43/43 [==============================] - 983s 23s/step - loss: 1.6053 - acc: 0.9840 - val_loss: 1.3299 - val_acc: 1.0000
Epoch 7/7
43/43 [==============================] - 974s 23s/step - loss: 1.6180 - acc: 0.9801 - val_loss: 1.5181 - val_acc: 0.9861

When I predict some test images, the result is always 0.

I already tried various things like adding more dropouts (or making the dropout rate bigger), data augmentation, batch normalization etc. and none of these have made it work properly.

What should I do?


texts: XGBoost for multi-label image classification

texts:
I am trying to use the xgboost classifier for a multi-label and multi-class image classification task. I have a list of images that can have up to 5 different labels in each of them. Before I use the classifier I want to also apply image augmentation.

import keras
from sklearn.model_selection import train_test_split
from keras.preprocessing.image import ImageDataGenerator
from xgboost.sklearn import XGBClassifier

train_idx, val_idx = train_test_split(mask_df.index,  test_size=0.2,random_state=28)

train_datagen=ImageDataGenerator(zoom_range=0.1,
                              fill_mode='constant',
                              rotation_range=10,
                              height_shift_range=0.1,
                              width_shift_range=0.1,
                              horizontal_flip=True,
                              vertical_flip=True,
                              rescale=1/255.)

train_generator=train_datagen.flow_from_dataframe(
                dataframe=mask_df.loc[train_idx],
                directory="home/DATA/train_images/",
                x_col="ImageId",
                y_col=columns,
                color_mode='grayscale',
                batch_size=32,
                seed=32,
                shuffle=True,
                class_mode="other",
                target_size=(100,100))

model = XGBClassifier()
history=model.fit_generator(generator=train_generator,
                steps_per_epoch=100,
                validation_data=validation_generator,
                validation_steps=100,
                epochs=5)

The last command gives me an error:

AttributeError                            Traceback (most recent call last)
<ipython-input-8-8c4c0504d559> in <module>
----> 1 history=model.fit_generator(generator=train_generator,
      2                     steps_per_epoch=100,
      3                     validation_data=validation_generator,
      4                     validation_steps=100,
      5                     epochs=5

AttributeError: 'XGBClassifier' object has no attribute 'fit_generator'

Does anyone have any advice on how to proceed since I can not use the fit_generator?


texts: Running two stage classification to predict relatively rare event?

texts:
I have a very imbalanced sample in which I am trying to predict probability of a rare event (Out of around 25,000 observations, this event is observed around 30 times) and am reluctant to try over/under-sampling on that directly just because of the degree of imbalance.

Just to illustrate what I had in mind with an example: Let's say I'm trying to classify if a gem is an emerald.

But out of 25,000 samples, I only observe 30 emeralds. However, I have some other green stones like Jade and Peridot which bring up my observations to 300.

Would it be a good idea to determine P(Green Stone) and then P(Emerald | Green Stone) by running a two stage classification.

The latter stage will have imbalance ratio of 1 emerald : 9 non-emeralds, which might be more suitable for balancing.

Would appreciate any thoughts/insights on pursuing this idea.


texts: Exploratory Data Analysis on dataset divided by winners and losers

texts:
I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players:
winner_age, winner_rank / loser_age, loser_rank

In order to conduct a proper EDA, do I have to concatenate both losers and winner to further analysis or split the data frame in two?


texts: On which algorithm for boosting is the method xgbLinear of the xgboost/caret- package based on?

texts: In caret package of R, there is a method 'xgblinear'. What is the working algorithm behind this method.


texts: Would it be okay to stop training my neural network?

texts:
When the validation error of my Neural Network that I am trying to train is slowly decreasing but not by much, is it okay to stop train the network at that point, or do I need to increase the training time until the minimum validation error is reached?

For instance, in the last 5 epochs my validations errors are shown below:

| end of epoch   1 | time: 3782.50s | valid loss 6.7914 | valid ppl 890.1194
| end of epoch   2 | time: 3802.14s | valid loss 6.6084 | valid ppl 741.2616
| end of epoch   3 | time: 3791.33s | valid loss 6.5249 | valid ppl 681.8797
| end of epoch   4 | time: 3792.55s | valid loss 6.4513 | valid ppl 633.5318
| end of epoch   5 | time: 3804.15s | valid loss 6.3884 | valid ppl 594.8927

so like between the 4th epoch and the 5th epoch, the loss decreased by ~0.975% (= (6.4513-6.3884)/6.4513 * 100)? would it be okay to stop training the network at this point?

Thank you,


texts: Identifying common keyphrase frequency in large dataset

texts:
I have a dataset of profiles that contain freeform text describing the work history of a number of individuals.
I would like to attempt to identify frequently used words or groups of words across the set of profiles in order that I can build a taxonomy (of skills) related to the profiles.
For example, if the words 'conversion rate optimisation' appear together 300 times across all profiles, I would see this on my list as a high-frequency keyphrase. I would expect to be able to filter the list based on single keywords, 2 words, and 3-word strings.
I would then be able to manually pick out frequently used keyphrases relating to skills, that could be added to a master taxonomy list.
I would also need some way of filtering out invalid words like ('I', 'and' etc)
What is the best way to get something like this done?


texts: XGB custom objective function - small change to default regression squared error objective function

texts: Where can I find the code for the default squared error objective function? I just want to make a small change to re-weight certain datapoints?


texts: Measuring impact of missing feature on model performance

texts:
Is there any general approach to approximate how a missing feature will impact the prediction performance of a regression model?

For example, if I train a model using 10 features, but want to make predictions afterwards only providing 8 of those 10 features - simply imputing the missing values -, how does that (statistically) affect my target metric. You could also think of it like some kind of confidence decline of the predictions.

I wondered if there is some approach like deriving / estimating the deterioration of performance from feature importances, or similar. I did not find any resources about that though, nor could I come up with some meaningful idea myself so far.


texts: Keras - add_weight() method not adding to total model parameters

texts:
I am creating a custom Keras layer FConv2D(), and adding a weight in its build() function using the add_weight() method as suggested in official Keras tutorial for creating custom layers.
def build(self, input_shape):
      shape = tf.TensorShape(input_shape).as_list()
      h = shape[1]
      w = shape[2]
      in_channels = shape[3]
      self.kernel = self.add_weight(
            shape=(h,w,in_channels,self.num_outputs),
            initializer="random_normal",
            trainable=True,
        )
      super(FConv2D, self).build(input_shape)

But when I print the summary of a single-layer model containing just this layer, the number of parameters in this layer is shown to be 0.
Model: "model_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_5 (InputLayer)         [(None, 7, 7, 2)]         0
_________________________________________________________________
f_conv2d_4 (FConv2D)         (None, 7, 7, 64)          0
=================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0

I tried the same method in the tutorial for custom layers on the official website, but their add_weight method seems to work right -
class SimpleDense(tf.keras.layers.Layer):

  def __init__(self, units=32):
      super(SimpleDense, self).__init__()
      self.units = units

  def build(self, input_shape):  # Create the state of the layer (weights)
    self.w = self.add_weight(shape=(input_shape[-1], self.units),
                               initializer='random_normal',
                               trainable=True)
    self.b = self.add_weight(shape=(self.units,),
                               initializer='random_normal',
                               trainable=True)

  def call(self, inputs):  # Defines the computation from inputs to outputs
      return tf.matmul(inputs, self.w) + self.b

input = tf.keras.layers.Input(shape = (1000,1))
output = SimpleDense(100)(input)
model = tf.keras.Model(inputs = [input], outputs = [output])
model.summary()

Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_2 (InputLayer)         [(None, 1000, 1)]         0
_________________________________________________________________
simple_dense_3 (SimpleDense) (None, 1000, 100)         200
=================================================================
Total params: 200
Trainable params: 200
Non-trainable params: 0

Could anyone please tell why the weights added in the custom layer are not showing in the model parameters?


texts: Use forecast weather data or actual weather data for prediction?

texts:
I'm planning to use weather data to predict crime in Chicago.
I'm confused which to use as features: historical weather forecast data or historical actual weather data?
And what is the underlying logic here?
I'm assuming later only forecast data is available for prediction. Thank you for any input.


texts:
Looking for language modeling, I have been finding CRF in a lot of places which is but looking online for the same isn't actually helping me a lot. I referred Edwin Chen's blog and Ravish Chawala's Medium article but rather than solving my problem, raises more question.

Could you please refer some interesting blog posts, Github repo's or research papers that are going to further help me with this?


texts: Timeseries of odds in race - how to pick a model

texts:
Being new to AI/ML I'd like some pointers to where to begin.
I got data from horse races. Specifically, I got the odds
for each runner during the race - ca 5 times per second.

t1  r1   r2   r3 ...
1   5.25 2.04 3.25
2   5.10 2.50 2.75
...

I also know if the runner won/placed/lost
My goal is to be able to say that runnerx X will win/place
after ca 50-75% of the expected racetime with say 80% accuracy.

My problem is that I don't know how to model this situation.
I've seem tournament strategies - ie who out of two runner will win -
but here's more data - both in time and in participants

What model should I pay attention to?

/Björn


texts: What knowledge should I gain for developing a supervised image processing software that learns how to edit photos based on past behavior?

texts:
I have done several machine learning projects but all of them have been connected to the traditional machine learning (predictions, classifications, etc.). I have currently been offered a project to finish in less than 6 months.

The idea is to develop/improve a pre-existing software. The software takes the image of a molecule from an advanced molecule and then tries to highlight the cell line with red, some times the software takes the background or the lines of other cells also as the highlighted part, and thus the user has to manually edit and trim such mistakes. The idea is to make the software learn from user's edits and behavior over time.

One thing I want to know is whether such a 6-month project is realistic for someone with no background in image processing and pattern recognition? Or is it going to be terribly difficult because I only have had experience with "data-oriented"/statistical machine learning?

My other question is; what type of concepts/topics should I dig deep into to learn the fundamentals of carrying out this project?


texts: Cost functions penalizing certain types of misclassification more heavily in tensorflow

texts: I have 2 classes of data A and C.   I want to create a NN with 3 output nodes, classifying data as either A, B, or C.   Classifying an A or C correctly should have zero cost (very good).  Classifying an A as a C or vice versa should have high cost (very bad).  However, for samples that are "blurry", I'd like to classify them as B, which would have relatively low cost (better than a misclassification, but still not ideal).   I am new to tensorflow, and am not comfortable with the different ways to define cost functions.   Can anyone point me in the right direction on this one?  Thanks in advance


texts: Output of a network to it's input, keras

texts:
I'm trying to create a neural network in keras for time series forecast. I've build a concept, and now I'm not quite sure if it is possible to implement using keras.
I have a potentially complicated network (there are several layers already), that receives $N$ values as input (i.e. $N$ consecutive values of a time series), and outputs a prediction of the next one.
Then input values are "shifted" and the predicted one is placed alongside with the $N-1$ most recent values.
I've already built a network, but stuck on passing just predicted value to the input.
It should look somehow like below:
 +------------+
 |            |
 V  +----+    |
--->|    |    |  passes it back as an input
    |    |    | (current inputs are shifted)
--->|    |    |
    | NN |----+---> yields the next value
... |    |
    |    |
--->|    |
    +----+

I've tried to use callbacks API, but it it didn't help, because there's no way to call callback on each value produced (except using batch size equal to one, but it produces exploding gradients).
What is the proper way to do such a thing using keras?

Moreover, this "hidden" NN is a bit more complicated, than the sequential one. It consists of $M$ sequential networks, that are concatenated and followed by single LSTM layer. Each of these $M$ "sub-networks" receive all $N$ inputs (all the same ones).
I think, it is important to mention.


texts: clustering with heterogeneous (quantitative and qualitative)data?

texts:
I'm a Phd student and I have the results of some approaches (algorithms) that I would like to analyze.  Data (results) are stored in csv files as follows: - the lines describe each algorithm with its parameters and the result obtained. - Some columns (characteristics)  are quantitative and  others are qualitative (enumerated types) and the last column is the results which presents the efficiency of the algorithm and it is a numerical value that I can convert in classes (of intervals of values).  I would like to highlight  at first what is the tendency of the good and worse algorithmes. Then, what are the features that contributed to obtain these results I'm beginner in machine learning and I searched a lot on the net but i didn't find the appropriate method . Could you point me to a clustering/ learning method for this purpose.
I would be very greatful.
Best regards.


texts: How is determined the context's dimension in Doc2Vec?

texts: I would like to know how is determined the dimension of the context in Gensim Doc2Vec.


texts: Reference about social network data-mining

texts:
I am not in the data science field, but I would like to examine in depth this field and, particularly, I would like to start from the analysis of the social networks data.

I am trying to find some good references, both paper, websites and books, in order to start learning about the topic. Browsing on the internet, one can find a lot of sites, forum, papers about the topic, but I'm not able to discriminate among good and bad readings.

I am an R, Matlab, SAS user and I know a little bit of python language.

Could you suggest any references from which I could start studying and deepen the industry?


texts: Predictive modeling on time series: How far should I look back?

texts:
I am building a model for classification on a dataset which is collected by recording a system's behaviour through 2 years period of time. The model is going to be used in the same system in real time.

Right now I am using the whole dataset (2 years) to build my classifier but I doubt it might not be the right approach. Since I am trying to model the behaviour of a system in real time, it is possible that older data points in the dataset have become irrelevant or uninformative compared to system's current environment (distribution of the system's input changing drastically through time for example).

My question is how can I determine which parts of the dataset to use for training, taking the last 1.5 years instead of all 2 years for example. Is there a statistical way to help me decide that a specific time period is not helping or possibly hurting the model to correctly classify more recent data points?


texts: Handling a regularly increasing feature set

texts:
I'm working on a fraud detection system. In this field, new frauds appear regularly, so that new features have to be added to the model on ongoing basis.

I wonder what is the best way to handle it (from the development process perspective)? Just adding a new feature into the feature vector and re-training the classifier seems to be a naive approach, because too much time will be spent for re-learning of the old features.

I'm thinking along the way of training a classifier for each feature (or a couple of related features), and then combining the results of those classifiers with an overall classifier. Are there any drawbacks of this approach? How can I choose an algorithm for the overall classifier?


texts: Does anybody knows python library to retrieve sentiment from Russian text. The dictionary with sentiment parameterization will be ok to. The idea of library something like in GPOMS in article.


texts:
I am familiar with Bayes nets (discrete/continuous/hybrid). I recently started to learn basics of Pyro and tried to model simple Bayes nets as Pyro programs. I also noticed the simple example answered in Question "Whats the Difference between probabilistic programming such as pyro and Belief networks?".

Can anyone provide a full working Pyro example that shows Bayesian inference for computing any node/variable X posterior distribution P(X|e) in a Bayes net given some set of evidence e? Inference can be exact or approximate. The 3-node example from the mentioned answer would suffice.


texts: What distance should I use for edges weights in textrank algorithm

texts:
I found this python implementation on github with 400+ stars which use levenshtein distance between each nodes.

But original paper (page 4) said:

  Next, all lexical units that pass the syntactic filter
  are added to the graph, and an edge is added between
  those lexical units that co-occur within a window of N word

So the question is: is levenshtein distance legit for this algorithm or better to rewrite with windowed edges?

Intuitively levenshtein distance must not work because it's not denote importance of word...


texts:
I'm trying to project lifetime of customers in my company, based on various parameters
I've reached a 64% correlation so far, between the valid and prediction data
I'm using light GBM regressor
I did a sanity check on my prediction data and calculated the series of errors: Y_valid - pred
The error distribution is a classic normal one!
If I can project the error based on it's distribution, can I normalize (or rather "un-normalize") the prediction, and improve the accuaracy of my prediction ?
Is there a way to set an "error offset" with the regressor in the parameters ?


texts: Clustering algorithm for a distance matrix

texts:
I have a similarity matrix between N objects. For each N objects, I have a measure of how similar they are between each others - 0 being identical (the main diagonal) and increasing values as they get less and less similar. Something like that (actual matrix would be 10 000 x 10 000 if I have say 10 000 elements to cluster together):

[ 0 5 12 7 ]
[ - 0 9  2 ]
[ - - 0  6 ]
[ - - -  0 ]

(Supposing index starting at 0) So here object_0 would be most similar to object_1 (5), but object_1 itself would be more similar to object_3 (2), etc.

I would like to cluster that into k groups, so as to minimize either the overall score or intra-cluster score. I'm not sure if that would practically make that much of a difference. I haven't checked the maths on that, but I feel that there could be cases where minimizing a specific cluster's score wouldn't necessarily minimize the overall score (across all clusters). At the end of the day, even thought rigorously speaking there may be a difference I may not practically speaking care if the results are close enough either way.

A few details:

Clusters don't have to be equal sizes
I don't really mind if the number of clusters is an input or if it is being decided by the algo itself in some fashion (thought I would prefer inputing it)
Typically I would have about 10 000 elements to cluster. Maybe 40 000 if I push it. But then I may need to repeatedly run it for a bunch of matrices that size.

On top of an algo/lib that does that, bonus points for:

Some library that's already implemented it, so I can hopefully just format the data properly for it and just see if it seems to give good results or not before I sink a whole lot of time in it
For such a lib, support for parallel processing
Other nice bells & whistles (like maximum number of iterations, ability to input some stopping criteria, etc.)

Failing that, of course it could also just be some pseudo-code that allows me to cluster them into groups.


texts:
Some time ago I read about the two assumptions of machine learning:

Data input has a correlation with data output.
Enough data is present to find it

However, I don't know, where I read it and I want to cite it in my thesis. Can give me a reference in literature?

Thank you!


texts: How to implement a POS tagger using Word co-occurence and Clustering concepts?

texts:
I am thinking of implementing a POS tagger by myself. a POS tagger, extracts the syntax role of a word in a sentence.
According to my studies, word co-occurrence is a technique to analyze word occurrence which can be used to construct a graph where nodes are words and the weights between them is their co-occurrence weight.
I am wondering if there is a way to apply clustering algorithms on this graph to group words based on their syntax roles? i mean i want to do the same as a POS tagger does. the main idea is that after constructing the graph of co-occurrence, how can i apply a clustering or community detection algorithm on this graph to group nodes based on their roles in a sentence? for example, to group nodes which are verbs, or group nodes that are nouns. in clustering i don't need to know this is verb or not, i only want to group nodes which have a similar syntax in graph. then i will analyze them to find out they are weather a verb or noun or ...
I would be so thankful if you give mind on starting this project.


texts: Why do histogram bars vanish when we keep the bins value high in matplotlib?

texts:
Also, the histogram bar widths are different on certain values of bin.
How to keep the bar widths uniform?
I have tried using the rwidth but that dos not solve my problem.

Data:

test    age
17 - Alpha OH PROGESTERONE - HORMONE ASSAYS 23
17 - Alpha OH PROGESTERONE - HORMONE ASSAYS 26
17 ALPHA HYDROXY PROGESTERONE   18
17 ALPHA HYDROXY PROGESTERONE   21
17 ALPHA HYDROXY PROGESTERONE   25
17 ALPHA HYDROXY PROGESTERONE   27

Code

axes = plt.gca()
axes.set_xlim(0, 100)
axes.set_ylim(0, 100)
plt.hist(df.loc[0:1,'age':'age'].to_numpy(),100, rwidth=0.9)

axes = plt.gca()
axes.set_xlim(0, 100)
axes.set_ylim(0, 100)
plt.hist(df.loc[2:5,'age':'age'].to_numpy(),10, rwidth=0.9)

axes = plt.gca()
axes.set_xlim(0, 100)
axes.set_ylim(0, 100)
plt.hist(df.loc[6:14,'age':'age'].to_numpy(),100, rwidth=0.9)


texts: I have an interesting problem where we have sensor readings in a 3D space (X,Y,Z) and sensor reading value V, I would like to model and machine learn to try and predict the next lag on sensor reading based on historic values. Can you point me to which direction to start from ?


texts: I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize the data before slicing the windows and that doesn't seem right, because every input will rely on an information that exists outside of it (mean and standard deviation). When I'm about to make a prediction, all I have is a single window. If I'm going to normalize it, I have to do it using it's own data, otherwise I'd have to know what were the statistics used when fitting the model, so that my input stays in the same "scale" as the training data, but I don't see people worrying about that. So, in my understand, every window should be normalized after the slicing, using it's own data, and not before, relying on the whole dataset. Is that correct?


texts: choosing model based on last or best iteration on validation set

texts:
This is a very basic question, however I haven't found a satisfying answer until now.
When training a neural network we must choose the number of epochs. The usual advice is to train as long as the loss on the validation set goes down. But it happens that the validation loss goes down, then a little bit up, then down again. So for me the most reliable way to find the point to stop training and get the best model would be to take a very high number of epochs and just choose the best validation loss out there (be it the last iteration or, more probable, something in between). One drawback is of course the long training time due to high number of epochs.
To avoid long training time and still get a good validation score, people use "early stopping" techniques (as described in the often cited paper Early Stopping — But When? from Lutz Prechelt). In the paper mentioned, the author points out that indeed the validation curve can go up and down and you can never be sure that you reached some kind of absolute minimum. In consequence, if you use "early stopping" methods, you introduce some kind of bias in your model training.
That is why I thought to train as many epochs as I can afford and choose the best epoch to sotp training afterwards. I thought this would be less biased and the most "correct" way to choose the number of epochs for training.
In stackexchange I found the following related questions:

Present results of the best or the last iteration on dev set? where the answer is, that we should choose the best iteration and not the last one.

Reporting test result for cross-validation with Neural Network where the answer is that if is not correct to choose the best iteration since this introduces also a bias. Instead, one should opt for a apriori fixed number of epochs and choose the last iteration.

If I put the theory for a moment aside, I think it is unwise to stop training too soon if the validation loss still goes down. Also it is unwise to choose a validation loss that is higher because I trained until overfitting. So clearly, I am inclined to take some minimum.

To sum it up, at the bottom is a plot of my validation loss of a 5-Fold-CV. Using early stopping could have stopped training at those strange plateaus. Using the best loss score seems also a bit "biased" in those highly jittering regions.

What is the correct way to stop training here??


texts: Which features are important in determining fluency of participants measured by weighted composite score?

texts:
I'm starting to learn about data science. I have sample data including different features run over several people, for each person n times. these features represent the fluency of the participants in some languages. they are including the number of correct words per sentence, the average number of the correct words in total, the average of speaking duration, the average of pauses in each turn, etc. So for features such as the number of correct words, the higher the score the better, and there is a feature such as average pauses which the lower the score the better.
I want todecide at the end which features are more significant and worth keeping and which features can be ignored.
I read about the weighted composite scores, and as far as I understood the main step is figuring out the coefficients (weights). but I didn't understand how these coefficients should be calculated. some cases are based on the analysis intuition, as I saw in some examples, but it can be so subjective.


texts:
I have written a code that uses DBSCAN and tries to find the most appropriate eps for my dataset, trying to include most of the data inside a cluster.

The problem is that DBSCAN is not a classifier and I want to make use of the results that I have gotten from the code to run it upon other dataset to check for anomalies.


texts: Are there any reasons to turn regression into classification - binning continuous targets into classes and then learning model on those instead? (Let's say the algorithm isn't a problem.)


texts:
I want to cluster a dataset without prior knowledge on the correct amount of clusters. For different algorithms (i.e. k-means, gmm...) I can iterate through different values and try to find the best solution for any given algorithm (i.e. ellbow-curve, silhouette-coefficient etc.).
But I get very different results - as expected with different algorithms. K-Means is good for spherical clusters, density-based approaches for totally different cluster shapes.
Now the actual question: How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset? Is there a scientific way to go? Any comparative metric (like rand index) that can be used? Some papers on that topic? Maybe even a flowchart?


texts: Allocation from a central stock in Pandas

texts:
I am struggling with creating an allocation algorithm against multiple demand.

eg: consider 2 banks operating in multiple cities. For each bank in each city there  are available funds that can be disbursed based on demand coming from multiple customers. Demand is catered to in a FIFO fashion. And post disbursement the available funds are reduced by the amount already given out.
A simplified version of the problem is:


texts: Can AI (NLP) convert user questions (text) into database SQL queries?

texts:
I have been reading about NLP but got confused and not able to figure out - if it is feasible for NLP to convert questions in natural language to transform into SQL queries (so that it can execute on concerned database and fetch the output).

Ex. If the user raises a question to the AI engine (application) -

  How many new customers have registered on my website this month?

The AI should parse this using NLP techniques and convert it into SQL and execute on database table say "User_management" -

select count(1) from User_management
where joining_month = 'Oct' and joining_year = 2019

Please share your thoughts and advice.

Update 1) Got this SO link and this NLTK chapter as starting point


texts: Is the output size of the last layer of a standard fully connected neural network the same as the input size?

texts:
Let's say I have a neural net with Dense layers. The input layer has 3 neurons, the single hidden layer has 5 neurons, and the final output layer has 2 neurons.
For layer 1, 3 inputs go in and 5 inputs go out. For layer 2, the 5 inputs from layer 1 go in and 2 inputs go out. So for layer 3, would 2 inputs go in and 2 inputs come out?


texts: Using forecasting values of wind speed at different hours in the future to predict power output at different hours in the future for a wind turbine

texts: I need to design a neural network model for a wind turbine which takes as input the forcasting values of wind speed as follows : windspeedPlus001hr , windspeedPlus002hr , windspeedPlus003hr , windspeedPlus004hr , WindSpeedPlus005hr etc.... The model needs to output the predictions of Power as : PowerPlus001hr , PowerPlus002 , PowerPlus003hr , PowerPlus004Hr , PowerPlus005 etc...  I currently have only the actual power output of the wind turbine with the corresponding timestamp. I also have the values for the forecasting wind speed at different hours in the future as inputs to the model. How do I proceed using python and tensorflow?


texts: How to load the pre-trained BERT model from local/colab directory?

texts:
Hi i downloaded the BERT pretrained model (https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip) from here and saved to a directory in gogole colab and in local .

when i try to load the model in colab im getting "We assumed '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json" . tried to laod the model in local machine and getting same error .

this is how i loaded the model:
from transformers import BertForMaskedLM
BertNSP=BertForMaskedLM.from_pretrained('/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/')

is this the correct way of loading model from the directory when i have downloaded the pretrained model ?
Im getting error " '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json' "
the downloaded model had these naming conventions where file name start with bert_ but the BertForMaskedLM class is expecting the file name to be config.json .

bert_config.json
bert_model.ckpt.data-00000-of-00001
bert_model.ckpt.index vocab.txt
bert_model.ckpt.meta

FULL ERROR:
Model name '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed '/content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

when i renamed the above 4 files by removing bert from all 4 file names , i get this error even though the "model.ckpt.index" files exist

ERROR:
"OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory /content/drive/My Drive/bert_training/uncased_L-12_H-768_A-12/ or from_tf set to False"


texts: Best way to evaluate performance for my case

texts:
I have dataset that looks like this

F1 F2 F3 F4 F5 F6 A1 A2 A3
1  0  0  0  4  3  X  X  X
0  3  9  0  0  0  X  A  X
0  0  0  0  1  0  X  X  X
0  0  3  2  0  0  X  X  X
0  5  0  0  0  1  X  X  X
0  0  7  0  1  3  X  X  X
0  0  0  1  1  0  X  X  B
1  3  0  2  0  1  X  X  X
7  0  0  2  1  3  X  X  X
4  0  0  0  5  0  B  A  X
0  0  7  2  1  3  X  X  X

where F1 to F6 are the features

and A1 to A3 are the prediction attributes.

as you can see most of the records are X X X

when I build my prediction algorithm and try to evaluate the performance I use f1_score

but due to most records have X X X values, I get high f1_score.

which I think this is not the right way to evaluate.

the main point of the evaluation is to find if the values A B and C are predicted correctly.


texts: Structuring extensive medical histories + demographic information for prediciting future medical outcomes

texts:
I'm looking for advice structuring extensive medical histories for predicting future outcomes, specifically hospital admissions.
Let's say I want to predict the whether or not someone will be admitted to the hospital in the next 60 days from today and for each person I have medical histories with tons of data like blood pressure readings, weight, hospital admissions, illnesses, etc. going back 5 years. Of course, stats like blood pressure readings and weight and other stats are are not all happening on the same day, which is where I think I'm getting tripped up.
I'm thinking of this following scenario -- Let's say blood pressure is a good predictor of having a future hospital admission.

Patient One had blood pressure readings that were rising 5 years ago and then had a hospital admission 14 days later.
Patient Two had blood pressure readings that were rising 3 years ago and then had a hospital admission 11 days later.
Patient Four had blood pressure readings that were rising 2.5 years ago and then had a hospital admission 17 days later, but in the last year there were many instances of rising blood pressure that did not lead to hospital admission.
Patient Four has shown rising blood pressure measurements in the past but has never been admitted to the hospital.

How could I structure this data so that if all patients were having rising blood pressure readings now, it might lead to a prediction that

Patients One and Two might have a hospital admission in the next 60 days
Patient Three and Four would likely not be predicted to have a hospital admission in the next 60 days.

I guess I'm having difficulty wrapping my heard around how to use extensive past histories that do not line up for any given patients, with respect to time and data availability (For example, I might not ever have blood pressure readings for a given patient, but because they do not know how to navigate the medical system so they take the ambulance to the ER every single month, therefore, they would likely be predicted to have a hospital admission in the next 90 days on any given day), while also taking into account recency.
I was initially thinking of having a row for every day for every patient, but this does not let me fully utilize a patients entire history which I think will be very important.  Here is another question I've been asking myself... let's say every row is a patient and i have a column called blood_pressure_reading.  Is there a way to represent all of a patients blood pressure readings over the last 5 years inside one cell?  Could I put an array of NULLs and actual readings going back the last 5 years here?


texts:
I have build a  conv net for image classification which work "well"

Now I extract features from last fully connected layer and use it for image retrieval (find image most similar to my target image) using hamming distance. it's working prety well even if I'm not able to predict how it'll be rotation invariant, sensible to noise..

I have try image retrieval only on class that have been seen by my model while training (but not training data). Do you know if it could work on class that have never been seen by the model while training?

e.g

Let's says model has been train on car and truck. I want to find most similar image to that yellow small car and it's work well it's return me yellow and small car.

Now let's imagine I apply it on a new data set with Plane. I want to find most similar image to that small yellow plane. Should I expect to find yellow small plane or that'll be totaly random?

Since network has never seen plane while training, is it possible to predict, or at least have an intuition, of the result?


texts: A generalized quadratic loss for deep neural network for multi-class classification

texts:
I'm evaluating the possibility to introduce a new loss for the subject described above.
Let $l$ be the number of examples, $q$ the number of classes,
 $p_{i,r}$ the $r$ classifier output on example $i$ and $y_{i,r}$ the binary indicator (0 or 1) if class label $r$ is the correct classification for the observation on pattern $i$. It would be: $\sum_{i=1}^{l} \sum_{j=1}^{l} (\sum_{r=1}^{q} y_{i,r} log(p_{i,r})) S_{i,j} (\sum_{r=1}^{q} y_{j,r} log(p_{j,r}))$
where the cross entropy loss is used and c.e. losses are correlated by a similarity matrix S positive semi definite.
There is any method to write this custom loss on a software framework for deep learning (e.g.: TensorFlow) and have a backtracking algorithm that is based on it?


texts:
I am trying to understand better how to read the autocorrelation plot here for a timeseries data.
I ran the following code and got the output as a chart show below.
from pandas.plotting import autocorrelation_plot
autocorrelation_plot(df("y"))

Here y is the dependent variable

Should I derive the following conclusions

There are no significant autocorrelations.
The data is random.
Most of the correlations (except for 2 lags) fall within 95% confidence limits
This timeseries is not worth forecasting

Please help me if my understanding is right ?


texts: How do I predict survival curves using xgboost?

texts:
The xgboost package enables survival modeling using parameter arguments: objective = "survival:cox" and eval_metric = "cox-nloglik".

The predict method for the resulting model only outputs risk scores (same as type = "risk" in the survival::coxph function in r).

How do I use xgboost to predict entire survival curves?


texts: How to supress previous results in a generative network?

texts:
Think of a generative network generating animal names.

It's been trained on a set of animal names so this shouldn't be too hard.

But say I want to generate 10 animal names. First I run the network and it produces ZEBRA. Next it produces HORSE. Next it produces HORSE again. This is no good.

I need a way to supress previous answers.

Since I know the patterns of weights that produced the words ZEBRA and HORSE, can I used this knowledge to tweek the network in able to give me a new unique result.

(I don't mean I want all 10 answers at once! I mean I want to run the network again and again but get different results each time.)

In otherwords I want to supress the maxima that gave me the previous answers.

(This task seems quite easy for a human who can list 20-30 animal names without repetition. I'm not sure if this is because the previous animal names are stored in the short term memory or the synapses for the previous animals are supressed somehow).


texts: Pre-processing irregular, high frequency time-series data in python

texts:
...posted originally in StackOverflow (might be better suited here)

Small Picture:

I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data points, as seen below in the timestamp field:

"timestamp": "2018-06-03T12:27:54.253"
"timestamp": "2018-06-03T12:27:54.409"
"timestamp": "2018-06-03T12:27:54.548"

I'm in the process of developing a sampling scheme for this time-series data so that I can reduce the number of data points, and standardize time steps without losing information or introducing any bias.

So far I've been using Pandas pd.resample() on just a small subset of our data (5 days ~ 2 million records) by using mean as the aggregation function and linear interpolation. I am on downsampling the data by seconds, minutes, and hours for experimental purposes which takes care of the irregular time steps of the original data.

Larger Picture:

I am working with many millions of records (data from April - today) that have been queried from elasticsearch and, ideally, want to pre-process/sample within this large pool of time series data to only obtain statistically significant data points. The purpose of this pre-processing step is for future data exploration/modeling.

Question:

How can I modify my current pre-processing scheme to ensure optimal processing for Millions of records and still maintain their statistical properties (no bias introduced)?

I'm aware that there is possibly a better way to approach the processing this kind of data. Any input is greatly appreciated...thanks!


texts:
I currently have a bunch of extracted news articles to perform news classification. However, the articles are unlabeled. There are about 160k articles therefore manually labeling them is impossible. I'm thinking of clustering the similar news articles together for easier labeling. Is this approach possible?

UPDATE: Now I only want to separate business related news and non-business related news apart.


texts: How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

texts:
NOTE:
If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that:
You encode categorical values as INTEGERES only if you're dealing with Ordinal Classes, i.e. College degree, Customer Satisfaction Surveys as an example.
Otherwise if you're dealing with Nominal Classes like, gender, colors or names, you MUST convert them with other methods since they do not specific any numerical order, most known are One-hot Encoding or Dummy variables.
I encorage you to read more about them and hope this has been useful.
Check the link below to see a nice explanation:
https://www.youtube.com/watch?v=9yl6-HEY7_s
This may be a simple question but I think it can be useful for beginners.
I need to run a prediction model on a test dataset, so to convert the categorical variables into categorical codes that can be handled by the random forests model I use these lines with all of them:
Train:

data_['Col1_CAT'] = data_['Col1'].astype('category')
data_['Col1_CAT'] = data_['Col1_CAT'].cat.codes

So, before running the model I have to apply the same procedure to both, the Train and Test data.
And since both datasets have the same categorical variables/columns, I think it will be useful to apply the same categorical codes to each column respectively.
However, although I'm handling the same variables on each dataset I get different codes everytime I use these two lines.
So, my question is, how can I do to get the same codes everytime I convert the same categoricals on each dataset?
Thanks for your insights and feedback.


texts: Boruta feature selection in R with custom importance (xgboost feature importance)

texts:
According to the documentation - CRAN

  Boruta is an all relevant feature selection wrapper algorithm, capable of working with any classification method that output variable importance measure (VIM); by default, Boruta uses Random
  Forest.

Source: Boruta documentation

The package has one parameter getImp that defines the importance to be used, which by default is random forests from the Ranger package. So theoretically one could use xgboost's xgb.importance() function as the source of feature importance but I cannot find an example of how to use such parameter in practice because so far I've been unlucky. Is it possible at all to do this with the Boruta package or do I have implement from scratch the Boruta algorithm again using xgboost? And if it's possible how?


texts: Select two best classifier using F1-score,Recall and precision

texts:
I have three classifiers that classify same dataset with these results:

classifier A:
              precision    recall   f1-score
  micro avg       0.36      0.36      0.36
  macro avg       0.38      0.43      0.36
  weighted avg    0.36      0.36      0.32

classifier B:
              precision    recall   f1-score
   micro avg       0.55      0.55      0.55
   macro avg       0.60      0.60      0.56
   weighted avg    0.61      0.55      0.53

classifier C:
               precision    recall   f1-score
   micro avg       0.34      0.34      0.34
   macro avg       0.36      0.38      0.32
   weighted avg    0.39      0.34      0.32

I want two select two best of them, and I know F1-score is a parameter for compare the classifiers because of its harmony between precision and recall.
So, at first I select classifier B for its best F1-score. for next, both A and C have a same F1-measure,
I want to ask how can I select between them?


texts: Get the prediction probability using prediction function

texts:
I'm new to SVM models. I took custom SVM classifier from the github. In there standard predict function was overwritten by custom predict function.
def predict(self, instances):
   predictions = []
   for instance in instances:
       predictions.append((int)(np.sign(self.weight_vector.dot(instance.T)[0] + self.bias)))#class is determined based on the sign -1,1

I want to get prediction probability of each class. [-1 probability, 1 probability]
I know from standard SVM predict_proba function I can get the probability. Since this is a customized SVM  how do I get that? Can somebody help me.
Thank you


texts: SMOTE oversampling for class imbalanced dataset introduces bias in final distribution

texts:
I have a problem statement where percentage of goods (denoted by 0) is 95%, and for bads (denoted by 1) it is 5% only. One way is to do under sampling of goods so that model understands the patterns properly for both the segment. But going with under sampling is leading to high loss of data which will directly lower down my model performance. Hence I have opted for over sampling of bads, but over sampling has its own problem too:
Check this code snippet:
from imblearn.over_sampling import SMOTE

sm = SMOTE(random_state = 33)

x = train_data.drop(['target'], axis = 1)
y = train_data[['target']]

x_new, y_new = sm.fit_sample(x, y)

y.target.value_counts()     # 0 ->  26454  1-> 2499
y_new.target.value_counts() # 0 ->  26454  1-> 26454

after oversampling, I get my equal no. of goods and bads, but the problem is that variable distribution is getting affected.
For e.g. I have 'age' variable, in case of good, the bucket wise variable distribution is
 1 - 25 years - 20%
26 - 50 years - 35%
50+     years - 45%

and distribution of bad is (Before OverSampling):
 1 - 25 years - 50%
26 - 50 years - 30%
50+     years - 20%

But post oversampling the distribution of bads is changing:
 1 - 25 years - 40%
26 - 50 years - 35%
50+     years - 25%

So now the distribution of this variable for Good Vs Bad is not that much effective as it was earlier (before oversampling).

Is there any way that doing oversampling does not affect my variable distribution?


texts: Wrapping legacy scripts for reproducibility

texts:
I'm setting up a system that transforms data through several steps using code written by many people with many levels of coding experience. I'm aiming to achieve reproducibility of the whole process by fully defining the inputs and outputs of all the executing steps.

The problem I'm having is that many of the script that need to run have side effects, which can break this determinism. I want to minimise the amount of rewriting of people's code that I have to do, since this system needs to remain usable by those people.

How can I effectively sandbox scripts so that their inputs and outputs are guaranteed to be well defined? My first thought was through containerisation using eg Docker, but our HPC cluster doesn't yet support this so I'd prefer lighter weight options. Feel free to assume python scripts, although the ideal answer would be language agnostic.


texts: Significance of comparing Receiver Operating Characteristic (ROC) curves

texts: An ROC curve plots the true positive rate (sensitivity) as function of the false positive rate (100-specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. This would permit one to compare the goodness of the model when varying the given parameter (e.g. number of trees in a Random Forest) to help optimize that particular classifier's parameters. Although should one be using ROC curves when comparing different types of classifiers (e.g. Random Forest vs Neural Net vs Logistic Regression)? I came across an instance of such an example on the second last slide here and was trying to understand the significance of these curves. As far as I am aware, the graph displayed appears to not be a necessarily fair comparison when comparing the area under the curve while tuning different parameters in different types of classifiers.


texts: Is deep learning a must in a Data Science MSc programme?

texts:
I am reading the programme outline of this two-year MSc in Data Science and I found that it has no deep learning content (as in many other european ones). I am no expert but as far as I've seen I think that DL is going to be a heavy weight of ML algorithms for a long time.

Do you think it is a bad idea to take a strong focus on classical models (e.g. bayesian) instead of teaching DL in a Data Science/ML MSc?


texts: Varying Image sizes in Tensorflow Malaria dataset | Dealing with unclean tensorflow data

texts:
I am trying to build a CNN based image recognition system for the Tensorflow malaria dataset. I loaded the dataset (~27k RGB images) using conventional tensorflow_datasets syntax.
After some data exploration, I found that all the images are not of the same size. A print statement for few instances is shown in below snippet:
import tensorflow_datasets as tfds

ds_train, ds_info = tfds.load('malaria', split='train', as_supervised=True,with_info=True)

ds = ds_train.take(5)  #selecting 5 images

for image, label in tfds.as_numpy(ds):
  print(type(image),image.shape, type(label), label)

OUTPUT
<class 'numpy.ndarray'> (103, 103, 3) <class 'numpy.int64'> 1
<class 'numpy.ndarray'> (106, 121, 3) <class 'numpy.int64'> 1
<class 'numpy.ndarray'> (139, 142, 3) <class 'numpy.int64'> 0
<class 'numpy.ndarray'> (130, 118, 3) <class 'numpy.int64'> 1

The varying sizes of images across the dataset affect the initial CNN layer as flattening each image tensor yields a different sized array.
I understand that all the images need to be converted to a common aspect ratio before the modelling step and we can achieve that by using padding or some other preprocessing techniques of keras.preprocessing.image but I am not sure about the steps to efficiently implement it.
I will be grateful if someone could provide an elegant way around this.
Thank you in advance!


texts: Supervised learning model for extracting terror attack motives from attack summary (NLP)

texts:
I have a terror attack (tabular) data set. Each row is one attack and there are columns like:

Date of the attack (daily resolution)
Location of the attack (long/lat, as well as city/country)
Number of casualties
Attack/weapon type
A few boolean columns like whether it was a suicide attack

Furthermore, I have a text column that holds a 2-3 sentence description of the attack. This is the main column I want to use for training/predicting.

There are several target columns of the form "is_left_wing", "is_right_wing", etc. The values are 0, 1, and -1. Here 0 means the attack didn't have the respective motive, 1 means it had the motive and -1 means it is unknown.

In short, my goal is to build a model that is trained on the 0 and 1 values in the target columns and makes predictions about the -1s.

The main thing I'm stuck on is how to extract features from the text column with the attack description. I have limited NLP experience and I want to use something more sophisticated than a simple bag of words model.

I would appreciate suggestions about the general approach to this problem (also some good readings on the topic).


texts: Formulate multivariate multistep time series forcasting using traditional machine learning, NOT deep learning

texts:
How do you represent multivariate multistep data using traditional machine learning? I know this seems like a tailored problem for RNN/LSTM, but I am wondering what the alternative machine learning solution would be.

Ex.

time, feature_1, feature_2, feature_3
1     25         150        0.7
2     25.3       147        1.1
3     25.5       145        0.8
...
10    26.7       165        1.2

Now I want to predict feature_1 at time 10 using the first 3 time steps of all features, and get a dataset like this

feature_1,       feature_2,       feature_3,       target
[25,25.3,25.5]   [150,147,145]    [0.7,1.1,0.8]    26.7


texts: What is the suitable ML algorithm for predicting root cause of an ITSM ticket?

texts: I have 11 months data from a ticketing tool which has columns like Incident Number, Description, Status, Resolution Notes, Priority, Root Cause. Example of Root Cause: Access Request, Database Issue, Server Down, etc. Most of the incidents are recurring and every month few new incidents will be recorded. Task is to predict the root cause for end of the month report based on the ticket description(only one independent variable). Kindly let me know which algorithm will best suit for this use-case.