texts: How to decide optimal threshold for my classification model from FPR, TPR and threshold

texts:
I am building my model in Python to classify customer in buyer/ non-buyer category. I used mutiple agorithms for this problem and then after evaluation selecting the best out of all.

sklearn package in python gives me an array of FPR,TPR and threshold for all my predicted records.

While analyzing the value of FPR,TPR and threshold I got confused how to interpret threshold value. I am getting threshold value for each record.

For example: My testing dataset consist of 100 records tehn I got array of size 100 for TPR,FPR and threshold.

How to decide the optimium threshold for my model using these values??

from sklearn import metrics
fpr, tpr,threshold = metrics.roc_curve( Y_test, status[:,1])


texts: How to Replace an object in Pandas array using replace with dictionary from excel file?

texts:
Suppose I have a data like this:

and I want to change the value of name and gender into an integer, and I have a dictionary like this:

I understand that I can use replace function in pandas, but I don't know how to use an excel dictionary to replace the value
thank you


texts: Hedge fund rank on their returns or rating predictions modeling problem

texts:
Problem:

Hi, I m a new machine learning practitioner. I have a dataset about hedge funds. It contains monthly hedge fund returns and some financial metrics. I calculated metrics for every month from 2010 to 2019 December. I want to predict or ranking or rating hedge funds returns basis on these metrics for next month. In other words, for a better explanation, in January 2013, fund A had the highest return and ranked first. which metrics have had an impact on this return potential, how much increase or change in which metrics had an impact on this highest return, how much of which metric increased or decreased, and what caused this result? so I want to find out what the relationship or association is between metrics and return potential. I am open to your suggestions and advisories as to what method or approach should be followed in modeling and editing this data set. I could not find many sources on the internet about this because I could not decide which approach this project looks like, whether it is a recommendation system or a credit rating & scoring or segmentation problem. What does this look like?

Sample Dataset:

My Idea:

my modeling idea is to add a new column to the dataset and write the monthly return ranking of each fund here. Repeat this for all past months and update it monthly. Then select this column as the target variable and estimate or classify it in this way. To use the Random Forest algorithm for this idea. And of course, I'll drop the return column. Do you think this approach will work?

I request your help.

Additional Questions:

1)
I think this data set is a time series or cross-sectional data so it is necessary to do Nested cross-validation to this data set. How should the data be split correctly?

2)
Do you think it is necessary to normalize the data?

3)And the last question is, what kind of encoding would you suggest for hedge fund's names?


texts: Generating Dinosaur names with Tensorflow RNN

texts:
I try to adapt "Text generation with an RNN" tutorial to generate new dinosaur names from a list of the existing ones. For training RNN tutorial text is divided into example character sequences of equal length:

# Read, then decode for py2 compat.
text = open(path_to_file, 'rb').read().decode(encoding='utf-8')
# The unique characters in the file
vocab = sorted(set(text))
idx2char = np.array(vocab)
# Creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(vocab)}
# The maximum length sentence we want for a single input in characters
seq_length = 100
examples_per_epoch = len(text)//(seq_length+1)
# Create training examples / targets
char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)
# Convert to sequences of the same length
sequences = char_dataset.batch(seq_length+1, drop_remainder=True)
# Sequences as text
for item in sequences.take(2):
    print("----")
    print(repr(''.join(idx2char[item.numpy()])))

Output:

----
'First Citizen:\nBefore we proceed any further, hear me speak.\n\nAll:\nSpeak, speak.\n\nFirst Citizen:\nYou '
----
'are all resolved rather to die than to famish?\n\nAll:\nResolved. resolved.\n\nFirst Citizen:\nFirst, you k'

My problem differs from tutorial in that I have a list of names of different length instead of monolith of text:

aachenosaurus
aardonyx
abdallahsaurus
abelisaurus
abrictosaurus
abrosaurus
abydosaurus
acanthopholis

In my case character sequences are names. As long as I can't train RNN on sequences with different length (please, correct me if I am wrong here) I need to pad all my names with spaces to a size of a longest name, which is 26.

My longest name is lisboasaurusliubangosaurus, so, for example, aardonyx shoud be padded as:

"lisboasaurusliubangosaurus"
"aardonyx                  "

I tried to pad my sequences with:

# Convert individual characters to sequences of the desired size.
sequences = char_dataset.padded_batch(seq_length+1, padded_shapes=seq_length, drop_remainder=True)

Which results in error:

ValueError: The padded shape (26,) is not compatible with the corresponding input component shape ().

Questions:

Is it possible to train Tensorflow RNN with sequences of variable length?
How to pad short sequences?

Thanks!


texts:
I am performing linear regression on one of UCI Machine learning repository data set. Below is the code :-

import tensorflow as tf
import matplotlib.pyplot as plt
import csv
import os
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

wine_data = pd.read_csv('data/winequality-red.csv', delimiter=';')
n_samples = wine_data.shape[0]

train_x = wine_data.iloc[:, :11].values
train_y = wine_data.iloc[:, 11].values

training_epochs = 1000
learning_rate = 0.03
n_input = 11
n_classes = 1
display_step = 10

X = tf.placeholder(shape=[None, n_input], dtype=tf.float32)
Y = tf.placeholder(shape=[None], dtype=tf.float32)

# Weights and Biases.
W = tf.Variable(tf.truncated_normal([n_input, n_classes]))
b = tf.Variable(np.random.randn())

# Construct Linear Model.
prediction = tf.matmul(X, W) + b
loss = tf.reduce_sum(tf.pow(prediction - Y, 2))/(2 * n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for step in range(1, training_epochs + 1):
        sess.run(optimizer, feed_dict = {X: train_x,
                                         Y: train_y})
        if step % display_step == 0 or step == 1:
            cost, x, y = sess.run([loss, W, b], feed_dict = {X: train_x,
                                                             Y: train_y})
            print (x)
            print (y)
            print ('Step ' + str(step) + ', Minibatch Loss = ' + \
                   '{:.4f}'.format(cost))
    print ('Training Done!!!')

Now, weights and biases get set to Nan only after single step.

[[14430.979  ]
 [  918.5443 ]
 [  484.25635]
 [ 4654.437  ]
 [  151.41441]
 [31196.336  ]
 [97207.76   ]
 [ 1732.7361 ]
 [ 5743.218  ]
 [ 1149.8185 ]
 [18074.775  ]]
1735.9697
Step 1, Minibatch Loss = 32277641302114304.0000
[[nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]]
nan
Step 10, Minibatch Loss = nan

What all reasons could result weight and biases carry Nan?


texts:
I've looked in to the Kaggle datasets to find what kind of web traffic data exists in general.
I found the dataset https://www.kaggle.com/mkechinov/ecommerce-behavior-data-from-multi-category-store which is more or less similar to what I need:

I already looked at their API, gained access, and managed to fetch report data by accessing the view, but not "more raw data".
Can I fetch similar data from Google Analytics, containing timestamp, url, session_id?
I need to access the the free version of Google Analytics und not the 150k p.a. one.


texts: Random Forest but keep only leaves with impurities below a threshold

texts:
Is there an algorithm out there that creates a random forest but then prunes all the leaves that have an impurity measure above a certain threshold that I would determine?
In other words, if I set min samples per leaf to be 500 and leaves have to have at least a 90% purity for example, the algorithm would only keep leaves that respect these parameters.
My dataset is extremely noisy so most leaves have a gini impurity around 0.5 but some leaves are almost around 0. I care only for the latter in my use case. Is there an algorithm that does something like what I described?


texts: ground truth fit is worse than cross validated fit on noisy data?

texts:
I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.
Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-valitation (CV), than when using the "ground truth weights" that I used to generate the data.
Note however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.
Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add guassian noise to the output (y) the "ground truth weights" perform better.
More details below.
Simulation of data
I am generating beta from a guassian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X.
python 3 code:
def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))

  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta

As you can see I also add noise to X.
Regression
I then project this noised data X_noise for different values of noise onto beta:
y_hat = (X_noise) @ beta

And compute the MSE:
mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.
To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:
beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!
PS: This is a crosspost from stack overflow


texts:
I was trying to create a design document for a machine learning project.

From a flow perspective, I know that I would have a training flow, a validation/test flow followed by a production flow.

However, from a design document perspective, I was no able to find any sample template that details out the different sections / subsections that a design document for a Machine Learning project should contain. I believe that the design doc for a ML project should be different from a normal project.

Also, what are the different documents that a ML project should have, from a process perspective?

Any ideas in this direction would be helpful.


texts: Modifying a distribution by adding in samples incrementally

texts:
I would like to calculate the distribution (e.g., Gaussian) of a set of samples.  However, I would also like to see how the distribution changes as I fit the samples into the distribution incrementally.

One way to do this would be to compute the distribution over all relevant samples every increment (e.g., first increment: calculate distribution of 2 samples, second increment: calculate distribution of 3 samples).  However, this is computationally intensive.

Would I be able to calculate the distribution of 3 samples from solely the 3rd sample and the properties of the distribution of 2 previous samples?

For example, say I have 5 ordered samples.

I start off by calculating the mean and standard deviation of the first 2 samples This is the 1st Gaussian distribution.
I then look at the third sample, and fit it into the first Gaussian distribution (knowing the mean, std, number of samples).  This is the 2nd Gaussian distribution.
I then look at the fourth sample, and fit it into the 2nd Gaussian distribution (knowing the mean, std, number of samples).  This is the 3rd Gaussian distribution.


texts: Prove Ridge regression duality of dropout

texts:
EDIT: Found one mistake, confused rows and columns in the step of summing over column operations.
I'm trying to prove the Ridge regression duality of Dropout, as described in section 9.1 of this paper. Check the image below for the section. I've taken the liberty to rewrite the distribution of $R$ as
$$R \sim Bernoulli(\{0^N, 1^N\}\vert p)^D$$
to accentuate that $R$ is a $N\times D$ matrix where each column is either entirely 0 or 1, with probabilities $1-p$ and $p$. The notation of the paper suggests that each element of $R$ is distributed independently, but this not how I understand dropout. Please correct me if I'm wrong.
$$\textrm{minimize}_{\textbf{w}}\mathbb{E}_{R \sim Bernoulli(\{0^N, 1^N\}\vert p)^D} \Vert \mathbb{y} - R*X\cdot \mathbb{w}\Vert^2$$
We can rewrite the dot product between $X$ and $\mathbb{w}$ as the sum over all the columns:
$$=\textrm{minimize}_{\textbf{w}}\sum_{c=1}^D\mathbb{E}_{r \sim Bernoulli(\{0, 1\}\vert p)} \Vert \mathbb{y} - r*X_c* \mathbb{w}_c\Vert^2$$
Expanding the expectation value to the weighted sum of the options:
$$=\textrm{minimize}_{\textbf{w}}\sum_{c=1}^D (1-p) \Vert \mathbb{y}\Vert^2 + p\Vert \mathbb{y} - X_c * \mathbb{w}_c\Vert^2$$
This is where I get stuck. I can see how $\Vert X_c\cdot \mathbb{w}\Vert^2$ would create the $\Vert \Gamma \cdot \mathbb{w}\Vert^2$. But the $(1-p)$ seems awkward here.
Question: How can one prove the equivalence with
$$\textrm{minimize}_{\textbf{w}}\Vert \mathbb{y} - p*X\cdot \mathbb{w}\Vert^2 + p(1-p)\Vert \Gamma \mathbb{w}\Vert^2$$
?
Section from the paper:


texts: Python + Neural Nets + Large dimension Large Dataset

texts:
I am building a NLP application. My dataset has 0.6M datapoints each of 0.15M dimensions. My feature vector is highly sparse - mostly 0s, atmost 20 1s in any feature vector. I am using a 32 GB machine on AWS.

Owing to size, Cant load entire dataset in one go in the memory. I am looking for a python based neural net library that supports (if not all then atleast some) of the following [Owing to the dataset size and dimension, i defined below mentioned abilities]:

Ability to train the net on small batches
library being able to exploit underlying multicore architecture (if available)
A way to directly feed the feature vectors in compressed representation (like scipy.sparse) [in my case

I have had a good look at the following libraries - PyBrain, Pylearn2, NeuroLab, FANN. To the best of my understanding, none of them has any of the desired abilities (Pybrain facilitates batch leaning. Even this takes close to 55 hrs).

Any suggestions ? (cant reduce dimentionality)  Or in case there is a shortcoming in my understanding and the any of the above libraries have (some) above capabilities, can you pls point out me to some sample code implementing/exploiting the same ?


texts: Which Machine Learning book to choose (APM, MLAP or ISL)?

texts:
I'm searching a book as a refresher in machine learning (I have taken a lecture in machine learning sometime ago). I will be applying machine learning in a project.

I have searched a lot of books and found the following three the one which best fits me:

Applied Predictive Modeling by Kuhn and Johnson
Machine learning an Algorithmic Perspective (second edition) by Marsland
An Introduction to Statistical Learning by Hastie

Applied Predictive Modeling and An Introduction to Statistical Learning uses R as examples and Machine learning an Algorithmic Perspective uses Python. The used language does not matter that much because I'm not really interested in language examples (I'm using Matlab) and I will probably skip such examples.

The advantage of Applied Predictive Modeling seems to be that it covers the whole machine learning procedure (feature selection etc.) and seems to be very well written. The advantage of Machine learning an Algorithmic Perspective (second edition) is that it covers more topics (ensemble learning, graphical models, gaussian processes) and has perhaps a bit more math.

Which one of these three books would you recommend (and why)?


texts: Fix missing data by adding another feature instead of using the mean?

texts:
I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These reports are helpful summaries of information for the user to read later and gauge their progress. The data looks something like this.

user  |  hours until first report
------|--------------------------
3     | 20
------|--------------------------
4     | NaN
------|--------------------------
7     | 24
------|--------------------------
9     | 13
------|--------------------------
6     | 500

A NaN value likely means that no report was ever written for this user because they unsubscribed before one could be written. I thought to fill in the blank values with the median, most common value, or the mean but this change in the data would show that unsubscribed users received written reports in the same time frame as the average user, which is incorrect.

Instead of filling the missing value with the mean, I am considering adding a feature that tells whether a report was written at all and changing the NaN values to -1 so that I can pass the data to a sklearn algorithm.

user  |  hours until first report  | report written
------|--------------------------------------------
3     | 20                         | 1
------|--------------------------------------------
4     | NaN                        | 0
------|--------------------------------------------

Is this a valid way of solving this particular problem? What are the risks that I should be aware of?


texts: Maximum Entropy Policy Gradient Derivation

texts:
I am reading through the paper on Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review by Sergey Levine. I am having a difficulty in understanding this part of the derivation on Maximum Entropy Policy Gradients (Section 4.1)

Note that in the above derivation, the term H(q(thetha(at|st))) should have been log (qthetha(at|st)), and that log refers to log base e (i.e. natural logarithm). In the first line of gradient, it should have been r(st,at) - log(qthetha(at|st)).
In particular, I cannot understand how the second summation term from t'=t to t'=T in the second line arises in the derivation. I managed to derive the formula by expanding the definition of expectation, but the result that I got do not have the second summation term. Can anyone give me ideas on where this second summation term comes from mathematically?


texts: In deep neural networks, sometimes we see the problem of exploding and vanishing gradients. So how is it handled by GPT 3 which has billions of parameters?


texts:
I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines.

Situation: I want to fill some missing values with the mean but using groups based on other feature. That's why I'm using this custom function:

def replaceNullFromGroup(From, To, variable, by):

    # 1. Create aggregation from train dataset
    From_grp = From.groupby(by)[variable].median().reset_index()

    # 2. Merge dataframes
    To_merged = To.merge(From_grp, on=by, suffixes=['_test', '_train'], how = "left")

    # 3. Create dictionaries
    to_cols = [col for col in To_merged.columns if 'test' in col]
    from_cols = [col for col in To_merged.columns if 'train' in col]
    dict_cols =dict(zip(to_cols, from_cols))

    # 4. Replace null values
    for to_col, from_col  in dict_cols.items():
        To_merged[to_col] = np.where(To_merged[to_col].isnull(),
                                     To_merged[from_col],
                                     To_merged[to_col])

    # 5. Clean up dataframe
    To_merged.drop(from_col, axis=1, inplace=True)
    To_merged.columns = To_merged.columns.str.replace('_test', '')
    return To_merged

Variables meaning:

From: Dataframe where I'm taking the information (Train dataset)
To: Dataframe where I will fill the missing values (Train and test dataset)
variable: variable with missing values
by: Variables I'm using to make groups

Can I use this function in a pipeline so I can use cross validation avoiding data leakage?

Thank you very much


texts: What is the procedure to create a bag of visual words model with SIFT?

texts:
I have more than 1500 black and white classified images in a training set and I want to create a probabilistic model to classified new images. To be more explicit, given a new black and white image, my model has to predict:

animal: 84%
vegetal: 12%
mineral: 4%

I watched this lecture and I still have questions about the procedure to create this model.

1. Extraction of the keypoints of each image

If I correctly understood the video, the first step is to extract all the keypoints with SIFT from all the images to create a kind of dictionary of visual words. Each word is described by a multidimensional vector.
Next, I have to use the $k$-means method to create groups of visual words.

Question 1: how many groups should I create? Does it exist a rule-of-thumb to determine the $k$ number I have to use?

2. Creation of histograms

Now, for each image, I have to create an histogram/a vector where each features corresponds to one group defined in part 1. The value associated to each feature corresponds to the frequency of the word in the image.

Question 2: how can I create this vector? Indeed, each image is unique and will never match perfectly with the words in my dictionary (words who is, by the way, mean of different words). Who can I bypass this problem?

3. Creation of the model

Finally, I have to create my model. In the video, the lecturer used SVM and has to create one classifier by category (binary classifier). In my case, I have 100 different categories (my introduction was a simplification) and I prefer to have only one classifier. Also, I want to get the probabilities, given an image, to be part of a category.

Question 3: is it possible to create only one classifier who gives probabilistic data?

To finish, don't hesitate to make suggestions or corrections about the procedure I described if you know a better way to classified my data.


texts: Isolated/noisy instances that have outsized effect on SVM hyperplane selection

texts:
Consider two linearly separable classes and the optimal separating hyperplane (image credit Prof Jiawei Han of UIUC):

Now consider if there were a single "rogue" point for the Yellow class - which may or may not be noise - and the effect it has on the corresponding optimal separating hyperplane (shown in red):

Is there a modified version of SVM to allow for this situation - to "consider" the isolated observation when calculating the hyperplane but not be strictly driven by it?  Would this be a case of using slack variables  - as shown in this slide (credit: CS @ CMU https://www.cs.cmu.edu/~tom/10701_sp11/slides/Kernels_SVM2_04_12_2011-ann.pdf)  ?

Is this problem - which is to say

  some observations may be noisy and we should not put too much weight on any individual one

the same one as

  instances of the two classes are not linearly separable

If not then would there be a different approach to representing this svm with noise then using slack variables ?


texts: Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks?

texts: Can the Apple M1's iGPU access the entire RAM as "video memory" when training with typical deep learning frameworks (e.g., tensorflow_macos)? If not, what memory do they use as video memory?


texts: How to use the finite difference to compute gradient for very complex loss function

texts:
I need to put the prediction into a complex function to calculate loss.
This means that i can't build the loss function by tensorflow's operator and can't get  gradient from Automatic Differentiation.

I want to get gradient from finite difference in my training step.


texts: How will Occam's Razor principle work in Machine learning

texts:
The following question displayed in the image was asked during one of the exams recently. I am not sure if I have correctly understood the Occam's Razor principle or not. According to the distributions and decision boundaries given in the question and following the Occam's Razor the decision boundary B in both the cases should be the answer. Because as per Occam's Razor, choose the simpler classifier which does a decent job rather than the complex one.

Can someone please testify if my understanding is correct and the answer chosen is appropriate or not?
Please help as I am just a beginner in machine learning


texts: How to increase number of outliers in a dataset?

texts:
I have a dataset with 1000 rows and 4 columns with 3 outliers .I want to add another 7 outliers related to them for detection by clustering.

     Example TO What I did
         Col1 col2 Col3 col4
     Out1 a1    b1   c1   d1
     Out2 a2    b2   c2   d2
     Out3 a3    b3   c3   d3

I get mean and std for 7 columns of normal data ten calculate

Out4  normal1+mean+stdcol1   norm1mean+stdcol2
 Out5  normal2+mean+stdcol1  norm2mean+stdcol2
 Out6 ...........

I don't know if  what i did is right or a good solution?

I don't  want outliers to be so easy for detection

Thanks


texts: What are some options for an offline chatbot on Android?

texts:
I have already created an online chatbot for my Android app using Dialogflow (formerly api.ai). It allows users to add a new item to their inventory. I now need to create an offline version of this chatbot. However, Dialogflow does not allow you to download the agent to work offline.

I have looked into several options, including RiveScript and ALICE (Pandorabots). These seem to be a bit too low-level for my purposes and require quite a bit of customization. I was wondering if there are any options similar to Dialogflow that can be used offline.


texts: How many dimensions to reduce to when doing PCA?

texts: How to choose K for PCA? K is the number of dimensions to project down to. The only requirement is to not lose too much information. I understand it depends on the data, but I'm looking more for a simple general overview about what characteristics to consider when choosing K.


texts: Is it valid to include your validation data in your vocabulary for NLP?

texts:
At the moment, I am following best practices and creating a "bag of words" vector with a vocabulary from the training data. My cross validation (and test) datasets are transformed using this model, using the same vocabulary created by the training set. They don't contribute any vocabulary, or affect the document frequency (for "term-frequency inverse document frequency" calculation).

However, this is restrictive in a few ways. Firstly, calculating the bag of words model is expensive, and so this prohibits me carrying out k-folds cross validation (since it would require constant re-calculation of the bag of words). My dataset is around 10 million words, and I'm calculating bag of words and bag of bi-grams, which takes around 5 minutes each time.

This also means I currently have holdout data for both my cross validation and test sets, which is data I can't use for training.

Would I be biasing my results significantly if I fit the bag of words on both the training set, and the cross validation set? In other words, if I use the vocabulary in the validation set to calculate the vocabulary for the bag of words? The way I figure, even though they might contribute to the vocabulary, there's no risk of overfitting since the frequency for those specific samples won't be seen at training. This allows me to slice the validation set later however I like, and I still have a "test" set for an accurate predictor of generalisation error (the test set won't be seen at all until test time).

I wonder if there's any precedent for something like this, and what your experiences are doing anything similar.


texts: Best approach for univariate time series predictions?

texts:
I have a univariate time series. where I'm trying to predict a current value of a variable based on the previous 10 values of the same variable. I tried three approaches:
1- linear regression where I use the previous 10 values as features.
2- ARIMA model
3- deep learning (LSTM) model
I computed the RMSE between the predicted and expected current values. To my surprise, the linear regression model is performing slightly better.
RMSE linear regression = 27.73
RMSE Arima= 29
RMSE LSTM= 28.7

I know that this could be a vague question, please let me know if you need any further information such as code snippets or plots.
My questions is: Is this expected ? I would have thought that LSTM would be the most powerful approach. Is it possible for a simple linear regression model to outperform an LSTM?
Do you have any advice on how to improve the performance of LSTM ?


texts: How can I Determine a Treshold According to the Precision and Recall?

texts:
I am gettin these precision and recall values from my classifier and I want to determine a treshold for the test data. How can I determine that treshold? Is these values enough or something else is needed?
Precision: 95.88%
Recall: 97.89%


texts:
What are the basic principles/tools necessary to make something like Alexa utterance parsing?

For reference, Alexa allows a designer to define phrases with "placeholders" that will be filled in. For example, the phrase "what is the horoscope for gemini" would match the underlying model (below), and return Sign=gemini.

what's the horoscope for {Sign}
what is the horoscope for {Sign}
give me the horoscope for {Sign}
tell me the horoscope for {Sign}

To clarify: I am interested in the theory behind how the language model and parsing (with regard to theory and algorithms) work, so I can build my own version.


texts:
I'm having some trouble understanding the way the Faster R-CNN algorithm works. Specifically, the way the authors describe the concept of anchors.

In their paper from here they describe anchors in the following way:

  At each sliding-window location, we simultaneously
  predict multiple region proposals, where the number
  of maximum possible proposals for each location is
  denoted as k. So the reg layer has 4k outputs encoding
  the coordinates of k boxes, and the cls layer outputs
  2k scores that estimate probability of object or not
  object for each proposal. The k proposals are parameterized relative to k reference boxes, which we call anchors. An anchor is centered at the sliding window
  in question, and is associated with a scale and aspect
  ratio (Figure 3, left). By default we use 3 scales and
  3 aspect ratios, yielding k = 9 anchors at each sliding
  position. For a convolutional feature map of a size
  W × H (typically ∼2,400), there are W Hk anchors in
  total.

I can't wrap my head around this. How do you incorporate the fixed scales (128x128, 256x256, 512x512) and aspect ratios (1x1, 1x2, 2x1) (which they mention later), into this, when they also predict the bounding box with its parameters x, y, width and height in the reg layer for each anchor?


texts:
I have a very large dataset consisting of approximately 15,000 rows of survey data. The surveys are quantitative summaries of interviews with investors regarding their rankings of financial asset managers. They rate a manager on a scale from 1-5 and we have approximately 5 years of data for each investor on asset manager.

The data is indexed as:

Investor name | Manager name | Year | Rating parameter 1 | R. parameter 2 | ... | R. parameter 10 |

To give you an idea, I made an example below. Currently the data is saved in an Access-database with a ton of SQL JOINT's - but I can convert the data pretty much how I want to. So compatibility is not an issue.

My question is:
I have a wild idea that somehow the answers are correlated between years. So maybe an increase in parameter 1 and parameter 2 is correlated with a fall in parameter 10.

I'm looking for ways to test my hypothesis without having to do it manually (this would be quite tedious). Maybe through machine-learning or subgroup discovery (I have not idea if this is correct).

Are any of you familiar with tools or algorithms that can help me test my hypotheses? Naturally I'll have to make sure my analyses are statistically significant.

I'm looking very much forward hearing what you think.

Example of data

Buffet Holdings | Goldman Sachs AM | 2014 |1|5|5|1|4|5|1|1|1|5|
Buffet Holdings | Goldman Sachs AM | 2015 | 4 | ........| 1 |
Gates Ltd. | JP Morgan AM | 2014  | 4 | ........| 2 |
Gates Ltd. | JP Morgan AM | 2015 | 4 | ........| 3 |
Buffet Holdings | JP Morgan AM | 2014 | 3 | ........| 5 |
Buffet Holdings | JP Morgan AM | 2015 | 2 | ........| 1 |


texts: Do the benefits of ridge regression diminish with larger datasets?

texts: I have a question about ridge regression and about its benefits (relative to OLS) when the datasets are big. Do the benefits of ridge regression disappear when the datasets are larger (e.g. 50,000 vs 1000)? When the dataset is large enough, wouldn't the normal OLS model be able to determine which parameters are more important, thus reducing the need for the penalty term? Ridge regression makes sense when the data sets are small and there is scope for high variance, but do we expect its intended benefits (relative to OLS) to disappear for large datasets?


texts: Why is averaging the vectors required in word2vec?

texts:
While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example links below.

https://www.kaggle.com/ananyabioinfo/text-classification-using-word2vec/notebook?scriptVersionId=11358361&cellId=7

https://www.kaggle.com/varun08/sentiment-analysis-using-word2vec?scriptVersionId=2185653&cellId=15

My questions are:

Is it just to create a single vector instead of vectors of the dimension size or to increase the accuracy or is there any reason behind this?
Is it mandatory to take average of the vectors or are there any alternatives to this.

I have gone through the original paper on word2vec but that doesn't give clear explanation on this.


texts: How to approach Multilingual Text Classification?

texts:
The company I'm working for runs social network sites, and we're classifying messages sent one-to-one as spam or not spam. The issue is that it's been trained exclusively on German training data, since most of the networks we operate are local to Germany. Soon we will need to support multiple languages for this, since the one trained on only German will often think messages in other languages are spam as well.

I'd prefer not to have to maintain different training data for different languages if possible, so my question is:

What's a reasonable approach to doing text classification for multiple languages?

A spam message in our cases usually contains obfuscated links and subtle references to other websites. Example of a message that should be classified as spam:

  hi funnyguy kennst du mich noch ? konnte hier kein bilder hoch laden komm mal bitte zu ( somethingelse.com) mein nik ist ( 19theusername91 ) mal sehen ob du mich noch kennst :)

  Somegirl

The current approach is the normal stemming, TF-IDF and LSA for pre-processing, then a two-level classifier: an ensemble of normal classifiers that's used as input for a linear classifier that will make the final decision.


texts: How to deal with severe overfitting in a UNet Encoder/Decoder CNN in a task very similar to image translation?

texts:
I am trying to fit a UNet CNN to a task very similar to image to image translation. The input to the network is a binary matrix of size (64,256) and the output is of size (64,32). The columns represent a status of a communication channel where each entry in the column is the status of a subchannel. 1 means that the subchannel is occupied and 0 means that the subchannel is vacant. The horizontal axis represents the flow of time. So, the first column is status of the channel at time slot 1 and the second column is the status at time slot and so forth. The task is to predict the status of the channel in the next 32 time slots given the the previous 256 time slots which I treated as image to image translation.
The accuracy on the training data is around 90% while the accuracy on the test is around 50%. By accuracy here, I mean the average percentage of correct entries in each image. Also, while training the validation loss increases while the loss decreases which is a clear sign of overfitting. I have tried most of the regularization techniques and also tried reducing the capacity of the model but this only reduces the training error while not improving the generalization error. Any advice or ideas? I included in the next part the learning curve for training on 1000 samples, the implementation of the network and samples from the training and test sets.
Learning curves of training on 1000 samples
3 Samples from the training set
3 Samples From the test set
Here is the implementation of the network:
def define_encoder_block(layer_in, n_filters, batchnorm=True):
    # weight initialization
    init = RandomNormal(stddev=0.02)
    # add downsampling layer
    g = Conv2D(n_filters, (4,4), strides=(2,2), padding='same',
               kernel_initializer=init)(layer_in)
    # conditionally add batch normalization
    if batchnorm:
        g = BatchNormalization()(g, training=True)
    # leaky relu activation
    g = LeakyReLU(alpha=0.2)(g)
    return g

# define a decoder block
def decoder_block(layer_in, skip_in, n_filters, filter_strides, dropout=True, skip=True):
  # weight initialization
  init = RandomNormal(stddev=0.02)
    # add upsampling layer
  g = Conv2DTranspose(n_filters, (4,4), strides=filter_strides, padding='same',
                         kernel_initializer=init)(layer_in)
    # add batch normalization
  g = BatchNormalization()(g, training=True)
    # conditionally add dropout
  if dropout:
    g = Dropout(0.5)(g, training=True)
  if skip:
    g = Concatenate()([g, skip_in])
    # relu activation
  g = Activation('relu')(g)
  return g

# define the standalone generator model
def define_generator(image_shape=(64,256,1)):
    # weight initialization
    init = RandomNormal(stddev=0.02)
    # image input
    in_image = Input(shape=image_shape)
    e1 = define_encoder_block(in_image, 64, batchnorm=False)
    e2 = define_encoder_block(e1, 128)
    e3 = define_encoder_block(e2, 256)
    e4 = define_encoder_block(e3, 512)
    e5 = define_encoder_block(e4, 512)
    e6 = define_encoder_block(e5, 512)
    e7 = define_encoder_block(e6, 512)
    # bottleneck, no batch norm and relu
    b = Conv2D(512, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(e7)
    b = Activation('relu')(b)
    # decoder model
    d1 = decoder_block(b, e7, 512, (1,2))
    d2 = decoder_block(d1, e6, 512, (1,2))
    d3 = decoder_block(d2, e5, 512, (2,2))
    d4 = decoder_block(d3, e4, 512, (2,2), dropout=False)
    d5 = decoder_block(d4, e3, 256, (2,2), dropout=False)
    d6 = decoder_block(d5, e2, 128, (2,1), dropout=False, skip= False)
    d7 = decoder_block(d6, e1, 64, (2,1), dropout=False,  skip= False)
    # output
    g = Conv2DTranspose(1, (4,4), strides=(2,1), padding='same', kernel_initializer=init)(d7)
    out_image = Activation('sigmoid')(g)
    # define model
    model = Model(in_image, out_image)
    return model


texts: Hello I am looking for a dataset for printed english characters but I am not able to find anything concrete. Is there any software which can generate multiple images from one image. I can build my dataset from that.


texts: Bootstrap clarification for datasets with multiple columns

texts:
How do I bootstrap a Dataset/DataFrame with multiple continuous and categorical columns?
For eg: Say I'm trying to bootstrap the colour distribution of M&M's and I have 50 bags (samples) each with recorded counts of 6 colours, but the samples/bags randomly come from two factories, and have 2 different time stamps on the sample/bag packets.
What would be a good method of bootstrapping this kind of dataset?
Using loops that resample slices by categorical values? eg: Split the dataset into serial_no groups E2 and E1, resample those with replacement and combine, then do the same for serial_time etc...
Or can the whole dataset be resampled at once?
If there happens to be subtle patterns within 'E2' data for eg, how is that preserved in bootstrapping?

# create dataframes for serial_no columns.
E1_bs = pd.DataFrame()
E2_bs = pd.DataFrame()

# proportinal sample with replacement
E1_df = df[df['serial_no'] == 'E2']
sample_size = len(E1_df)*10
for i in range(len(E1_df)):
    E1_swr = pd.DataFrame(E2_df.sample(sample_size, replace=True))
    E1_bs = pd.concat([E1_bs,E1_swr],axis=0)

E2_df = df[df[['serial_no'] == 'E1']
sample_size = len(E2_df)*10
for i in range(len(E2_df)):
    E2_swr = pd.DataFrame(E2_df.sample(sample_size, replace=True))
    E2_bs = pd.concat([E2_bs,E2_swr],axis=0)

bootstrap = pd.concat([E2_bs,E1_bs],axis=0)

If I do this for all of the categorical columns (serial_no and serial_time in this case) and add them together as one dataset, I get different means (significantly) compared to if I just sample the whole thing at once using the same total amount of samples).
# Compared to bootstrapping the whole thing
sample_with_replacement = pd.DataFrame(df.sample(len(bootstrap), replace=True))


texts: Suggestions for binary classification algorithm

texts:
I am new to machine learning, so was hoping you might have some insights one what classification algorithm might be best suited for my data.

I have a data set of hundreds of predictions from each of ~100 forecasters. Each prediction is binary, and the outcome variable is binary.

So basically let's say I have data on 200 events. For each event, I have 1 prediction from each of 100 forecasters. Each of the 100 forecasters predicts each of the 200 events.

I am hoping for an output of some formula, for which I input the predictions of the forecasters for that particular outcome, and it renders an output for the probability of that outcome.

My main interest is in predictive accuracy.

Do you have any suggestions for what algorithms might suit this problem?


texts: How to handle columns with categorical data and many unique values

texts:
I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?


texts: Calculating distance between data points when there are more than 3 features in KNN algorithm

texts:
I've been reading about K-nearest neighbors algorithm and want to clarify few things.
If we have 2 features we could simply plot it on 2-d plane and calculate distance by using euclidean distance or Manhattan distance.
When there are more than 3 features, exceeding 3-d going into 4-d and more I've read that we use PCA to reduce dimension to 2-D and then calculate distance on PCA plot.
But my question is, is that only way? so in order to use KNN for more than 3 features we must use PCA?


texts:
Suppose I have $f:\mathbb{R}^{d_i}\to\mathbb{R}^{d_o}$.
Let $X \in \mathbb{R}^{n \times d_i}$ and I apply $f$ to each row of $X$, obtaining $Y = f(X) \in \mathbb{R}^{n \times d_o}$.
I would like to compute a tensor $Z$ which is defined as

$$Z_{i,j,k} = \frac{\partial Y_{i,j}}{\partial X_{i,k}}$$

using pytorch. How should I define v to achieve that?

def get_row_wise_jacobian(X, f):
     n = X.shape[0]
     d_i = X.shape[1]

     x = X.clone().detach().requires_grad_(True)

     y = f(x)
     d_o = y.shape[1]

     v = ?
     y.backward(v)

     Z = x.grad
     return Z


texts: How to represent an image as state in a Q-table

texts:
I'm trying to do Q-learning with the Atari games using the gym python's package.

I want to use the image as the state of my algorithm, but I came up with a doubt: Is the state all the possibilities for all the pixels of the image?

I've seen many Q-learning implementations, and in the Q-table they always store all the possible states as rows, and the actions as columns (for example, https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/)

Without doing maths, all the possible states for, let's say, a 200x200 grayscale image are insane. How can I handle this problem without doing neural networks?

I guess there must be an approach for this scenario, but I haven't found any good example.

Thank you!!


texts:
I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights matrices in the hidden and output layers.

The cost function is given as:

$J(\Theta) = \sum\limits_{i=1}^2 \frac{1}{2} \left(a_i^{(3)} - y_i\right)^2$

where $y_i$ is the $i$-th output from output layer.

Using the gradient descent algorithm, the weights matrices can be updated by:

$\Theta_{jk}^{(2)} := \Theta_{jk}^{(2)} - \alpha\frac{\partial J(\Theta)}{\partial \Theta_{jk}^{(2)}}$

$\Theta_{ij}^{(3)} := \Theta_{ij}^{(3)} - \alpha\frac{\partial J(\Theta)}{\partial \Theta_{ij}^{(3)}}$

I understand how to update the weight matrix at output layer $\Theta_{ij}^{(3)}$, however I don’t know how to update that from the input layer to hidden layer $\Theta_{jk}^{(2)}$ involving the ReLU activation units, i.e. not understanding how to get $\frac{\partial J(\Theta)}{\partial \Theta_{jk}^{(2)}}$.

Can anyone help me understand how to derive the gradient on the cost function...?


texts: How would you optimize this python/pandas code?

texts:
In [1]: df.head()
Out[1]:

   uf    city
0  mg    araguari
1  mg    uberlandia
2  sp    sao-paulo
3  go    goiania
4  mg    belo-horizonte

In [2]: len(df)
Out[2]:

982

In [3]: indicators.head()
Out[3]:

     city               uf         population
 0   abadia-goias       go         6876
 1   abadia-dourados    mg         6704
 2   araguari           mg         19000
 3   abaete             mg         22690
 4   uberlandia         mg         700000

In [4]: len(indicators)
Out[4]:

5554

Now here I'm trying to include 'population' in df, using the corresponding uf and city. But this next code is too slow.

df['population'] = np.nan

for city, uf, index in zip(df.city.values, \
    df.uf.values, df.index):

    df['population'].iloc[index] = \
    indicators.population[indicators.uf == uf] \
    [indicators.city == city].values[0]


texts: Replace missing values of numerical features with unique numbers

texts:
Let's suppose that I have a dataset with 5 numerical features of which each of them has some missing values and all of them have only non negative values.

Some suggested ways to deal with missing data are:

Remove the rows which have even one missing value
Impute the missing values

I do not prefer (1) because then you miss some valuable information from the rest of the features for these rows.

I do not prefer (2) because in general (it depends on the application) it introduces quite a lot of noise in the data.

What I am thinking is to:
 3. Replace the missing values with a unique value (eg -1 or -999)

As I said, in my example the features have only non-negative numbers so values such as -1 or -999 will be only encountered by the algorithm for missing data.

What are your thoughts on (3)?

What are the advantages and disadvantages of this approach?


texts: How to split a CSV into a dataframe without newline

texts:
Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same line?

#Loads the data in the textfile
df = sc.textFile("data.csv").map(lambda line: line.split(","))
df.take(1)

[['2',

  '20200118',

  '0',

  'AE',

  'ABV',

  'AUH',

  '20200416',

  '0',


texts: How to measure a relationship between interests of topics between two users?

texts:
I was trying to find a way to calculate the relationship between two Users and their interests. A user can have multiples interests, and an interest can be duplicated in a user's interests. Example:

User1: {
   interests: {"golf": 3, "baseball: 1", "tenis: 2"}
}
 User2: {
   interests: {"golf": 5, "baseball": 1, "football": 4}
}
 User3: {
   interests: {"golf": 1, "baseball": 1, "tenis": 1}
}

Comparing User1 & User2:

They have in common 2 interests, but User2 seems to be a very fan of golf.

Comparing User1 & User2:

They have in common 3 interests, while User3 seems to have a slightly interest in all common interests, not being a very fan of any interests listed.

What I want do to is

The relationship between User1 and User2 should be a higher number than the relationship between User1 and User3, knowing that User3 has only a superficial interest in the topics, and User2 is more likely to be a fan like User1 to golf, for example, meaning that they may want to know each other and etc.

e.g. User1&User2  = 2.5; User1&User3 = 1.8

I tried to use the mean of the intersection of all interests between User-k and User-y and it seemed a reasonable approach, but I think it can have a better way to do this. Suggestions?

Thanks!


texts: NLP how to go beyond simple intent finding--using context and targeting objects

texts:
I'm manually applying NLP rules to a chatbot.

Currently, I've a simple set of rules--actions that follow certain trigger words.
Ex: "Create the match on saturday."

This has been working for relatively simple phrases like the above example, where I would expect words like "create" and appoint it as an action word, expect "match" then appoint it as an object entity; "saturday" as a time, etc.

When I try to expand the scope of what the bot can handle, it becomes more complicated. Here's an example of something I'm trying to handle:

"Update the memo title of the match on saturday to 'Game Day'."

I'm not sure how to move forward.

I considered manually expanding(expecting) the entities, then trying a method where I still parse for action words, but if a certain threshold of varying objects is passed, then I execute a subset of that action.

For example: update will obviously signal an action, but the addition of "memo", "title", "'Game day'", signals a subset of the action as there's more to this than a simpler "create match". Then, checking the additional objects like "title" against a predetermined set of entities will narrow down the intent to update + title.

I see many holes in this logic, esp. as the complexity even slightly increases.

This led me to the field of dependency parsing.
But upon looking into dep. parsing, I wonder if this is something feasible in manual implementation.

I'll be using python code, it won't be deep learning based.

What do you think of the basic rule-based method I've outlined? Is it something that sounds workable for a domain-specific bot?

Should I be looking into using NLP libraries offered in Python such as NLTK or spaCy and use their features? My concern with using these features such as spaCy's dependency parser was that it was overkill, or too much added complexity for handling my domain specific tasks. Furthermore, and likely because I haven't yet seen effective use of it in another project, I guess I have doubts the practicality of dependency parsers outside of academia or research.

Edit: Apologies if this isn't the proper space for this kind of question. Would appreciate if you can please point me in the right direction


texts:
Assuming that I have a list of Users with a list of skills: (each value is a different skill)

And a list of Tasks with a list of demanded skills:

Based on a manual classification that returned: (History list)

Task1 --> Recommended: User3
Task2 --> Reccomended: User5
Task3 --> Recommended: User6
Task4 --> Recommended: User9
Task5 --> Recommended: User8

Given a new Task(6) with skills (A,B,C...):

What would be the best way or approach to build a recommendation system that learns with past data (history list) and returns a list of the "best ranked" users to perform this Task?


texts: How to scale outputs from AutoEncoder from multiple models?

texts:
I have a problem for which I have not been able to find any answers in my search so far.

BACKGROUND

I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building a model file per machine because the machines' temporal behaviour varies quite a lot.

I have 5 features:

Numerical integer ranging between 0 and x (x varies per machine)
The other 4 features are categorical (After trying LabelEncoding, my architecture prefers One-Hot encoding)

I have tried to scale the numerical feature using Normalisation (MinMaxScaler and StandardScaler) which did not yield very good results at all.

As an alternate to scaling the inputs - I decided to scale the outputs using MinMaxScaler from scikit. This is so that I can have 1 generic threshold I can apply across the different models to identify anomalies.

Although this has yielded the best results so far - in practice, the outputs become too polarised to either the 0 or the 1 and consequently I am missing outliers that I shouldn't be.

PROBLEM

What scaling technique(s) can I use on the output from my auto-encoder such that I can apply 1 generic threshold across all models to identify anomalies?


texts: how does word variety depend on total words?

texts: I want to compare the word variety of several books. But some are short, while others are long. So how can I correct for the fact that longer books will generally contain a larger number of unique words. I tried simply dividing the number of unique words in each book by the number of total words in each book. But I think that was overdoing it, because now the shorter books appear to have the most variety. What is the best approach to this?


texts: Are there any python based data visualization toolkits?

texts:
Data visualization is an important sub-field of data science and python programmers need to have available toolkits for them.

Is there a Python API to Tableau?

Are there any Python based data visualization toolkits?


texts: How to work with string data with a lot of NAs in an aggregation task with R

texts:
I have recently taken up a data wrangling task and inherited a data from my predecessor which contains keyphrases extracted by hand like the following :

data1 = data.frame (ID = c(1,2,3,4,5),
               Date = c(20081011,20081011,20081011,20081011,20081011),
               name = c("John Doe1", "John Doe2", "John Doe3","John Doe4","John Doe5"),
               numeric= c(1,1,1,0,2),
               keyphrase1 = c("mutual interests","support democratic development","fundamental political reforms","minimum conditions not in place","friends"),
               keyphrase2 = c("trade",NA,"economic reforms","elections","speak same language"),
               keyphrase3 = c("justice",NA,NA,"referendum","same values"))

And I have a dictionary within which these key phrases are assigned to different labels, yet again by hand, like the following:

dict = data.frame (Conditionality = c("same values","minimum conditions not in place","speak same language",
                                  "economic reforms","fundamental political reforms","economic reforms"),
                   Interest = c("trade","mutual interests",NA,NA,NA,NA),
                   Politics = c("justice","referendum","elections",NA,NA,NA))

So my task is to create binary variables in the data1 for each label in the dict data and create a data like the following:

data2 = data.frame (ID = c(1,2,3,4,5),
               Date = c(20081011,20081011,20081011,20081011,20081011),
               name = c("John Doe1", "John Doe2", "John Doe3","John Doe4","John Doe5"),
               numeric= c(1,1,1,0,2),
               keyphrase1 = c("mutual interests","support democratic development","fundamental political reforms",
                              "minimum conditions not in place","friends"),
               keyphrase2 = c("trade",NA,"economic reforms","elections","speak same language"),
               keyphrase3 = c("justice",NA,NA,"referendum","same values"),
               Conditionality = c(1,0,1,1,1),
               Interest = c(1,0,0,0,0),
               Politics = c(1,0,0,1,0)
               )

As you can see the new variables indicate whether one or more keyphrases in the data1 fall under a particular label in the dict data or not.

The problem is the actual data1 is relatively large and riddled with NA's. So far, I have tried transforming both the data1 and the dict to long format, get rid of the NAs in the process and merge by phrases using dplyr join functions, merge function from the base. They did not work very well. So any suggestions on how can I go about it ?

The second problem is when I am done with the task, I am supposed to deliver a wide format data and as you can imagine once I transpose the data to long, I can not transpose it back to wide because not each rows, naturally, uniquely identified. Any suggestion would be highly appreciated !


texts:
I am using machine learnig to measure probability for the outcome of tennis matches. If the winer is 1 that means that p1 won otherwise p2 won. in Columns LG, SVC, RF and NN there are probaiblities from the models. I am wondering why the NN probability is almost always really different. Should I substract the NN value from 100 and if yes, why?
#LOGISTIC REGRESSION
clf_A = LogisticRegression(penalty='none', random_state=420, solver='sag')
m1 = RFECV(clf_A,scoring='accuracy')
m1.fit(X_train,y_train)
df_test['accuracy'][0] = round(m1.score(X_test,y_test)*100,2)
df_output['LG'] = m1.predict_proba(X_test)
df_output['LG'] = round(df_output['LG']*100,2)

#SVC
clf_B.fit(X_train,y_train)
df_test['accuracy'][1] = round(clf_B.score(X_test,y_test)*100,2)
df_output['SVC'] = clf_B.predict_proba(X_test)
df_output['SVC'] = round(df_output['SVC']*100,2)

#RANDOM FOREST
clf_C = RandomForestClassifier(criterion='gini',min_samples_leaf=3,min_samples_split=20)
m3 = RFECV(clf_C,scoring='accuracy')
m3.fit(X_train,y_train)
df_test['accuracy'][2] = round(m3.score(X_test,y_test)*100,2)
df_output['RF'] = m3.predict_proba(X_test)
df_output['RF'] = round(df_output['RF']*100,2)

#NEURAL NETWORK
network = models.Sequential()
network.add(layers.Dense(units=64, activation='relu', input_shape=(len(X_train.columns),)))
network.add(layers.Dense(units=32, activation='relu'))
network.add(layers.Dense(units=1, activation='sigmoid'))

network.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

es = EarlyStopping(monitor='val_loss', mode='min', verbose=0, patience=500)
mc = ModelCheckpoint('/content/best_model.h5', monitor='val_loss', mode='min', verbose=0, save_best_only=True)

history = network.fit(X_train, y_train,
                epochs=1000, verbose=0, batch_size=512,
                validation_data=(X_test, y_test), callbacks=[es, mc])

saved_model = load_model('/content/best_model.h5')
df_output['NN'] = saved_model.predict(X_test)
df_output['NN'] = round(df_output['NN']*100,2)


texts:
I am currently working on GANs.
I am trying to learn probability distributions, but I have troubles with the architecture of the GAN.
These are my classes for the discriminator and generator:
class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(800, 400),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(400, 200),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(200, 100),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(100, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        output = self.model(x)
        return output

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(50, 100),
            nn.ReLU(),
            nn.Linear(100, 200),
            nn.ReLU(),
            nn.Linear(200, 400),
            nn.ReLU(),
            nn.Linear(400, 800)
        )

    def forward(self, x):
        output = self.model(x)
        return output

With these classes I got negative values when I sample new data, but I am learning probability distributions, so I need values between 0 and 1. I tried to add a Sigmoid() at the end of the sequential model in the Generator, but then my GAN is not learning at all.
Do you have any suggestion?
Thanks a lot!


texts: What are the differences between Reinforcement Learning (RL) and Supervised Learning?

texts:
What is the difference between Reinforcement Learning (RL) and Supervised Learning?

Does RL hava more difficulty in finding a stable solution?

Does Q-learning have more difficulty in finding a stable solution?

Does getting stuck in a local minimum happen more in supervised learning?

Is this figure correct saying that Supervised Learning is part of RL?


texts: Database for storing nested s3 objects from broom, sweep, or tidy in R

texts:
I'm working on a product that requires automated forecasting of time-series data in R & Shiny. Using R's sweep, broom, & tidy packages, I get these 'tibbles', which are like dataframes with nested objects. See below.

In order to scale the product, I have to figure out where to store my data - currently keeping them in local .RData files. Need to separate the data storage from the application.

Is there a way to reliably store this data in it's native format, or do I have to parse each of these nested objects to separate tables for a traditional database?


texts: Count the number of false positives with respect to the first class

texts:
Can anybody tell me the formula how to find the number of false positives with respect to the first class?

where $y$ is the truth / target and $a(x)$ is the prediction


texts: Why model.fit_generator in keras is taking so much time even before picking the data?

texts:
In Keras, I am training a model as in below

model.fit_generator(data_generator(),
                samples_per_epoch = count,
                validation_data=(x_val, y_val),
                nb_epoch=50,
                callbacks=getCallBacks(),
                verbose=1)

In data_generator function, I am printing few debugging statements.

On running above fit_generator function, it takes really long time to print what I had put in the data_generator().

Is there any series of steps which keras takes before picking out the data batch for the training which makes the process so slow or there is some other caveat?


texts: Difference between Factorization Machines and PCA?

texts:
Factorization Machines (FMs) are a means to express the high dimensional data into lower dimensions, despite the original data being sparse.

How is it different from PCA which itself is a dimensionality reduction technique?

Are there pros-cons of either approach?


texts: How can I detect if an image was photoshopped?

texts:
I would like to check JPG files if they were manipulated to change the content.

What I consider NOT photoshopped:

Cropping
Rotating
(Scaling)
Image resolution
Automatic changes smartphones might make

What I consider photoshopping:

Adding a new image on top of parts of the old image
Changing text of a part of an image

How can this automatically be checked?

(And: Are there ready-to-use libraries for it?)


texts: Scientific name for the problem given in the description. (In machine learning what kind of problem it is termed as)

texts:
Features marked as boolean values (around 50,000) , need to score a set of fixed output (around 25,000).

I have wikipedia topics relevant to a particular url as features. And the output is the advertisement keywords which will perform well on those url. If I have the training data, i.e. advertisement keywords performance on urls. Given a url (I can extract the features i.e. relevant wikipedia topics), I need to predict which keyword might perform well on that url.

What is the scientific name for such kind of problems in which there is a big boolean input set and I need to score a big fixed output set. Can you name few algorithms for such kind of problems.


texts: Classifying sequences of mouse clicks with different number of clicks per sequence

texts:
I have a series of sequences, each sequence contains multiple mouse click with the following features: [x coordinate on screen, y coordinate on screen, duration of click], and each sequence is labeled with 1 of four classes which represents the user that did the sequence of clicks. I need help determining the right model/approach for this problem. Here's what I've tried so far:
 1. RNN with LSTM where I had to pad the sequences with zeros since they're not equal in length
 2. Draw an image with points on the coordinates clicked, and color coding the points based on the duration of the click, generating an image for each sequence, then running the images through a CNN and classifying them depending on the points

Both of these yield ~40% accuracy on submission. I need ideas/different approaches to the problem. Thank you in advance!


texts: Fuzzily join two large sets of postal addresses

texts:
I have two tables of postal address information - the one is about 2 million records, the other roughly 40 million. They have quite bad quality, and also are not quite compatible with each other (different conventions in both sets, some fields cut off in an impractical way... - in other words, Real World Data).

They may not be the largest ones around, but compared to the available hardware, they are non-trivial (I cannot simply spin up a lot of worker nodes, or do a simple $O(n²)$ loop...).

I wish to join-update the two tables on the postal address. Something like

select table1.id, table2.id
from table1, table2
where fuzzy_match([table1.street, table1.city, ...],
                  [table1.street, table1.city, ...]) > $MINIMUM_DISTANCE;

The match is then used to store table2.id as a reference in table1.

Obviously this is only pseudo-code. I'm looking for a tool to do this (either some existing solution, or a library to use in our application, or hints towards a non-naive algorithm or data structure to "DIY" it).

I need to do this at least once/initially over the full data, and then periodically for deltas/updates.

This is the technical background:

Linux server
Oracle RDBMS
The programming environment is primarily Ruby-based, but other tools and languages can be installed if open source.

I can think of the following options:

Oracle Text contains a fuzzy search. But this cannot be used for non-technical reasons outside of the scope of this question.
Elasticsearch has a fuzzy query based on Levenshtein. I'm in the process of getting my data into that and trying it for fitness and performance.
Program it myself in Ruby or other languages, although doing it from (almost) first principles does not seem too appealing to me - I'd hope that problem has been solved somewhere already.
Online solutions or closed source non-free solutions are not applicable, e.g. I cannot upload any of the data to Google Maps or whatever along that line.

Do you have any hint on how to approach this? I'm happy for tool/library suggestions, or a suggestion for some data structure, an algorithm which I can then research/implement myself.

I am aware of How to do postal addresses fuzzy matching? and will probably also try out  https://github.com/openvenues/libpostal as a base, but am afraid that that is mostly meant to be used for individual (?) addresses, so I'll run into $O(n^2)$ problems again...


texts: What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

texts: I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.


texts: Four parameter self-starting function based on SSfpl

texts:
I am currently working with a self-starting function for four parameters which I based on SSfpl but with a different formula.

This is the formula for my self-starting function:

(b1 * ((b2 * x)^b4)) / (1 + ((b2 * x)^b4))^(b3 / b4)

The code below is for SSfpl with a formula A + (B - A)/(1 + exp((xmid - input)/scal)):

ir <- as.vector(coef(lm(x ~ I(log(prop/(1-prop))), data = xy)))
pars <- as.vector(coef(nls(y ~ cbind(1, 1/(1 + exp((xmid - x)/ exp(lscal)))),

data = xy,

start = list(xmid = ir[1L],
lscal = log(abs(ir[2L]))),
algorithm = "plinear"))
value <- c(pars[3L], pars[3L] + pars[4L], pars[1L], exp(pars[2L]))

I can't figure out how to set up the code above properly for my self-starting function. Any help is highly appreciated. Thank You.


texts:
I would consider myself a journeyman data scientist.  Like most (I think), I made my first charts and did my first aggregations in high school and college, using Excel.  As I went through college, grad school and ~7 years of work experience, I quickly picked up what I consider to be more advanced tools, like SQL, R, Python, Hadoop, LaTeX, etc.

We are interviewing for a data scientist position and one candidate advertises himself as a "senior data scientist" (a very buzzy term these days) with 15+ years experience.  When asked what his preferred toolset was, he responded that it was Excel.

I took this as evidence that he was not as experienced as his resume would claim, but wasn't sure.  After all, just because it's not my preferred tool, doesn't mean it's not other people's. Do experienced data scientists use Excel?  Can you assume a lack of experience from someone who does primarily use Excel?


texts: [Guidance Needed]Research in data science

texts:
IMAGINE

You're a research intern and you're an undergraduate student. Have
some experience in data science and now new to research.

Your task is to conduct research on vision transformers. Oh and by
the way, you're new to transformers concepts.

You first learn transformers and make your fundamentals strong. You
implemented and checked vision transformers original code. All fine!

You even checked and ran current SOTA papers in vision transformers.
Now you have gained enough knowledge to conduct your own research.

THAT'S WHAT I THOUGHT UNTIL.....
ISSUE

I am unable to develop good ideas in my mind and obviously unable to implement by myself.

I tried and tried and tried, which only leads to big bad errors.

I thought maybe I understand other author's codes and practice their
implementation to get hang of it. Understanding the code itself takes
DAYYSSS(sometimes weeks).

I seriously don't understand if there's any right way to do this.
Sometimes I wish I had proper guidance to conduct research.

I did come up with various strategies, but nothing works. It's been
almost 2 months and I feel depressed.

Any suggestions will be highly
appreciated.!


texts: What is the difference between bootstrapping and cross-validation?

texts:
I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I'm aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see the main difference between them in terms of performance estimation.

As far as I see, bootstrapping is also producing a certain number of random training+testing subsets (albeit in a different way) so what is the point, advantage for using this method over CV? The only thing I could figure out that in case of bootstrapping one could artificially produce virtually arbitrary number of such subsets while for CV the number of instances is a kind of limit for this. But this aspect seems to be a very little nuisance.


texts: Scaling label encoded values for Linear Algorithms

texts:
I have encoded categorical variables to numerical values. As we know that for feeding values to Linear Algorithms like SVM or KNN, we scale the values for columns having large variations.

I have three label encoded columns, one of which has unique values from 1-3, another has 1-150 and another has 0,1,-1 where -1 represents missing values. How will the MinMaxScaler affect here? Or it is not needed to scale these columns?
If not, how can I avoid these specific columns? Generally we scale the whole dataframe.

P.S. I'm using LinearSVC algorithm.


texts: Datamodel for cluster analysis terms & segmentation

texts:
Hey fellow data lovers!

I have a data modelling problem for a clustering analysis that I can't wrap my head around. Perhaps I'm thinking to advanced and I should simplify my analysis.

I have 2 data sets:

Persons with their properties:

PersonId
Gender
Age
Country
Etc.

Conversations and their properties:

ConversationId
Sentences of the conversations
Keyword of a sentence + it's frequency
Polarity of the sentence (pos, neg or neutral)

I would like to cluster terms that have the same polarity for a group of people. Eg. male persons between 20 and 35 speak positively about economy and neg about privacy
Eg. male persons from BE, NL and DE speak neg about climat changes.

Problem is that I need to cluster multiple terms for an undefined group of people. For one term (eg. privacy) it would be 'easy' to determine the properties of the different segments as this is a classification problem. As I would like to cluster multiple terms together, I'm strugling to model my data as this means I have multiple records per person (they can talk about multiple terms).

Assumption: you can assume I have one record per person per term.


texts: Compressing profiles with a large number of dimensions

texts:
I 'think' this is a related question, but not sure how to apply it.

I'm trying to build out a very crude recommendation system using Amazon ML, Facebook likes, and historical actions.

So lets say we have a number of users within a system that promotes products within several categories. To better predict which categories of items to present to the user, we will consider their past interactions with specific items, and the past interactions of other users who share a similar profile. The profile consisting of basically the users Facebook likes data and some demographic info.

I'm unsure of how to distill the Facebook likes data in a way that lets me make meaningful comparisons between users.

I'm sure its obvious, but I'm completely new to machine learning, and data science in general. I'm currently limited to the capabilities of Amazon ML. Let me know if the question needs more clarification, constructive criticism is appreciated.

Edit:
As @liangjy pointed out, the solution to the recommendation system in general will be to use the collaborative filtering technique. This is most useful when their is sufficient data to link users based on their individual actions. Because we do not have enough/any data on new users, we are trying to use additional data (Facebook likes in this case) to help create that initial link. Where I'm stuck at is the vast number of profile/sites any one user may have liked. We could be comparing the likes of one user against tens of thousands of possibilities the rest of the users have presented. What is the best way to make this disparate data into something manageable?

I've considered taking a subset of users and pulling the top n sites (1,000?) from each demographic groups (age/gender). Then compare all other users to this base set, and create sub-groups based on their relationships to those sites. However, something about this approach feels like it would be skewed. I'm not sure what, but I'm pretty sure I won't get the results I'm looking for.


texts: Сlassification using convolutional layers

texts:
I have a simple neural network of 2 layers, recognizing 10 classes:

self.layer1 = nn.Sequential(
    nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2))
self.fc1 = nn.Linear(14 * 14 * 32, 10)

How can a classification be made using the convolutional layer in this case?


texts: calculating the precision and recall for a specific threshhold

texts: I have a logistic regression model in which I calculate the tpr, fpr and thresholds using the roc_curve. After looking at the accuracy rates for different thresholds, I found the most optimal threshold to be 0.63. I have been told that I need to calculate the new precision and recall based on the most optimal threshold which in this case is 0.63. Is there any way to this calculation is python?


texts: update model prediction for each day closer to the event

texts:
I want to predict whether people will renew their yearly subscription.  I want to make this prediction though for each user on every day of their subscription up until the day before the subscription ends (i.e., make predictions for each user on day 1, 2, 3... 364).

A simple approach is to obtain training data by randomly sampling users at some point in their history (e.g., look at information we knew about user 1 when they were 300 days away from renewal; look at what we knew about user 2 when they were 23 days away from renewal, etc) and then build a model with a "days away" variable.

One issue with this approach is that unless you have a ton of data, the model won't perform as well on any given "days away" (e.g., 30 days before) than a model in which all data was trained with same "days_away" (all users data was limited to what we knew about them exactly 30 days before they were up for renewal).

I'm curious know about other people's thoughts about how to best train a model(s) that can update its prediction each day as the prediction-event gets closer in time. Are there better ways to approach it than the approach outline above?


texts: Train/Validation/Test split and K-fold Cross Validation

texts:
I have a dataset that I have split in train, validation and test subsets. I want to evaluate several CNN architectures and hyperparameters so I have trained several models with different hyperparameter combinations and evaluated them on the validation set.
For the final decission which CNN architecture performs best, I want to take the best hyperparameter combination for each architecture determined in the previous step and perform k-fold Cross Validation on the combined training and validation dataset for each architecture. I get the average accuracy as an estimate how good the final model performs and how robust wrt. data variability the model is.
Finally, I train a model with the train and validation dataset and evaluate it on the test set.
Does my approach make sense? I cannot use k-fold Cross validation for hyperparameter search as it is computationally very expensive.


texts:
I would like  train a classifier to spot news articles which would spot articles that announce an event. The issue is that I do not have a large pre-labeled dataset for this task (I only have 200 examples). So here are my two questions:

Have you heard of a training set labeled for such task
I heard of few shot learning that can be helpful to train a classifier with few example would it be applicable to this case and is there any library/reading you would recommend.

Thanks in advance


texts: Is it possible to change the sequence of convolution layer and pooling layer?

texts: Many CNN architectures sequentially consist of convolution layers and pooling layers. I just wonder if it is possible to change the sequence of convolution layers and pooling layers. If not, could you explain the reasons why a convolution layer precedes to a pooling layer?


texts:
Long short-term memory is a recurrent neural network architecture introduced in the paper Long short-term memory.

Can you please tell me where the name comes from?

("Memory", as the network can store information because of the recurrence - but where does the "Long short-term" come from?)


texts: What is the best way to encode an arbitrary collection of strings into int categorical variables?

texts:
I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm.
The problem is I don't have a prior list of the categories, so that I can't just define a dictionary or mapping function before hand.

Say for example that I am using food labels - my current data has the following labels:
['Steak','Potatoes','Soup'], but it is possible that later, I will gate data with the labels 'Asparagus' or 'Chow mein', and I have no way of knowing the list of all potential labels before hand. Moreover it is possible that some of the incoming labels are proper names or strings that are idiosyncratic and not part of any standard vocabulary, e.g. 'Double Super Mac-Whopper'.

I thought of simply building my own hash map, but then I would have to put a lot of effort into saving and versioning the resulting map to maintain consistency across experiments and later in production.

I tried using the int.from_bytes function in Python 3, but it gives wildly varying int sizes (I think because it is using string length):

> int.from_bytes('steak'.encode('utf-8'),'little')
461195539571
> int.from_bytes('milk'.encode('utf-8'),'little')
1802266989
> int.from_bytes('Bok Choy'.encode('utf-8'),'little')
8750327238520172354

I looked at the sklearn categorical encoders (preprocessing.LabelEncoder() or sklearn.feature_extraction.FeatureHasher), but they all seem to require knowledge of the number of categories before hand (by having to specify a dictionary or fitting an encoder to the available data, etc...)

I thought about using some word embeddings like word2vec, but they return pretty large vectors, and all I need is an int, and I don't really care about semantic similarity etc...(i.e. using a word embedding is overkill).

Is there some sort of preprocessing utility from and ML library, or some publicly available string to int hash map that is stable that I can use?


texts:
I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this:

    pos product sales   week
0   1   car     250      1
1   2   tank    400      2
2   2   car     300      1
3   1   tank    500      2

The goal is to normalize the data between 0,1 for each point of sale and product, e.g the minimum and maximum relate to the minimum and maximum of sales within a specific product and a specific point of sale. I guess I can just create a column for each combination (example below) and then normalize each column but I'm looking for a more elegant solution.

    pos     1         2
    product car tank  car   tank
week
1           250  0    300     0
2           0    500    0   400

Thanks,
Al


texts:
Currently, I use image files and transform them into a *.npy file(saved as a numpy array) as training data. At present this training data set is nearly 3GB.
Now I have more image files, so the training data set will become even larger, maybe up to 40 GB. And I am afraid that the *.npy files can not save this big of a data file.

Is there another possible way to store such a large file?

Since I use Keras to build a neural network model, is it acceptable to split the training data into small parts so as to train the model without having to use all the training data?

Does this sound like a reasonable approach?


texts: DTW - What is fan-in nodes and why not just take the nearst points from each timeserie?

texts:
It has been a while that I am trying to understand the dtw (dynamic time warping) algorithm but I steel can't figure out the loginc behind this algorithm..

In many references it's explained that to compute the path between to time series I need to implement this recursivity:

I have many questions here and I hope some one can give me an answer:

What we mean exactly by the path between two time series ?
What is the (cost path matrix described here:

Why not just compute the distances between a given point of t and other points of r and select the closest one an repeat this until matching all points from t to points from r ??

Thanks in advance for any helpfull answer / comment.


texts:
I am fitting a model with 100,000 samples x 10 features (6 ints and 4 floats), using SKLearn KernelRidge:

model = KernelRidge(kernel='linear')

Looking at the task manager, 'Python' process takes ~40GB.

Can you please explain why is there such a high demand?
What kind of matrix is built in the background?


texts: Is it possible to calculate gradient of a filter applied on the objects (that censor them)?

texts:
I want to find an optimal censoring function, that removes objects from a given set, as to maximise the set's quality.
Suppose I am given a data set consisting of N objects, each is represented by a fixed number of numerical features.
I have a data quality function, that acts on the whole data set, and is differentiable with respect to all objects' features. The function has no parameters.
I also have a second, censoring function which I want to train, that acts on a single object and returns a value that represents the relevance of the object to the data set. The function is parametrized and is differentiable wrt the object's all features and its parameters.
I want to find optimal values of the parameters of the censoring function, so it filters out the objects from the data in such a way as it maximises the data quality function.
Of course, I can use non-gradient methods, such as genetic algorithms, but I ask about the application of a gradient descent method. I can't simply compute the derrivative of the data quality function wrt parameters, because the relationship is discrete. But maybe there is a way to re-phrase the problem in such a way, that it becomes continuous.


texts: Are there labeled multivariate time series data sets where only a subset/partial number of time series is relevant for each class?

texts:
I am looking for a data set of multivariate time series data which contains classes that only require a subset of time series for their identification. Assume a human activity recognition data set with sensors mounted on different body parts. Not for every activity, I need to consider all sensors. For instance, a sensor mounted on the leg is not relevant for detection of a "hand shake".
Does anybody know of such a data set? I need it for research purposes but cant find one. I think in real life data sets there are often difference relevances of attributes for each class. For instance, monitoring a whole system where some classes are only related to smaller parts of the system and other classes to the whole system. I think such data sets are very interesting. For any ideas or data sets, I am very thankful!
Thanks in advance!


texts: ValueError: Error when checking input: expected conv2d_1_input to have shape (30, 30, 3) but got array with shape (30, 30, 4)

texts:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import tensorflow as tf
from PIL import Image
import os
from sklearn.model_selection import train_test_split
from keras.utils import to_categorical
from keras.models import Sequential, load_model
from keras.layers import Conv2D, MaxPool2D, Dense, Flatten, Dropout

data = []
labels = []
classes = 43
cur_path = os.getcwd()

#Retrieving the images and their labels
for i in range(classes):
    path = os.path.join(cur_path,'train',str(i))
    images = os.listdir(path)

    for a in images:
        try:
            image = Image.open(path + '\\'+ a)
            image = image.resize((30,30))
            image = np.array(image)
            #sim = Image.fromarray(image)
            data.append(image)
            labels.append(i)
        except:
            print("Error loading image")

#Converting lists into numpy arrays
data = np.array(data)
labels = np.array(labels)

print(data.shape, labels.shape)
#Splitting training and testing dataset
X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=42)

print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)

#Converting the labels into one hot encoding
y_train = to_categorical(y_train, 43)
y_test = to_categorical(y_test, 43)

#Building the model
model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu', input_shape=X_train.shape[1:]))
model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(rate=0.25))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Dropout(rate=0.25))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(rate=0.5))
model.add(Dense(43, activation='softmax'))

#Compilation of the model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

epochs = 15
history = model.fit(X_train, y_train, batch_size=32, epochs=epochs, validation_data=(X_test, y_test))
model.save("my_model.h5")


texts:
I struggle with figuring out how to proceed with taking PCA into production in order to test my Models with unknown samples.
I'm using both an One-Hot-Encoding an an TF-IDF in order to classify my elements with various models, mainly KNN.
I know i can use the pretrained One-Hot-Encoder and the TF-IDF encoder in order to encode the new elements in order to match the final feature Vector.
Since these feature vectors become very large i use an PCA in order to reduce the dimension of them.
This is an example of how i currently pre process the features:
if method == "pca_tfidf":
        df['tfidf'], featurenames= hero.clean(df['OpcodeString']).pipe(hero.tfidf, return_feature_names=True)
        X = df['tfidf']
        X = pd.DataFrame(item for item in X)
        if maxFeatures >= X.shape[1]:
            maxFeatures = X.shape[1]
        y = df["%n class"] # Split label from data
        ActualY = df["actual class"]

        Final_PCA = PCA(n_components= maxFeatures ,random_state=42)
        Final_PCA.fit(X)
        X_pca = Final_PCA.transform(X)

        total_var = Final_PCA.explained_variance_ratio_.sum() * 100
        printPCA(X_pca, total_var)

        X = pd.DataFrame(item for item in X_pca)

Now what i don't understand is how to proceed with new unknown samples. Since PCA is calculated in relation to every element in the train dataset, would I have to recalculate every component new with all new samples? Wouldnt that falsify with what values the model has been trained?
Can i just use Final_PCA.fit(X_new) with the TFIDF Values of the unknown samples in order to produce a feature vector which the model can then classify?
Additionally im questioning myself weather my approach is good at all. I have read a few times that PCA seems to not be great for categorical feature values. However i was able to achieve great results for my approach when i calculate the TFIDF and PCA first and then do the Test/Train split afterwards.
Thanks for any help!


texts: Choosing the best set of features when forced to choose M out of N available features

texts:
Given: N features that map to some label Y using a Neural Net(NN)- it's a classification problem.
Problem: I want to get away by using only a subset of features denoted by M, where M<N.
Now I am free to choose any of the $\frac{N!}{(M!*(N-M!))}$
available features to achieve this.
But I want the set of features that maximizes my prediction accuracy when I use these new features in a NN to map to label Y.
How do I do this?


texts: Best way for data preparation to have accurate prediction

texts:
I'm trying to experiment if an opportunity will win or lose in Azure Machine Learning studio. However, am still in Data preparation method.

In my Data base I have opportunity table and products table.

For example, one opportunity has multiple products. Should I deal with the many products and put them in one record?

Will it affect the prediction if we have duplicate records for an opportunity like (a) or it’s better to have one record per opportunity in order to feed it to ML studio. And if yes which one will be better approach (b) or (c).

Approach a

Approach b

Approach c

oppid |first product |first technology|2nd product |2nd technology
1      out-services   active directory   TRN-items Adobe Acrobat


texts: Updating Google News Word2vec Word Embedding?

texts:
Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ?

Google News Word2Vec - Word Embedding clearly helps us to come with a robust set of word vectors but it unfortunately cannot be used for most business case.
For example:

embeddings.most_similar('python')

[('pythons', 0.6688377857208252),
 ('Burmese_python', 0.6680365204811096),
 ('snake', 0.6606293320655823),
 ('crocodile', 0.6591362953186035),
 ('boa_constrictor', 0.6443518996238708),
 ('alligator', 0.6421656608581543),
 ('reptile', 0.6387744545936584),
 ('albino_python', 0.6158879995346069),
 ('croc', 0.6083582639694214),
 ('lizard', 0.601341724395752)]

This output is clearly not what we want. We could create a custom word2vec model using gensim library for this business case but it would not be exhaustive (vocabulary will be comparatively less). What is best practice in such cases ? Is is possible to update the weights of a pretrained Word Embedding model so that the word embedding also learns from domain text data?


texts:
I have the question which is mentioned in the above picture. It is trying to find the derivative of f with respect to weight matrix W1. Can any one help me how I can accomplish that?

I solved in this way :


texts: How to do predict a new sms to be spam or not?

texts:
I have trained a model for spam classification -

This is my code -

X_train, X_test, y_train, y_test = train_test_split(data['text'], data['label'], test_size = 0.4, random_state = 1)

cv = CountVectorizer()
cv.fit(X_train)
cv.fit(X_test)
X_train = cv.transform(X_train)
X_test = cv.transform(X_test)

model = LogisticRegression(solver='lbfgs')
model.fit(X_train, y_train)

After that I had also completed testing, it gives me an accuracy of about 97%
.

Now I want to add predict a new SMS/Email to be spam or not. What I am doing is -

new = 'Hey there you got a sale here on website'
new = cleanText(new)
cv.fit([new])
new = cv.transform([new])
model.predict(new)

It gives me an error

  ValueError: X has 4 features per sample; expecting 4331

Please tell me where I am going wrong?


texts:
I'm working through the Coursera NLP course by Jurafsky & Manning, and the lecture on Good-Turing smoothing struck me odd.

The example given was:

  You are fishing (a scenario from Josh Goodman), and caught:
  10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish
  ...
  How likely is it that the next species is new (i.e. catfish or bass)
  Let's use our estimate of things-we-saw-once to estimate the new things.
  3/18 (because N_1=3)

I get the intuition of using the count of uniquely seen items to estimate the number of unseen item types (N = 3), but the next steps seem counterintuitive.

Why is the denominator left unchanged instead of incremented by the estimate of unseen item types? I.e., I would expect the probabilities to become:

  Carp : 10 / 21
  Perch : 3 / 21
  Whitefish : 2 / 21
  Trout : 1 / 21
  Salmon : 1 / 21
  Eel : 1 / 21
  Something new : 3 / 21

It seems like the Good-Turing count penalizes seen items too much (trout, salmon, & eel are each taken down to 1/27); coupled with the need to adjust the formula for gaps in the counts (e.g., Perch & Carp would be zeroed out otherwise), it just feels like a bad hack.


texts: I have 32k black and white images. Want to do clustering on them

texts:
As the title says I'm trying to do clustering on a set of black and white images. These images are all 200x200 with black dots on a white canvas Example pics here (These are not actual photos from the data set. Just a general idea of what they look like). The idea is to hopefully find an underlying pattern when it comes to general shape of these images and hopefully cluster by that.

What I've done so far is turned each of my 200 x 200 image in a numpy array of size 40,000. Then put all the images together into a singly numpy array of 32k x 40k.

I'm kind of not sure where to go from there. What I did next was use scikit learn's TruncateSVD on my data set and set the paramater 'n_componenets=100' and fitted and transformed it, so now my data set of images are 32k x 100.

From here on out I don't know what's the best plan of action. Should I just start using k_means algos on my data set? and how would I visualize it?

Sorry if this is a lot. Any help/tips would be much appreciated it.


texts: Regularization term in Matrix Factorization

texts:
I'm trying to build a naive recommender system using latent factor model for MovieLens dataset. From the observed set of ratings I'm trying to build a model which will decompose the sparse matrix to N * K and K * M, where N is the number of users, M is the number of Movies and K is the number of dimensions in the latent space that I'm trying to learn. The loss function that I'm using:

$$
L = \sum_{u, i}(\mathbf{x}_u^T \cdot \mathbf{y}_i  + \mu + avg_u + avg_i -  r_{ui})^2
$$

$r_{ui}$ is the rating given by user $u$ to movie $i$, $\mathbf{x}_u$ is the latent representation of user $u$, $\mathbf{y}_i$ is the latent representation of movie $i$, $\mu$ is the average global rating, $avg_u$ is the average rating given by user $u$ and $avg_i$ is the average rating given for movie $i$.
Note: $avg$ rating here is adjusted as per the average deviation from the global mean rating.

If I consider regularization term as

$$
\lambda\sum_{u}||\mathbf{x_u}||^2 + \lambda\sum_{i}||\mathbf{y_i}||^2
$$

Model will tend to penalize more commonly occurring training users/movies, therefore popular movies tend to get penalized more with this regularization term compared to the movies that are there in the long tail.

Is it fine to consider the above regularization term or is there a better way to uniformly penalize items irrespective of their occurrence in training data?