texts: Value Updation Dynamic Programming Reinforcement learning

texts:
Regarding Value Iteration of Dynamic Programming(reinforcement learning) in grid world, the value updation of each state is given by:

Now Suppose i am in say box (3,2). I can go to (4,2)(up) (3,3)(right) and (1,3)(left) and none of these are my final state so i get a reward of -0.1 for going in each of the states. The present value of all states are 0. The probability of going north is 0.8, and going left/right is 0.1 each. So since going left/right gives me more reward(as reward*probability will be negative) i go left or right. Is this the mechanism. Am I correct? But In the formula there is a summation term given. So I basically cannot understand this formula. Can anyone explain me with an example?


texts: Convolutional neural network giving high confidence on wrong classification

texts:
I am doing a project on sign language recognition through CNN.

My Dataset

300 images for each alphabet and one special symbol for space. Images are taken from three individuals.
Each image has hand centered in white background.
Resolution is 640*480.
So in total 27 labels and 8100 images.

Samples Images

Code

I am using pretrained AlexNet in PyTorch and have changed fully connected layers according the need of the project. I am training for 10 epochs. Batch size is 32. Loss function is Negative Log Likelihood Loss and optimizer is Adam.

AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Linear(in_features=9216, out_features=1024, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=1024, out_features=512, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=512, out_features=27, bias=True)
    (7): LogSoftmax()
  )
)

Problem

The problem I am facing is that the model is giving very high confidence about 90-100 % on images that have no hand gesture in them or contain wrong hand gesture. I have tried adding dropout layers, training for more epochs but nothing seems to solve this problem. Is there something wrong with the model or the dataset needs to be changed/updated?


texts: Preprocessing and dropout in Autoencoders?

texts:
I am working with autoencoders and have few confusions, I am trying different autoencoders like :

fully_connected autoencoder
convolutional autoencoder
denoising autoencoder

I have two dataset , One is numerical dataset which have float and int values , Second is text dataset which have text and date values :

Numerical dataset looks like:

date ,        id ,             check_in , check_out , coke_per , permanent_values , temp
13/9/2017     142453390001    134.2       43.1        13         87                 21
14/9/2017     142453390005    132.2       46.1        19         32                 41
15/9/2017     142453390002    120.2       42.1        33         99                 54
16/9/2017     142453390004    100.2       41.1        17         39

           89

Any my text dataset looks like :

data              text
13/9/2017         i totally understand this conversation about farmer market and the organic products, a nice conversation ’cause prices are cheaper than traditional
14/9/2017         The conversation was really great. But I think I need much more practice. I need to improve my listening a lot. Now I’m very worried because I thought that I’d understand more. Although, I understood but I had to repeat and repeat. See you!!!

So my questions are:

Should i normalize my numerical data values before feeding to any type of autoencoder? if they are int and float values still i have to normalize?

Which activation function should i use in autoencoder? Some article and research paper says , "sigmoid" and some says "relu" ?

Should i use dropout in each layer ? like if my artichare for autoencoder looks like

encoder (1000 --> 500 -- > 256 ----> 128 ) --> decoder (128 --> 256 --> 500--> 784)

something like this?

encoder(dropout(1000,500) --> dropout( 500,256) --> dropout (256,128) )----> decoder(dropout(128,256),dropout(256,500),dropout(500,784))

For text dataset , If i am using word2vec or any embedding to convert text into vector then i would have float values for each word , should i normalize that data too ?

text ( Hello How are you  ) -- > word2vec(text) ----> ([1854.92002 , 54112.89774 ,5432.9923 ,5323.98393])

should i normalize this values or directly use in autoencoder ?


texts: Which F1-score is used for the semantic segmentation tasks?

texts:
I read some papers about state-of-the-art semantic segmentation models and in all of them, authors use for comparison F1-score metric, but they did not write whether they use the "micro" or "macro" version of it.
Does anyone know which F1-score is used to describe the segmentation results and why it is so obvious that authors do not define it in papers?
Sample papers:
https://arxiv.org/pdf/1709.00201.pdf
https://arxiv.org/pdf/1511.00561.pdf


texts: Unsupervised clustering of unstructured text by document type

texts:
I have 100,000+ PDF healthcare documents from which I have extracted text. I would like to cluster these documents by type (e.g. pathology report, doctor visit notes, prescription orders, etc.)

The format and structure of the documents are unknown ahead of time. However, we can assume that the name of the document type occurs explicitly somewhere within the first 50 words of each document. E.g. A pathology report will contain the words "pathology report" somewhere near the top.

The contents of the documents are otherwise irrelevant. I simply want to cluster documents together by document type.

I do not know the names or number of document types ahead of time. I would like the algorithm to automatically determine the best number of clusters.

I'm not very experienced in machine learning, but I am a competent programmer. What's the best way to attack this problem? Thanks.


texts: Any non Deep Learning python packages for sequence classification.?

texts:
Stats model or any other machine learning python packages for doing sequence classification(that can be multi class) and sequence prediction (Both next step and regression).

PS : Input data will be n number of sequences with their respective target (class/regression)

Also each sequences having finite number of events like E1,E2 ..


texts: Test for heteroscedasticity in time series

texts: I want to test heteroscedasticity in time series. The tools in python like: statsmodels.stats.diagnostic.het_breuschpagan require residuals as input obtained by fitting model to data. Since this kind of test relies on the goodness of the model trained. I want to test the heteroscedasticity on time series without training any model, directly on the data itself.  So I use McLeod.Li test in R to test it on raw time series. I analyzed individual features have heteroscedasticity. To remove heteroscedasticity, I added 1 to all entries in data (since it has 0 entries) and computed heteroscedasticity the pvalues moved to 0. Why?


texts: Decision Tree : how to determine target in a model with no labels?

texts:
I am studying classification algorithms using decision tree approach in Python.
I would have some questions on this topic, specifically regarding the target (y) in my dataset.
I have a dateset made by 20000 observations and a few fields:

Customer
recorded date
amount
status (if married or not)
children (if any children in the family)
nationality (if American or not)

And so on.
Most of these fields are binary (yes/no).
Based on this I would like to determine if this customer is trustworthy or not. As you can see, I have no label about trusting, but I have some initial information: for example the amount. If the amount is 0 or < 0, the customer has no money so he/she can be considered not trusted.
Then, I could consider status: if he/she is married, then it could be considered trustworthy, as there could be another salary to take into account.
And so on.
My doubt are in splitting my dataset, as it asks about y variable.
What would it be in this case? I have no explicit target..


texts: Finding a vector that minimize the MSE of its linear combination

texts:
I have been doing a COVID-19 related project. Here is the question:

N = vector of daily new infected cases
D = vector of daily deaths
E[D] = estimation of daily deaths

N is a n-dimensional vector, n is around 60. E[D] is another n-dimensional vector. Under certain assumptions, each entry of E[D] can be calculated as a linear combination of the entries of N.

We want to find the vector N such that the E[D] derived from N has least mean squared error when compared to actual D data. I think a gradient descent algorithm is needed here. However, I am not very familiar with gradient descent.

This seems to be a basic data science problem, but I am kind of lost right now. Does anyone has any idea about which algorithm should I dig into?


texts:
I have the following code written using the pandas library. I would like to know if there are ways to optimize the code.

for column in df:
 for index, row in df[column].iteritems():
    if type(row) == str:
        if 'R$' in row:
            n = row.replace('R$', '')
            n = n.replace(' ', '')
            n = n.replace('.', '')
            df[column].iloc[index] = float(n)

Just want to remove unwanted string parts.


texts: Including validation set in my code for a Linear SVM classifier returns a Type error

texts:
I'm using a predict function for a linear SVM classifier:
def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
"Train model using sklearn pipeline"
train_df = self.read_data(train_file, lower_case)
dev_df = self.read_data(dev_file, lower_case)
learner = self.pipeline.fit(train_df['text'], train_df['truth'])
# Fit the learner to the test data
test_df = self.read_data(test_file, lower_case)
# assuming dev_df is validset
dev_df['pred'] = learner.predict(dev_df['text'])
test_df['pred'] = learner.predict(test_df['text'])
return test_df, dev_df

And the predict function refers also to:
    def run_classifier(files: Tuple[str, str, str],
               method: str,
               method_class: Base,
               model_file: str,
               lower_case: bool) -> None:
"Inherit classes from classifiers.py and apply the predict/accuracy methods"
train, dev, test = files  # Unpack train, dev and test filenames
result = method_class.predict(train, test, lower_case)
method_class.accuracy(result)
# Plot confusion matrix
make_dirs("Plots")
print(result)
fig, ax = plot_confusion_matrix(result['truth'], result['pred'], normalize=True)
ax.set_title("Normalized Confusion Matrix: {}".format(method.title()))
fig.tight_layout()
fig.savefig("Plots/{}.png".format(method))

Where the dev set is not really taken into consideration. When I run predict, I receive this error: predict() missing 1 required positional argument: 'lower_case but I don't understand how to solve it.


texts: Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

texts:
I have read on the several answers here and on the Internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.

But I am confused that which two accuracies/errors amoung test/training/validation should I compare to be able to see if the model is overfitting or not?

For example:

I divide my data for 70% training and 30% test.

When I get to run 10 fold cross-validation, I get 10 accuracies that I can take the average/mean of. should I call this mean as validation accuracy?

Afterward, I test the model on 30% test data and get Test Accuracy.

In this case, what will be training accuracy? And which
two accuracies should I compare to see if the model is overfitting or not?


texts: What initial steps should I use to make sense of large data sets, and what tools should I use?

texts:
Caveat: I am a complete beginner when it comes to machine learning, but eager to learn.

I have a large dataset and I'm trying to find pattern in it. There may / may not be correlation across the data, either with known variables, or variables that are contained in the data but which I haven't yet realised are actually variables / relevant.

I'm guessing this would be a familiar problem in the world of data analysis, so I have a few questions:

The 'silver bullet' would be to throw this all this data into a stats / data analysis program and for it to crunch the data looking for known / unknown patterns trying to find relations. Is SPSS suitable, or are there other applications which may be better suited.
Should I learn a language like R, and figure out how to manually process the data. Wouldn't this comprimise finding relations as I would have to manually specify what and how to analyse the data?
How would a professional data miner approach this problem and what steps would s/he take?


texts: Does zero_as_missing parameter affects categorical features? LightGBM

texts:
Dealing with categorical features while training LightGBM model implies encoding them as integers and providing categorical_feature parameter with their indices or names.

LightGBM documentation says that:

  Tree decision rule works best when categorical features are presented
  by consecutive integers starting from zero

and

  all negative values will be treated as missing values

My question is: when zero_as_missing parameter is set, does it refer only to continuous features, and categorical will still be considered missing if their value is negative?

  zero_as_missing , default = false, type = bool

set this to true to treat all zero as missing values (including the unshown values in LibSVM / sparse matrices)


texts: Extremely high MSE/MAE for Ridge Regression(sklearn) when the label is directly calculated from the features

texts:
Edit: Removing TransformedTargetRegressor and adding more info as requested.

Edit2: There were 18K rows where the relation did not hold. I'm sorry :(.  After removing those rows and upon @Ben Reiniger's advice, I used LinearRegression and the metrics looked more saner. The new metrics are pasted below.

Original Question:

Given totalRevenue and costOfRevenue, I'm trying to predict grossProfit. Given that it's a simple formula totalRevenue  - costOfRevenue = grossProfit, I was expecting that the following code would work. Is it a matter of hyperparameter optimization or have I missed some data cleaning. I have tried all the scalers and other regressions in sklearn but I don't see any big difference.

# X(107002 rows × 2 columns)
+--------------+---------------+
| totalRevenue | costOfRevenue |
+--------------+---------------+
| 2.256510e+05 | 2.333100e+04  |
| 1.183960e+05 | 2.857000e+04  |
| 2.500000e+05 | 1.693000e+05  |
| 1.750000e+05 | 8.307500e+04  |
| 3.905000e+09 | 1.240000e+09  |
+--------------+---------------+

# y
+--------------+
| 2.023200e+05 |
| 8.982600e+04 |
| 8.070000e+04 |
| 9.192500e+04 |
| 2.665000e+09 |
+--------------+
Name: grossProfit, Length: 107002, dtype: float64

# Training

import numpy as np
import sklearn

from sklearn.compose import TransformedTargetRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=13)

x_scaler = StandardScaler()

pipe_l = Pipeline([
        ('scaler', x_scaler),
        ('regressor', Ridge())
        ])

regr = pipe_l

regr.fit(X_train, y_train)

y_pred = regr.predict(X_test)

print('R2 score: {0:.2f}'.format(sklearn.metrics.r2_score(y_test, y_pred)))
print('Mean Absolute Error:', sklearn.metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', sklearn.metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(sklearn.metrics.mean_squared_error(y_test, y_pred)))

print("Scaler Mean:",x_scaler.mean_)
print("Scaler Var:", x_scaler.var_)
print("Estimator Coefficient:",regr.steps[1][1].coef_)

Output of above metrics after training(Old Metrics with 18k rows which did not confirm to the relation)

R2 score: 0.69
Mean Absolute Error: 37216342513.01034
Mean Squared Error: 7.601569571667974e+23
Root Mean Squared Error: 871869805169.7842
Scaler Mean: [1.26326695e+13 2.14785735e+14]
Scaler Var: [1.24609190e+31 2.04306993e+32]
Estimator Coefficient: [1.16354874e+15 2.59046205e+09]

Ridge(After removing the 18k bad rows)

R2 score: 1.00
Mean Absolute Error: 15659273.260432156
Mean Squared Error: 8.539990125466045e+16
Root Mean Squared Error: 292232614.97420245
Scaler Mean: [1.57566809e+11 9.62274405e+10]
Scaler Var: [1.20924187e+25 5.95764210e+24]
Estimator Coefficient: [ 3.47663586e+12 -2.44005648e+12]

LinearRegression(After removing the 18K rows)

R2 score: 1.00
Mean Absolute Error: 0.00017393178061611583
Mean Squared Error: 4.68109129068828e-06
Root Mean Squared Error: 0.0021635829752261132
Scaler Mean: [1.57566809e+11 9.62274405e+10]
Scaler Var: [1.20924187e+25 5.95764210e+24]
Estimator Coefficient: [ 3.47741552e+12 -2.44082816e+12]


texts:
I have been provided with a text cleaning task and I am assuming this involves some amount of natural language processing.

I have  a collection of names which does not have any specific pattern and I need to separate individual names from that list.

Below is an example:

John Batista Barack Obama George W. Bush Serafim Batzoglou Vijay Pande Fei Fei Li Oussama Khatib Kwame Nkrumah John Oliver

As you can see there are two things peculiar with the paragraph.

1) space " " is the only delimiter, however, it is unusable as there are names which have middle names.

2) I have french, German, Spanish, Chinese and other ethnic sounding names written in English.

Now the challenge I am facing is to find names from this paragraph. I am very new to NLP and use Python and R for modeling work.

Any help will be appreciated.


texts: Evaluating Language Model on specific topic

texts:
I have finetuned a pretrained Language Model(GPT-2) on a custom dataset of mine. I would like a way of evaluating the ability of my model to generate sentences of a specific predefined topic, given in the form of either a single keyword(e.g. 'Computers') or a bag-of-words(e.g. 'Computers', 'Linux', 'Server'...).
For example given a LM, how relative are the outputs of the model to the topic specified by the word Computers?
What I have already tried: Generating a large enough number of sentences from the LM and taking the average cosine similarity between these sentences and the target topic(or every word in that topic we have more than one) as described here . I am not sure if this is a valid way to go and furthermore the cosine similarity between sentences yields poor results in many cases.
Thanks in advance for any help.


texts: Books/reviews/papers on recommending groups of items?

texts:
Looking for books (chapters?)/reviews/papers on the task of recommending a few (possibly non-independent) items.

Example: office supply shop recommending a person a pre-compiled package of items of different kind (pens/pencils/erasers/paper/...) based on one's and other users purchases and views. The total cost of the package may be limited; other constraints could be introduced as well.

Any hints?

Thanks in advance.


texts: When scaled conjugate gradient backpropagation is suitable for backpropagation?

texts:
I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled conjugate gradient backpropagation" for optimization in deep learning methods or in artificial neural networks?

Is there any case that using scaled conjugate gradient backpropagation is sufficient? Could you please provide some details when this method can be used and why it should be used?


texts: Updating Weight Using Updates on Related Data

texts:
Suppose
$$
x=Ay
$$

The $x$ is $M\times 1$, $y$ is $N \times 1$ and $A$ is $M\times N$

We have the data $x$ and would like to know what $y$ is.

However, the matrix $A$ is too large for pseudo-inverse. And thus we would like to approximate $A^{-1}$ using machine learning as it is possible to parallelize it.

Here for parallelization, we divide the given problem into:
$$
x^l = A^l y
$$
where $x = [x^1 , x^2,\dots,x^L]^T$ and similarly, $A = [A^1,A^2,\dots,A^L]^T$

So, $x^l$ is $\frac{M}{L}\times 1$ and $A^l$ is $\frac{M}{L}\times N$

The problem could thus be arranged as $$y = W^l x^l$$ and the $W^l$ could be trained using gradient descent.

$$
W^l_{i+1} = W^l_{i} + \eta\left(y_i-W^l_ix^l_i\right){x_i^l}^T
$$

Here, $W^l$ is $N \times \frac{M}{L}$ and $W = [ W^1,W^2,\dots,W^L ] $

Here we can directly use updation for every $l^\mathrm{th}$ block. However, as diffusion techniques converge to lower MSD, I would like to employ them.

However, the $W^l$ and $W^k$ aren't directly related and thus diffusion techniques can't be directly applied to them but will require matrix operations before combining them. However, if we could've afforded inverse matrix operations, we won't have moved to gradient descent.

Could you give a solution where we don't need to apply inverse operations for employing diffusion techniques?


texts: Regression Decision Tree - Normalize or Split into Ranges a continuos feature

texts:
I have in my dataset a feature named distances which ranges goes from 200 to 12000 (more or less). Since the other features have got values under 50 I need to do some transformations in distances.

The first thing that came to my mind is to normalise it, to keep this feature in a range from 0 to 10 for example. But with this approach I may have lot's of different value which may generate too many splits.

The other approach I consider is make ranges. For example

From 0 to 1000 => 1
From 1000 to 3000 => 2
From 3000 to 6000 => 3
From 6000 to 12000 => 4
//Or +6000 => 4

I believe this approach will be better since the decision three will have just 4 branches for this split.

Now my questions:

Is the correct approach to assign ranges on our own to a feature of
this sort, or is it better to just normalise and then  use some algorithm to set the ranges
for us?
In case we are the ones who should decide the ranges, is it correct
my example? Should be constant the range of the groups, or at least
lineal the relationship between the ranges and the unit assigned.

Something like this:

From 0 to 3000 => 1
From 3000 to 6000 => 2
From 6000 to 9000 => 3
From 9000 to 12000 => 4
//Or +9000 => 4

The reason why I followed the first approach is because distances has got more observations with lower values than high values. So I was trying to get a balanced feature with a similar amount of observations in each of the 4 categories.

With the second approach almost all the observations will be of type 1 or 2, since most of the distances are less than 5000 units. However I am not sure if this is something I should be concern about in Regression Decision Trees.


texts: Decision boundary in a classification task

texts:
I have 1000 data points from the bivariate normal distribution $\mathcal{N}$ with mean $(0,0)$ and variance $\sigma_1^2=\sigma_2^2=10$ with the covariances being $0$. Also there are 20 more points from another bivariate normal distibution with mean $(15,15)$ with variance $\sigma_1^2=\sigma_2^2=1$ and with the covariances being $0$ again. I used the least squares method to calculate the parameters of the decision bounday $\theta_0 + \theta_1 x_1 + \theta_2 x_2=0$, that is $$\theta = (X^T X)^{-1}(X^Ty)$$ where $y$ is a column matrix with labels $+1$ for points from the first class and $-1$ for points from the second.
The resuliting plot is as follows:

It is obvious that the decision boundary failed to be correct, as it passes right through class $-1$ and therefore it won't classify correctly future points that might stem from the same distribution. Now, there is the issue of why this is happenning. I understand that the main problem here is the imbalance of the data set, as there are $1000$ points from one class but only $20$ from the other. This, intuitively, makes sense.
What I want someone to help me with, if possible, is to understand how this imbalance problem is incorporated into the process of minimizing the least squares cost function $$J(\theta)=\sum_{n=1}^{200}(y_n-\theta^T x_n)^2$$
How does the fact that there are only $20$ points from the second class causes the minimzation task $\frac{\partial J(\theta)}{\partial \theta}=0$ to "fail"? How do the insufficient amount of these points causes this line to pass right through them? If there is some mathematical way to show me this, it would be nice, as I already got the intuition.


texts: I am looking for a 3D plotting and post analysis tool, specifically which can generate figures like the following. It is preferred the tool has a Linux distribution. The figure is copied from here.


texts:
I'm wondering how to implement this with pytorch built-ins. I've got a 3 dimensional input of uints called policy. Most of the entries are zero, and if I were to L1 normalize this I would have a (target) probability distribution.

I've also got the output of a linear layer, called 'logit', with the same shape as 'policy'. I must turn this into a probability distribution by taking the softmax, but only over the entries where policy is non-zero.

The loss is then -sum(log(logit_masked_softmax) * policy_normalized))

I've implemented this manually with the nn.functional module using boolean indexing. The problem is that I want to do this in batches, where a 4 dimensional tensor represents the batches of 3 dimensional inputs. I am convinced that there must be a built-in way to achieve this and it probably is also faster and more numerically stable.


texts: Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

texts:
The ones I've tried so far
Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN instead of Local.
Spylon-kernel: Kernels connects, but gets stuck in the initializing stage.
Apache Toree: I would've loved this so much only if it worked. Lots of language support, magics, incubated by apache. However, this kernel doesn't connect. Get's stuck on the "Kernel Connecting" stage.
Are we to never run Spark with the freedom and flexibility of Jupyter?
What curse is this?


texts: Wasserstein distance between Gaussian and the empirical distribution

texts:
Wasserstein distance between two gaussians has a closed form solution. Does the same hold for the distance between a Gaussian with a fixed variance(say 1) and the empirical data distribution?

Empirical data distibution defined as:
$$ p(x) = \frac{\sum_i \delta (x - x_i)}{n} $$

And the 1-d Gaussian with $\sigma^2 = 1$ and some unknown mean $\mu$
$$ q(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$


texts: Why not use loss value for stopping the training in Siamese Networks?

texts: I was building the training pipeline for a siamese network for one-shot learning. Everybody on the internet uses accuracy rather than loss for stopping training. Why use one-shot accuracy over loss value for stopping? Aren't the concepts of overfitting and underfitting relevant here or am I missing something fundamental in one-shot learning?


texts:
The constrained optimization problem in SVM is given by
min 1/2 ||w||^2
s.t y(i)(w^T x(i) + b >= 1 for all i
Now converting this to an unconstrained optimization problem gives the lagriangian L as shown in the picture
And while deriving this we also get an equation for w as shown in picture
I've been instructed to implement this using gradient descent and give reason if this is not possible
This is the approach I tried to follow:

Use gradient descent to minimize the lagrangian and get the minimum alpha vector
Then using equation for w which uses this alpha vector ...compute w vector
Also I've included b (intercept) in w itself .

But it's not working
I'm very new to ML and I couldn't understand all the math behind this. Help


texts: adding constraints in PuLP optimization problems in python? pyschedule required?

texts:
I want to create an optimal meal plan with minimum sugar intake for
7 days but the everyday diet plan should include food from 3 different categories. How do I add constraint that I could get food from 3 different food categories and give meal plan for 7 days?

my dataframe is as follows:

id      name               energy   sugar   Food_Groups
1       4-Grain Flakes      140     58.8    Breakfast
2       Beef Mince, Fried   1443    8.0     Meat
3       Pork                1000    3.0     Meat
4       cake                1200    150     Sweet
5       cheese              1100    140     Sweet
6       Juice               700     85      Drink
7       cabbage             60      13      vegetarian
8       cucumber            10      10      vegetarian
9       eggs                45      30      Breakfast

# Create the 'prob' variable to contain the problem data
prob = LpProblem("Simple Diet Problem",LpMinimize)
#create data variables and dictionary
food_items = list(df['name'])
calories = dict(zip(food_items,df['energy']))
sugars = dict(zip(food_items,df['sugar']))

food_vars =LpVariable.dicts("Food",food_items,lowBound=0,cat='Integer')

#Building the LP problem by adding the main objective function.
prob += lpSum([sugars[i]*food_vars[i] for i in food_items])

#adding calorie constraint
prob += lpSum([calories[f] * food_vars[f] for f in food_items]) >=
1800.0, "CalorieMinimum"
prob += lpSum([calories[f] * food_vars[f] for f in food_items]) <=
2200.0, "CalorieMaximum"

prob.writeLP("SimpleDietProblem.lp")

prob.solve()

print("Status:", LpStatus[prob.status])

EDIT:

I should not loop over prob.solve() 7 times to get 7 day meal plan.  Should I? Should I use LpAffineExpression to define a compound objective function ? or should I have an array of days and include it as constraint ? Please guide me

I tried the following code but it does not work

# Creates a list of the Food_groups
food_choices = list(df['Food_Groups'])

food_chosen = LpVariable.dicts("Chosen",food_choices,0,1,cat='Integer')

# Creates a list of the Days
Days = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']

days_menu = LpVariable.dicts("days_menu",Days,0,1,cat='Integer')

#updating my objective function
prob += lpSum([sugars[i]*food_vars[i]*days_menu[i]*food_chosen[i] for i in
food_items])


texts: how could RFECV ever give the same score to all number of features selected?

texts:
I built an XGB and ran RFECV over 250 features. After an hour or so, I plotted the grid_scores_.
All numbers of features are within 0.02, as clearly visible on the y-axis.

To me this plot would indicate that it doesn't matter if I have 1 variable or 250 in my model. Which has a lot of scary ramifications.
Has anyone ever come across this in their work? If so, what did you determine? How is this even possible?


texts: How to find the datasets for skill test (like java, python, c c++, etc.)

texts:
I want to extract the skills from a resume using Natural languge processing.

To train my model I don't any dataset. where can I get a sample dataset for extracting skills from resume?


texts: How to measure the performance of a domain adaptation /Transfer learning technique?

texts: Given that the performance you achieve depends on how far the target from the source domain is, how can you judge the performance of an algorithm?


texts: How to think about design of experiments in the context of more complex machine learning

texts:
I have experience with multifactor DoE, but in the context of optimizing treatment of a single or or a small number of populations. Are there any articles people recommend to help get my head around how to approach DoE for more complex machine learning models where there are more personalized forecasts / recommendations so that the experiment best informs future predictions.
This seems like it would be a combination of want to get more data away from a local optimum and to focus on areas of the model where predictions are relatively worse or above a certain threshold given the application of the model.
Anyways, I am looking to get some more information to help inform my thinking on the topic so any information would be greatly appreciated.


texts: How to combine two different random forest models into one in R?

texts:
I have two different dataset with same variables. I have built training models separately using random forest. Now i want to combine both these models. Could anyone tell me how can i this be achieved? Do we have something called combine() function in R?

Regards,
Arun


texts: How to detect vanishing and exploding gradients with Tensorboard?

texts:
I have two "sub-questions"

1) How can I detect vanishing or exploding gradients with Tensorboard, given the fact that currently write_grads=True is deprecated in the Tensorboard callback as per "un-deprecate write_grads for fit #31173" ?

2) I figured I can probably tell whether my model suffers from vanishing gradients based on the weights' distributions and histograms in the Distributions and Histograms tab in Tensorboard. My problem is that I have no frame of reference to compare with. Currently, my biases seem to be "moving" but I can't tell whether my kernel weights (Conv2D layers) are "moving"/"changing" "enough". Can someone help me by giving a rule of thumb to asses this visually in Tensorboard? I.e. if only the bottom 25% percentile of kernel weights are moving, that's good enough / not good enough? Or perhaps someone can post two reference images from tensorBoard of vanishing gradients vs, non vanishing gradients.

Here are my histograms and distributions, is it possible to tell whether my model suffers from Vanishing gradients? (some layers omitted for brevity) Thanks in advance.


texts: R plot(surv(), newdata=...) draws same lines many times - why?

texts:
I'm new to R and cannot make plotting work as desired. The problem is that R seems to draw the same four lines over and over again, redundantly. The details of the case I'm having are as follows.

I have a dataset:

> str(dataset)
'data.frame':   57641 obs. of  3 variables:
 $ duration : num  3 8 7 2 4 8 2 2 8 8 ...
 $ graduated: logi  FALSE TRUE TRUE FALSE FALSE TRUE ...
 $ group    : num  651 651 671 671 651 651 651 651 651 651 ...

Then, I fit a Cox proportional hazards regression model to it:

survObj <- Surv(time = dataset$duration / 2, event = dataset$graduated)
model <- coxph(survObj ~ group, data=dataset)

Next, following this example, I create a frame that would hopefully group the survival functions by the group number:

frame <- data.frame(group = dataset$group)

> str(frame)
'data.frame':   57641 obs. of  1 variable:
 $ group: num  651 651 671 671 651 651 651 651 651 651 ...

There's four groups in the data:

> unique(dataset$group)
[1] 651 671 652 681

Using this new frame, I create a fitted survival model:

fitObjGrouped <- survfit(model, newdata = frame)

Finally, I plot the thing:

color_set <- rainbow(4)
plot(fitObjGrouped, col=color_set)

The result has the correct lines, but drawn many times over each other:

As you can see, there's two red lines and two blue lines drawn last. They're the correct ones, one for each category, but closer observation reveals that there's a green or other color lines underneath each of them. When converting this to PDF the file size is 273 times larger than what it should be!

So the question is: why is R drawing the lines so many many times and how could I achieve correct model fitting and plotting at the same time?

Can somebody please help me to better understand the R commands I'm using? Thanks in advance!


texts: Definition of Type 1 and Type 2 Errors in Classification

texts:
Why are the type 1 and type 2 errors as defined in bankruptcy prediction, different from type 1 and type 2 errors based on confusion matrix?

In bankruptcy literature:

Type 1 error: predicting a bankrupt company as a nonbankrupt one.

Type 2 error: predicting a nonbankrupt company as a bankrupt one.

In confusion matrix:

Type 1 error: predicting a negative case (nonbankrupt company) as a negative (bankrupt) one.

Type 2 error: predicting a positive case (bankrupt company) as a negative (nonbankrupt) one.

Thanks in advance.


texts: Using Tensorflow object detection API vs Keras

texts:
I am new to machine learning. I am curious to know what is the difference between using Keras instead of TensorFlow object detection API. We need to manually configure hidden layers and input layer in Keras so what is the advantage to use Keras and how to know how many layers should configure to achieve object detection using Keras.

Please check two different types of implementation 1) Using Keras 2) Using Tensorflow Object detection API without Keras

Thanks !!!


texts: Neural Networks - Strategies for problems with high Bayes error rate

texts:
I am building a Neural Network for a binary classification problem where the Bayes error (lowest possible error rate) is probably close to 50%.

What makes the task easier is that I don't need to make a prediction for each observation of the test sample. I only want to make a prediction for the observations where the model has a fairly high confidence. However a high rate at which  predictions are made is better than a low one.

So far, I have used a standard neural network (feed-forward, cross-entropy loss, L2 regularization and sigmoid activation on final node). In the testing sample, I only take into account the observations for which the final node's value $(\hat{Y}_i)$ is outside of an interval of low confidence:
$$\text{predicted class}_i = \begin{cases}
 1 &\text{ if } \hat{Y}_i > 0.5 + a \\
 0 &\text{ if } \hat{Y}_i < 0.5 - a \\
 \text{NA} &\text{else}
\end{cases} \\
\text{where } a\in [0, 0.5] \text{ indicates the level of confidence required}$$

To tune the hyperparameters (including $a$), I have designed a metric that depends positively on:

Test-sample accuracy (only counting predictions different from NA)
Percentage of predictions that are different from NA.

I am not yet satisfied with the performance achieved with this approach, and I am sure that there are smarter ways to approach this, for example a custom loss function. Advices, links to articles, or even related search keywords are welcome.


texts: Interpreting machine learning coefficients

texts:
My dog show predictive tool is having some trouble with its neural net. Broadly, I start with a couple of factors--age, weight, height, breed (which is a set of dummy variables), a subjective cuteness score--and predict whether the animal will win best in a show (a binary variable). I've got a few hundred shows, so tens of thousands of rows. I'm stuck using the nnet package instead of neuralnet, but I can pretty easily create a decent neural net using
model <- nnet(Win ~ ., x=X_training, y=Y_training, size=30, maxit=500)

which works fine. I've tested it against the holdout test data and the AUC and confusion matrix looks good, and I can use it for predicting future winners. The problem is, how do I determine which variable is most important in predicting the outcome for each new dog show? When I run a prediction for future winners, I use the following code
winpred = predict(model, x, type='raw')

which provides a score between 0 and 1 of the likelihood of winning. What I'd also like is some way of identifying for each row which variable--age, weight, height, breed dummies, and cuteness--contributed the most and least to the score in my predictions. If I were doing this for a single dog, I would use LIME or something else based on this article, but unfortunately, I want to run this on a few hundred odd dogs at a time.
Basically, I want to be able to say to dog owners that their pup is doing well because of x, but what's most holding it back is y (not terribly helpful if it's breed, of course). It seems like there isn't a way to do this with neural networks.  Are there other machine learning techniques now that would allow this?  If not, any suggestions on other models that would work?
I was thinking of using OLS as a linear probability model or logit/probit, perhaps with LASSO or Ridge.  I could use my machine learning model to score dogs, but in the background pull from the regression to identify which variables contribute most strongly, positively, and negatively, to the score.  Logit/probit are better from a statistical perspective, but is there any easy way to interpret in R/RStudio, for a few hundred dogs at a time, which attribute contributed most strongly to their score?


texts:
I am working with package randomForestin R for predicting a price in an hourly basis.

For creating the model, after partitioning the dataframe, I use the following syntax:

  rf_regresor <- randomForest(x = train_df[c(all_predictors)],
                              y = train_df[,"target_variable"],
                              ntree = 50,
                              xtest = test_df[c(all_predictors)],
                              ytest = test_df[,"target_variable"],
                              mtry = 3,
                              importance = TRUE,
                              keep.forest = TRUE)

As the target variable to predict is a time series I would like for a specific purpose to train the model with all available data.

Let's say I have the training data from "2017-01-01" to "2017-12-31". I would like to use the whole period for predicting the first week of 2018, that is, from "2018-01-01" to "2018-01-07" before re-training the model with the latest inputs.

Is it possible for the xtest, ytest part from the randomForest to be removed so I cna use the whole period?

If not, would it be reassonable to use all the period but one day and use the last day to test (altough we would not care about the result for that specific day)?

Thank you


texts: Match users based on the content of their articles

texts:
I have users in my database that I would like to match up or group togetter based on the content of there articles. I cant seem to find how this kind of problem is being solved today. Any advice will help.

Available Data:

1) Each user's posts (anything written by them, like a blog).

2) Tags for each post (tags that the user gave to their post when they created it)

Goal:

1) Match/Group users based on available data.

2) Produce match percentage.

Attempt:

I matched people based on the number of exact matches of their tags.

Example: user1 has [car,honda,sports], user2 has [car,food].

This will give a 33% match.

As you can image this does not work very well. Most users have 20 tags but typically get a match percentage of 0% even if they are talking about similar things.

Problem:

Tags that have a clear relationship like CAR and HONDA are NOT matched.

Question:

How can I match/Group users based on tags or the content of there articles?


texts: 1 - Nearest Neighbor , dealing with same distance

texts: I know that as a question it may seem stupid, but in case it is applying K NN with k = 1 and I have two neighbors at the same distance , what is the best approach to carry out the classification ?


texts: Deep Neural Network - Backpropogation with ReLU

texts:
I'm having some difficulty in deriving back propagation with ReLU, and I did some work, but I'm not sure if I'm on the right track.

Cost Function: $\frac{1}{2}(y-\hat y)^2$ where $y$ is the real value, and $\hat y$ is a predicted value. Also assume that $x$ > 0 always.

1 Layer ReLU, where the weight at the 1st layer is $w_1$

$\frac{dC}{dw_1}=\frac{dC}{dR}\frac{dR}{dw_1}$

$\frac{dC}{w_1}=(y-ReLU(w_1x))(x)$

2 Layer ReLU, where the weights at the 1st layer is $w_2$, and the 2nd layer is $w_1$ And I wanted to updated the 1st layer $w_2$

$\frac{dC}{dw_2}=\frac{dC}{dR}\frac{dR}{dw_2}$

$\frac{dC}{w_2}=(y-ReLU(w_1*ReLU(w_2x))(w_1x)$

Since $ReLU(w_1*ReLU(w_2x))=w_1w_2x$

3 Layer ReLU, where the weights at the 1st layer is $w_3$, 2nd layer $w_2$ and 3rd layer $w_1$

$\frac{dC}{dw_3}=\frac{dC}{dR}\frac{dR}{dw_3}$

$\frac{dC}{w_3}=(y-ReLU(w_1*ReLU(w_2(*ReLU(w_3)))(w_1w_2x)$

Since $ReLU(w_1*ReLU(w_2(*ReLU(w_3))=w_1w_2w_3x$

Since the chain rule only lasts with 2 derivatives, compared to a sigmoid, which could be as long as $n$ number of layers.

Say I wanted to updated all 3 layer weights, where $w_1$ is the 3rd layer, $w_2$ is the 2nd layer, $w_1$ is the 3rd layer

$\frac{dC}{w_1}=(y-ReLU(w_1x))(x)$

$\frac{dC}{w_2}=(y-ReLU(w_1*ReLU(w_2x))(w_1x)$

$\frac{dC}{w_3}=(y-ReLU(w_1*ReLU(w_2(*ReLU(w_3)))(w_1w_2x)$

If this derivation is correct, how does this prevent vanishing? Compared to sigmoid, where we have a lot of multiply by 0.25 in the equation, whereas ReLU does not have any constant value multiplication. If there's thousands of layers, there would be a lot of multiplication due to weights, then wouldn't this cause vanishing or exploding gradient?


texts: Difference between test statistic and sample statistic?

texts: Test statistic and sample statistic are used to check statistical significance. How do I understand the procedure in background of reaching a conclusion about an effect-size estimate.


texts: I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this method in detail?


texts:
I want to use a pre trained Resnet 50 as a backbone for Unet model.
But the issue is resnet 50 is expecting the size of image as 197 x 197 3D channel but the image of mine is 128 X 128 x 1D channel. Is there any keras or tensorflow implementation on those.


texts: How will a rotation matrix affect contestants in machine learning contests?

texts:
Machine Learning contests like Kaggle usually layout the machine learning task in a human-understandable way. E.g. they might tell you the meaning of the input (features). But what if a Machine Learning contest doesn't want to expose the meaning of its input data? One approach I can think of is to apply a (random) rotation matrix to the features, so that each resulting feature doesn't have obvious meaning.

A rotation on the input space shouldn't change a model's ability to separate the positives from the negatives (using binary classification as an example) -- after all the same hyper plan (when applied the same rotation) can be used to separate the examples. What could be changed by the rotation is the distribution of each feature (i.e. when looking at a single feature's values across all examples) if a contestant cares about them. However, rotation is PCA-invariant, so if a contestant decides to work on the PCA-ed version of the input then the rotation doesn't change anything there.

How much do contestants reply on statistical analysis on the (raw, i.e. non-PCA-ed) input features? Is there any (other) thing that I should be aware of that a rotation can change for a contestant during such a contest?


texts: use machine learning to predict a next day of visit for customer


texts: How to use a NN architecture that is too big for GPU?

texts:
Initially posted in Stack Overflow.
I would like to implement a model which is actually 2 neural networks stacked together. However the size of these 2 architecture is too big to fit in GPU at the same time.
My idea was the following :

Load the first model and run it for 1 batch
Unload the first model from GPU, load the second model
Run the second model from the output of the first model
Unload the second model from GPU
Repeat for every batch

I actually don't need to train the first model, since it's pre-trained. But I need to train second model.
Is it possible to do something like this ? Is my approach correct ? What are the pitfalls I should be aware of ? What about performance ?
Edit
I already tried the idea of computing the output of the first model for the whole dataset at first, and then use it as input for the second dataset. However the output of the first dataset is really big, and I don't have the available space for storing the whole pre-processed dataset. That is why I wanted to do it each batch.
Edit 2
After the very nice answer from @Gal Avineri, just one more precision :
I would like to implements my architecture using only one GPU.


texts: In CNNs when do we update the kernel parameters using back propagation? Suppose I have batch size of 50 and training data of 1000. Do I back propagate after each batch has been presented to network or after each data sample?


texts: Large scale Autoencoder for friendship recommendation

texts:
I have a user friendship graph for about 30 Million users. I am trying to use an auto encoder [30 Million, 512, 512, 1024, Dropout(0.3), 512, 512, 30 Million]. But I am not learning anything.
Has anyone used Autoencoder for Friendship recommendation on a large scale graph?


texts: Confidence intervals for binary classification probabilities

texts:
When evaluating a trained binary classification model we often evaluate the misclassification rates, precision-recall, and AUC.

However, one useful feature of classification algorithms are the probability estimates they give, which support the label predictions made by the model.

These probabilities can be useful for a variety of reasons depending on the use case. When using these probabilities it would be useful to have a confidence interval rather than a single point estimate.

So, how can we estimate a probability confidence interval given that the misclassification error may not always serve as a proxy for the error between the estimated probability and the actual probability (which is often unknown)?

I've considered using brier score but I'm sure there is a better way. Can anyone point me in the right direction or offer your own insight?

For example, If I have classes [C0, C1] and my probabilities for a given  instance $(x^{(i)}, y^{(i)})$
 are {C0: 80, C1:20} then I will classify this instance as C0. Let's suppose that C0 is the correct class label, at this point the model has done it's job and made the correct classification.

I want to go another step further and use the probabilities {C0:80, C1:20} which could be useful for a variety of reasons.

Let's say C0 and C1 respectively represent a customer keeping and closing their account with a bank.

If we wanted to create an expected value $EV$ of dollars at risk of leaving the bank we could calculate $EV$ as $P(C1) * account \space balance$. This would give us a point estimate of $EV$, which is fine but may not tell the whole story given our uncertainty in the model.

So, how can we provide a lower limit and upper limit of the probability this instance is of class C1 with 95% confidence?


texts: What is the best practice for tuning hyperparameters using validation data?

texts:
I'm building a binary classifier, using task-transfer from resnet and a total training set of 300 images.
Initially I put aside 100 images as validation, and tuned the hyperparameters, each time training on 200 and testing on 100, until I got a validation accuracy of 93%.
Happy with this accuracy I tried the same parameters on the test set (another 170 images) and got really bad accuracy (around 65%).
What did I do wrong?

Should I have used cross validation?
What is the best practice here? How should I got about tuning my hyperparameters?
Can I repeat my process and check on the test set again? If so, how many times can I do this before it's "cheating"


texts: How does orange encode categorical values for a neural network? What function does it do and how? I can not find it in the code.


texts: A2C learning very slowly when I try to make it learn on batches as compared to making it learn on each step

texts:
I tried this on openai gym environment - LunarLander-v2.
I wrote two algorithms with just one difference:

Made it learn on each step.
Made it learn at the end of each episode.

There is a significant difference between the performance of both. First one started touching around 200 reward after 3000 episodes whereas second algorithm is getting only -50 reward after 8000 episodes. Btw 3000 episodes of first algo took the same run time as the 8000 of second.

I tried the same experiment on CartPole too the results were same initially but after some time first algo's rewards dropped and kept on fluctuating but second algo surpassed it and was more stable. I am not sure if this is supposed to happen.
Also, I am not doing any normalization on either the observation state or the rewards. Should it be done and if yes then how?
a


texts: GAE - Understanding the TD based advantage function

texts:
The advantage function in GAE is defined as (eq 1)

where (eq 2),

The question is, in Eq 2, why is a value function (estimator, that needs to be trained) used at all? Especially when the value function is used to estimate a reward whose true value is already available to us? Why not just use that reward as the value?
During the training process, the value function is estimated with an $ MSE_{loss} = (Value-RewardToGo_{Discounted})^2$ loss function and gradient descent [3]. Here, why is a separate value function is being used to estimate reward? Why not just used the actual reward that's calculated in Eq 2? If the value function is being used to estimate values of states other than the trajectory taken then it makes sense, but here, since we have traversed the particular trajectory we already have the true value through the reward.
Does this make sense? Please share your thoughts


texts: Does the predict function in machine learning understand categorical data

texts: I understand that before feature engineering one has to split the dataset into train and test data, so as to avoid bias in the analysis. I also understand that the machine learning model does not understand data apart from numerical data, thus encoding is required, which is a part of feature engineering. My question is, do I encode the test data separately or does the prediction function understand categorical data.


texts: How should I evaluate whether new features are effective or not? Should I build a new model with the new features then compare with the old one with the same hyper parameter?


texts: imbalanced dataset in text classififaction

texts:
I have a data set collected from Facebook consists of 10 class, each class have 2500 posts, but when count number of unique words in each class, they has different count as shown in the figure

Is this an imbalanced problem due to word count , or balanced according number of posts. and what is the best solution if it imbalanced?

update
My python code:

data = pd.read_csv('E:\cluster data\One_File_nonnormalizenew2norm.txt', sep="*")

data.columns = ["text", "class1"]
data.dropna(inplace=True)
data['class1'] = data.class1.astype('category').cat.codes
text = data['text']

y = (data['class1'])
sentences_train, sentences_test, y_train, y_test = train_test_split(text, y, test_size=0.25, random_state=1000)
from sklearn.feature_extraction.text import CountVectorizer
num_class = len(np.unique(data.class1.values))

vectorizer = CountVectorizer()
vectorizer.fit(sentences_train)

X_train = vectorizer.transform(sentences_train)
X_test  = vectorizer.transform(sentences_test)

model = Sequential()
max_words=5000
model.add(Dense(512, input_shape=(60874,)))
model.add(Dense(20,activation='softmax'))####
model.summary()
model.compile(loss='sparse_categorical_crossentropy',
  optimizer='rmsprop',
  metrics=['accuracy'])

model.fit(X_train, y_train,batch_size=150,epochs=10,verbose=2,validation_data=(X_test,y_test),shuffle=True)
predicted = model.predict(X_test)
predicted = np.argmax(predicted, axis=1)
accuracy_score(y_test, predicted)
predicted = model.predict(X_test)
predicted = np.argmax(predicted, axis=1)
accuracy_score(y_test, predicted)

0.9592031872509961


texts: Removing outliers with orders of magnitude differences

texts:
I have a dataset of virtual currency earn and spend events from a mobile game app. Unfortunately, people cheat in the game to get more currencies. These cheaters use different techniques so its quite hard to detect them all in game. The dataset is also about 50 TB so the only option I have is to use SQL (on Google Bigquery).

It tried to make a standard outlier detection system in which I find the average and standard deviation of the currency earn and spend in each level. This works for the biggest outliers. However, some people cheat to earn, for example, 1e15 gold, while other "only" cheat to get 10000 gold. The normal gold earn rate should not be higher than about 1000. The standard outlier detection system works for the 1e15 gold earn person, but as the avg and std are so high due to that person, the 10000 gold is not found to be an outlier.

Does anyone have an idea how to find these outliers successfully?

My dataset looks something like this. (It should be noted that cheaters don't show up this often and that the dataset has in the order of billions of these rows):

user_id, currency, earn_or_spend, source_or_sink, amount
'user_1', 'gold', 'earn', 'quest', 3
'user_1', 'cash', 'earn', 'building_collect', 10000
'user_3', 'gold', 'spend', 'quest', 1
'user_2', 'gold', 'earn', 'quest', 4
'user_1', 'cash', 'earn', 'building_collect', 50000
'user_1', 'gold', 'earn', 'quest', 5
'user_4', 'gold', 'earn', 'quest', 99999 # cheater
'user_3', 'gold', 'spend', 'quest', 3
'user_5', 'gold', 'earn', 'quest', 1E15 # cheater
'user_3', 'cash', 'earn', 'level_up', 100000
'user_1', 'gold', 'earn', 'quest', 3
'user_1', 'cash', 'spend', 'build_building', 50000

It can be generated in Google BigQuery with this code:

WITH data as (
  SELECT
    *
  FROM UNNEST
  (
    ARRAY<STRUCT<user_id STRING, currency STRING, earn_or_spend STRING, source_or_sink STRING, amount FLOAT64>>
    [
      ('user_1', 'gold', 'earn', 'quest', 3),
      ('user_1', 'cash', 'earn', 'building_collect', 10000),
      ('user_3', 'gold', 'spend', 'quest', 1),
      ('user_2', 'gold', 'earn', 'quest', 4),
      ('user_1', 'cash', 'earn', 'building_collect', 50000),
      ('user_1', 'gold', 'earn', 'quest', 5),
      ('user_4', 'gold', 'earn', 'quest', 99999), # cheater
      ('user_3', 'gold', 'spend', 'quest', 3),
      ('user_5', 'gold', 'earn', 'quest', 1E15), # cheater
      ('user_3', 'cash', 'earn', 'level_up', 100000),
      ('user_1', 'gold', 'earn', 'quest', 3),
      ('user_1', 'cash', 'spend', 'build_building', 50000)
    ]
  )
)

SELECT * FROM data


texts: Finding change maximum change in the value using Redshift

texts:
Following is the problem I want to solve. But I don't know how to implement it. I am using Redshift to store data.

Following is the format of the data stored in Redshift. It is sales history for every product for all year by month.

ProductId   Year    Month   Sales
A           2018    1       ...
A           2018    2       ...
A           2018    3       ...
A           2018    4       ...
A           2018    5       ...
B           2018    1       ...
B           2018    2       ...
B           2018    3       ...
B           2018    4       ...
B           2018    5       ...
....
....
....

Let's assume following are the sales chart of Product A, B and C. Each line represent sales in different year. I have a query: "Sales of which product increases in the month of October".
The answer should be Product A followed by B. (Product A has fewer sales then Product B but percent change in sales value of Product A is higher)

How can I extract such a pattern from the data?


texts: The gradient of expectation of an objective function

texts:
Denoting $x$ : 2D image,
$\delta $~$ N(0,I)$,
$\theta=x+\sigma\delta$,
$F(x)$ : objective function,
$\pi(\theta|x)$ : search distribution,
we have $\nabla_x E_{\pi(\theta|x)}[F(\theta)]= E_{\pi(\theta|x)}[F(\theta)\nabla_xlog(\pi(\theta|x))]$.

But suddenly they say that $\nabla_x E_{\pi(\theta|x)}[F(\theta)] \approx \frac{1}{\sigma n}\sum_{i=1}^{n}\delta _i F(\theta+\sigma\delta_i)$.

How is the approximation formed?

We can approximate the expectation using $n$ samples, but why $\delta_iF(\theta+\sigma\delta_i)$?

I'm reading a machine learning paper and want to know the missing links as stated above in section 2.1.1.

The paper is : A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box Adversarial Attacks with Limited Queries and Information,” Available: http://arxiv.org/abs/1804.08598.

Thank you in advance.


texts: Why does my model produce too good to be true output?

texts:
I am trying to run a binary classification problem on people with diabetes and non-diabetes.
For labeling my datasets, I followed a simple rule. If a person has T2DM in his medical records, we label him as positive cases (diabetes) and if he doesn't have T2DM, we label him as Non-T2DM.
Since there are a lot of data points for each subject, meaning he has a lot of lab measurements, a lot of drugs taken, a lot of diagnoses recorded, etc, I end up with 1370 features for each patient.
In my training, I have 2475 patients and in my testing, I have 2475 patients. (I already tried 70:30. Now am trying 50:50 still the same result (as 70:30))
My results are too good to be true as shown below

Should I reduce the number of features? Is it overfitting? Should I retain only the top features like top 20 features, top 10 features etc? can help me understand why is this happening?
Detailed Update
We check for the presence of T2DM by a set of diagnosis codes (like icd9,10 codes for T2DM and its complications etc). For ex: let's say if a patient has an icd9 code of 250 in his diagnosis records, we know that he was diagnosed with Type 2 diabetes. Let's not worry about the accuracy of this labeling approach at this point. Similarly, we label all other patients as T2DM and Non-T2DM.
But when we extract features, all his medical records are treated as features. The frequency of the drug/condition/lab tests will be used as a feature value. So, basically, the same diagnosis code (250) will be an input feature as well. Does it mean I should drop the diagnosis codes which were used to label a dataset from being used as features? But those are really very good features that can help me find out whether a patient is diagnosed with T2DM or not (when I apply my model on a totally different dataset). My objective is not to find out whether a patient will develop diabetes in the future or not but my objective is only to find out whether a patient is diabetic or not (from his records). So, I label my dataset with an imperfect heuristic as stated above and build a model. Once I build this model, I would like to validate this model at another site and find out how good is the model built using this imperfect heuristic in identifying whether a patient is diabetic or not. Hope this helps


texts: How does YOLO algorithm detect objects if the grid size is way smaller than the object in the test image?

texts: In YOLO algorithm how do these grids output a prediction if some grids only see a small black portion of the car if the model was trained on datasets with full images?


texts: K-fold cross-validation with validation and test set

texts:
For a project I want to perform stratified 5-fold cross-validation, where for each fold the data is split into a test set (20%), validation set (20%) and training set (60%). I want the test sets and validation sets to be non-overlapping (for each of the five folds).
This is how it's more or less described on Wikipedia:

A single k-fold cross-validation is used with both a validation and test set. The total data set is split in k sets. One by one, a set is selected as test set. Then, one by one, one of the remaining sets is used as a validation set and the other k - 2 sets are used as training sets until all possible combinations have been evaluated. The training set is used for model fitting and the validation set is used for model evaluation for each of the hyperparameter sets. Finally, for the selected parameter set, the test set is used to evaluate the model with the best parameter set. Here, two variants are possible: either evaluating the model that was trained on the training set or evaluating a new model that was fit on the combination of the train and the validation set.

Right now, I've implemented something that looks like the following (described here):
kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

for train_index, test_index in kf.split(X):
    X_tr_va, X_test = X.iloc[train_index], X.iloc[test_index]
    y_tr_va, y_test = y[train_index], y[test_index]
    X_train, X_val, y_train, y_val = train_test_split(X_tr_va, y_tr_va, test_size=0.25)
    print("TRAIN:", list(X_train.index), "VALIDATION:", list(X_val.index), "TEST:", test_index)

Although it does quite nicely provide me with 5 folds whereby for each fold I have a validation set, test set and training set, the validation sets have overlap between each of the five folds (so some instances present in fold 1 are also present in fold 2, etc.). For the test sets this is not the case (i.e. they do not have overlap between folds).
Is there a way to prevent this overlap between folds from occurring for the validation set as well?


texts: practical improvements worth trying over plain LSTM in text classification?

texts:
I have a dataset of about 1 million tweets corresponding to about 30,000 user accounts, labelled with binary data (classifying the tweet as written by a bot).

With that amount of data, I could use a LSTM-based NLP approach to take advantage of the sequence data in the text.

I would like to get the best accuracy possible, but don't know what models are available that could be combined with or build on LSTM-based approaches.

I also wonder, as one part of the data is sequential and the other stationary (account metadata), whether some kind of hybrid classifier could do better than using just an LSTM.

Are there any hybrid models or any other proven models that would be worth investigation further, to implement for my project?


texts:
I am working on a classification of two feature sets derived from a dataset. We first obtain two feature matrices derived from two feature extraction methods. Now, I need to compare them. However, the recognition accuracy for two feature sets, reaches almost the same recognition accuracy. My question is:

Is there a way to design a meaningful experiment to show the difference between the two methods? What are your suggestions?


texts: Comparing two time series data to find deviations between them

texts:
This is a use case that I have and I am trying to automate this. Any pointers would be helpful.

Use Case:

When we deploy any new version of a web service, we keep monitoring it (while deploying to live) to make sure it is not introducing any new errors. For this, we just visually compare with last weeks trend of errors (within the same time frame) and if they look similar, we approve the new build, or if the number of errors seems to increase, we decide to rollback.

What I am looking for is to automate this decision making. Basically, compare the error data during the push with last weeks (or any other time frame) and decide if the two error trends are similar and to what degree they are similar.

The data that I have is
x axis -> Time stamp
y axis -> # of errors at that time stamp.

I also have details such as number of requests at that timestamp, latency, etc


texts:
For example, you can reorder a softmax + nl (negative likelihood)
 to  log_softmax + nll (negative log-likelihood)

Essentially changing log(softmax(x)) to softmax(log(x))

However, what are the rules to reordering logging of things?


texts: What type of classification algorithm for recognizing events in a set of data?

texts:
I would like to know what type of machine learning to use for recognizing events in a set of data. (or at least be pointed in the right direction so I can do my own research on the topic.)

I'm trying to identify M&M color using an RGB sensor. Here is an example of some data I collected. I want to train a neural network by saying "this is what it looks like when a blue M&M passes by the sensor... And a yellow..." So it then can tell me when and what color M&M has passed by.

My type of problem is similar to this. In this case, they record accelerometer data and use it to identify the exercise someone was doing.

How can you teach an ANN to do this? And how do you teach it what an "event" looks like?

** P.S. I'm new to this forum so any modifications to my question to make it more relevant and understandable are welcome.


texts:
I'm trying to detect anomalies in an univariate time series.
I trained a RNN LSTM and currently I get one-step-ahead predictions.

Could someone explain if it's possible to output a confidence interval (or maybe a prediction interval) with RNN LSTM instead of just a predicted value.

When I want to flag observations as anomulous on real-time data I think I need more information than just the predicted and observed value for point x (f.e. I need a predicted confidence interval).

Please point me into the right direction.
Currently I'm using an Adam optimizer, MSE as the loss function and a Dense output layer with a linear activation function.


texts: Are the image data augmentation generators in Keras randomly applied

texts:
I am working on an image classification problem and using data augmentation in Keras.

from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(
    rotation_range=2,
       horizontal_flip=True)

I would like to know if the ImageDataGeneratorapplies the transformation randomly to image patches. That is for example, rotation may be applied on one image, whilst flip may be applied to another image. I want to know if the decision to apply rotation or a flip is randomly determined.


texts:
I want to cluster image, since varibility intra and inter class of images is huge I think reducing dimensions with a convolutional autoencodeur can be a good tools. Then I apply clustering on the feature vector

My question: is there a theorical link between my convolutional autoencodeur loss and the potential for clustering of my extracted feature vector?

To add more detail, can we says if I can minimise loss then it'll be easier to do clustering on feature vector? My intuition tel me answer is NO

But if we can't use loss to select potential best model for clustering, then isn't it almost impossible to do a good clustering using this methodology?


texts: Input 0 is incompatible with layer conv2d_2: expected ndim=4, found ndim=3 I get this error in Tensor flow, What does it mean and how can I fix it?

texts:
import pickle

import keras

from keras.models import Sequential

from keras.layers import Dense, Dropout, Flatten, Activation

from keras.layers import Conv2D, MaxPooling2D

from keras.utils import to_categorical

import numpy as np

X = pickle.load(open('cancer_image_features.pickle','rb'))

y = pickle.load(open('cancer_image_lables.pickle','rb'))

X = X/255.0

y = to_categorical(y)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=X.shape[1:]))

model.add(Conv2D(64, (3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, kernel_size=(3, 3),activation='relu'))

model.add(Conv2D(64, (3, 3), activation='relu'))

model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Activation("softmax"))
model.add (Flatten())

model.add(Dense(3))

model.compile ( loss = 'binary_crossentropy', optimizer = 'adam' , metrics = ['accuracy'])

model.fit(X,y, batch_size = 512, epochs = 10,  validation_split= 0.3)


texts: Fashion Clothing images Dataset with image description

texts:
I am searching for Fashion Clothing image dataset where each image is associated with description. Basically description should contains all information about cloth. like -

image_01.jpg - red shirt has collar with long-sleeves.

Is there any good source to collect these type of dataset ??


texts: How to predict orders with a range of items? And total orders which sum up to the total?

texts:
So I do have data like this:

With the help of distinct order IDs, I can figure out how many orders are there and from units shipped, I can get the number of items in the order. Now I want to predict future orders having x items for different ranges (e.g. 1-10 items, 10-20 items, 20+ items). Like tomorrow there might be 6 orders one of them will have 1-10 items, 4 will have 10-20 items and one will have 20+ items. The problem is I can predict them (using LSTM for now) separately using data for only those kind of orders but they obviously do not sum up to the amount predicted by model for totals. So how can I link them up so that total sums up to the values from the individual models?
Approach I am currently using:
I just predict orders for each item range and then manually sum them up. This way they do sum up but error accumulates


texts: Computing generalized variance for high-dimensional data

texts:
What are some practical methods to compute the generalized variance $|\Sigma|$ of a multivariate Gaussian, for high-dimensional data (d>200)?

I'm having troubles computing the product of eigenvalues to get the determinant: right now my eigenvalues are less than 1 and their product underflows to 0, which makes the computation intractable. I have tried numerous numerical methods, without much luck so far.

Are there some known methods to compute the variance for very high dimensions? In particular, I'm looking to use it to compute the differential entropy.


texts:
I receive Searchable PDF files that comprise of scanned pages from multiple documents (structured documents such as forms and unstructured documents such as letters).  As the files do not contain a separator demarcating the end of a document, I have had to resort to classifying the start page of a document (a small subset of the total pages).

Given that the class labels of interest are a small subset of the pages, I think I could improve accuracy if I could identify and classify documents rather than the start page of a document.

I am unclear on how I would go about identifying documents within the stream of pages.  For now, we can assume that the document pages are sequential, and that I am able to extract text, text position and font details from the searchable PDF.  I also am able to extract the images.

Thanks.


texts: When Should I Use ggplot2 Instead of Tableau?

texts:
I am a student getting started with Tableau for the first time. My proficiency with ggplot2 is intermediate. I can create custom versions of the most popular kinds of charts in ggplot2 but nothing too fancy (and not very time-efficiently).

I am interested in creating more polished visuals of my data in Tableau. My ggplot2 ones aren't bad, but they are not what I would call aesthetically pleasing either.

What I would like to know is under what situations I should stick with ggplot2. The only advantages that come to mind with my current (albeit limited) understanding of Tableau is to keep my work reproducible and open source. ggplot2 is also probably more customizable, but I am continually impressed by Tableau's offerings as I learn to use it.

This question is not intended to be a flippant jab at ggplot2. As I did not hear any talk about Tableau in the academic circles I was in, I am wondering what obvious drawbacks to Tableau may exist that would explain its unpopularity there.


texts:
I have a rate-step plan for consumption with which I'm trying to total up costs. I have a rate plant that looks like this:

First 20 Kwh is Free;
Second 5 Kwh is $.1
    Next 100 Kwh is $.053
Next 875 Kwh is $.042
    Over 1000Kwh is $.039

Can someone provide a worked example for me, e.g., with $\text{consumtpion}=625$? If possible, with R code for calculating the total cost.

For my own approach, I was thinking of using a loop, but if it's a lot of accounts the loop will take too long.  Can anyone think of a vector-based approach for calculating the result?


texts: How to determine the weights of features when calculating the weighted average?

texts:
I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I want to calculate a value called "Popularity" by taking the weighted average of Retweet Count and Favorite Count for each Tweet. For this, I need score values (or weights) for Retweet Count and Favorite Count. How do I determine these weights?
I am writing a research paper and need to justify why I assigned those weights for Retweet Count and Favorite Count.


texts: Shapley summary plot interpretation doubt?

texts:
I have question when interpreting SHAP summary plot. I have attached the sample plot
Here, If I am interpreting it correctly, low values of feature 1 are associated with high and negative values for the dependent variable. However, Feature 1 takes negative values as well. So will the low value of feature 1 include negative and high values?
In other words, is it right to say: negative and high values of feature 1 are associated with high and negative values for the dependent variable?


texts:
So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then use them in supervised DNN. But recently I read a blog post which suggested that training the word embeddings while training the neural network gives better results. This is the other link.
So my question is which one should I follow?
Some YouTube videos that I referred:

Deep Learning for NLP without Magic Part 1, 2 and 3


texts: Clustering using both text and numerical features

texts:
I have a dataset that contains 2 types of features, one is generated from doc2vec and one is numerical feature. I would like to perform clustering analysis on them. However, due to the size of doc2vec features, if I simply combine them into one array, clustering algorithm would distribute the "weight" on the doc2vec features more, how do I overcome this problem?
For example, for a given label, say I have features from doc2vec that look like this [1,2,3,4,5], and numerical feature [2]. I don't want to simply combine them into [1,2,3,4,5,2] and perform clustering analysis. Ideally, I would like my clustering algorithm to give the numerical feature equal importance as the doc2vec feature.


texts: What kind of techniques that I can use to see which type of of entries in the dataset became more popular lately(over a time-span)?

texts:
This is for my school project. I've been digging the internet in to find a solution that is backed-up really well but I couldn't find a metric that makes sense.
I have a dataset, that has objects say: ['A', 'B', ... 'E'], and I have how many transactions that has been done to buy these objects and when (with all timestamps) over a year.
Now I need to find which objects became more and less popular lately, so the company would produce more or less number of those objects to optimize the profits. I don't have the number of objects in the inventory as well. Since there are 1500 objects I cannot really make a visualization to have an overall view. Things that came to mind:

Compare the mean number of sales per month for the first 3/4 of the year and the mean number of sales per month for the last 1/4 of the year.

Calculate the standard deviation for the whole period in the number of sales and compare the mean of the last 1/4 of the year to see if it's above or below standard deviation. If so, how many standard deviations away is the value that is being checked.

I was wondering if there is another more sophisticated solution for this problem, or if my solutions are good enough - if so which one is better. Should I use both maybe?


texts: Count occurences of rows based one cell content with pandas

texts:
I am currently struggling with the pandas framework.
Let's imagine the following Data Frame:
customer | order_number | product
------------------------------------
A        | O1           | Product_1
A        | O1           | Product_2
A        | O2           | Product_X
B        | O5           | Product_1
C        | O6           | Product_2
C        | 06           | Product_X

What I am trying to do is to count the occurences of "products" based on "order_number". This means that I would like to know how how often e.g. Product_1 and Product_2 were sold within one order. Of course I am interested in every available combination of products. I tried to use df.groupby['order_number'].count() but this is certainly not the data I want. At this point I am not interested in order_numbers nor the customers! Just the available sets of products and how often they appear within the data frame.
Thanks in advance!
Update: Based on my example this explains what I would like to get
Product_1 && Product_2 | One Time
Product_X | One Time
Product_1 | One Time
Product_2 && Product_X | One Time


texts: Weighted accuracy, sensitivity and specificity

texts:
I have a confusion matrix
TN= 27 FP=20
FN =11 TP=6
I want to calculate the weighted average for accuracy, sensitivity and specificity. I know the equation but unsure how to do the weighted averages.


texts: I work in the medical domain, so class imbalance is the rule and not the exception. While I know Python has packages for class imbalance, I don't see an option in Orange for e.g. a SMOTE widget. I have read other threads in Stack Exchange regarding this, but I have not found an answer to how to tackle class imbalance in Orange without resorting to Python programming. Thanks


texts:
let me explain what I want to do,

I want to predict the trend of the price of something (1 if it increases in the next hour and 0 otherwise). I have gathered tweets about that and grouped them in each hour. what I want to do is to predict the next hour trend using tweets.

I decided to look at 40 previous tweets and based on that predict the label which means if I want to predict the next hour label I may use tweets of last 2 or 3 hours (which is fine)

So here is the question, the data structure should be in a way that y is the label of next hour, and x is the 40 previous tweets, I mean the iteration happens on y, not x so, for example, x_train[1] and x_train[2] are not different in one tweet like other LSTM models data, it may have 39 or 1 or... tweet in common.

Is it ok to use this type of data structure in LSTM or x must be in sequence?

p.s: you may say that iterate x instead of y. I can't because I want to add news titles to my model too, so, for example, x_tweet and x_news data should result in the same y.


texts: Recommender system based on binary likes/disklikes?

texts:
I am building a recommender system. I have a list that shows me what a user has disliked and I use it to create a dataset. The dataset shows me: ProductID, Likes, Disklikes where the values are counts from other users that have disliked the same products.

What are some of the algorithmic approaches I can use from this base? From this data I know I can get a probability. How would I also get a weighted average? Maybe some sort of Bayesian filtering approach is called for in these scenarios?


texts: What kind of data (in context of trends in data) is Logistic Regression appropriate for?

texts:
I'm not able to visualise what kind of 'trends' I would have to observe in multi-featured data to be able to say 'Logistic Regression would work well here'.
For example if I have only 1 feature and if the data is something like all negative data is class 0 and all positive data is class 1. Then I can clearly say that Logistic Regression works well here.
So what kind of 'analysis' of the data (multi-featured) would I have to do to decide whether Logistic Regression would work well or not?


texts: How do you measure performance for word prediction tasks?

texts:
Say I have to predict the next word in a sentence, given the initial few words.

Suppose the prefix is "I went to _____". This prefix is common enough that it might appear 10 times in the training data with a few different variations:

I went to college: 5

I went to California: 3

I went to London: 2

In such a case, suppose my model predicts "college" as the right answer. It would still get only a 50% score if I used accuracy as a metric, the data naturally has multiple (correct) answers for the same input. How do I solve this?


texts:
I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions.

Should i use a dummy for all of them? For example, only type variable has 33 values itself, or i can use clustering and reduce them? Or any other way if you know)
Is there a difference in terms of behaviour between categorical variables which have a rank or not? For example type is "A","B",..."S" so no rank between A and B but quality is "A1","A2","A3" which A1 means highest quality.

I don't know why, I can't find enough information about variable selections and making data ready. So now i have lots of variable and I think I should choose between them and also reduse number of dummies.


texts: Is it possible to modify the layers of the models present in the Tensorflow Object Detection API

texts: I would like to know if there is any way in which we can change the base layers of the models offered by the Tensorflow Object Detection API and if so, is it possible to change things like pooling/padding size, number of layers and also hyperparameters like optimizer used, initial weights and the learning rate.


texts: SciKit Learn: Multilayer perceptron early stopping, restore best weights

texts: In the SciKit documentation of the MLP classifier, there is the early_stopping flag which allows to stop the learning if there is not any improvement in several iterations. However, it does not seem specified if the best weights found are restored or the final weights fo the model are those obtained at the last iteration. If not, is possible to restore the best weights found?


texts: Exceptionally high accuracy with Random Forest, is it possible?

texts:
I need your help to find a flaw in my model, since it's accuracy (95%) is not realistic.

I'm working on a classification problem using Randomforest, with around 2500 positive case and 15000 negative ones, 75 independent variables. Here's the core of my code:

# Splitting the dataset into the Training set and Test set
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)

# Fitting Random Forest Classification to the Training set
from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(n_estimators = 900, criterion = 'gini', random_state = 0)
classifier.fit(X_train, y_train)

# Predicting Test set results
y_pred = classifier.predict(X_test)

# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)

I've optimized the hyperparameters through grid search and performed a k-fold cross validation, reporting 0.9444 as accuracies mean.
Confusion matrix:

[[3390,   85],
 [ 101,  516]]

showing 97.6% accuracy.

Did I miss something?

NOTE: the database is composed by 2500 Italian mafia firms' financial reports, and 15000 lawful firms randomly sampled from the same regions as negative cases.

Thank you guys!

EDIT: I upload the metrics and cm. The model is actually performing well, but looking at the metrics and cm, it shows more realistic values regarding logloss and recall, so I assume it is fine.


texts:
I'm trying to work on a ML project and I have a dataset and I'm trying to see if there is a correlation between some of the features in my dataset.

The dataset contains inspection notes for car parts in an industrial process along with some information about the car parts themselves.

I have posted a Jupyter Notebook with all of my code on GitHub. The data used is in the same folder. But to get back to my question, I've plotted a simple correlation matrix to show how the features correlate.

I've done some data augmentation so that I have all numerical values and be able to calculate and plot the matrix.

But after giving this a thought and reading about correlation, I'm not sure that my the correlation makes much sense (in the real world).

If my understanding is correct, correlated variables would be useful in a

The below plot is a representation of correlation between variables for all of the car parts that have been thrown away after inspection.

However, I'm not 100% sure that my following statement is correct, based on the correlation matrix below:

"There seems to be some correlation between the MONTH and the MINUTE features in my dataset."

But although I believe the numbers show this, I don't think this is valuable information and after reading I believe that correlation would work best with measurement values, not inspection values.

I'm seeking help to confirm or help me improve my understanding of what correlation is and how it would apply in my scenario.

I have a hunch that it could be applicable, but I need to tweak my dataset. Like I need to do a count aggregate on all features and build the correlation matrix and plot that, but I'm not sure yet...


texts: What is the definition of imbalanced data set

texts:
I have thousands of data sources generating data from similar type of hardware. The different sources create different dynamics in the datasets though!

Even though the features are the same the data sets have very diverse characteristics.

I am working on a multiclass classification problem trying to see how much specific models can be used to tackle that domain.

The number of classes differ on different data sources so different models need to be built. That means that in the end I have many many different models to evaluate. Similar input but the number of classes to be predicted at the output is differrent.

Since this is a multiclass classification problem things like confusion matrices are used and multiple ROC curves.

Now I am trying to see in more details what might be causing poor performance in the poorest performing models. Typically the reasons are:
1.not enough measurements
2.heavily imbalanced datasets
3. a combination of 1 and 2

The problem is that I do not have a definition on a multiclass problem what is an imbalanced dataset. Ideally if I could use a specific "rule" to label my datasets, I would be able to see things like correlation of imbalanced set and precision.

When it comes to imbalanced dataset for multiple class a threshold is not enough, since is the distribution of the available measurements between the classes that is important. For that I have no idea on how to handle that.

How would you handle this case ?

Thanks a lot for reading this and contributing to this community.

Regards
Alex


texts:
I have a dataset where the feature set consists of hour of the day (between 0 to 23), day of the week (Monday to Sunday), number of shops (a positive integer) and road category (0 to 8 on an ordinal scale). I wish to predict the pedestrian volume which is a continuous variable. Hence I was looking a regression approaches and was thinking along the lines of using KNN. Will it be appropriate for this task? I am especially worried about the hour and day data as they are cyclic in nature.

df_test[['Day', 'Time','shop_count','road_category','pedestrian_volume']].iloc[0:5]

           Day  Time        shop_count       road_category  pedestrian_volume
  0    Tuesday    20                22                   0             210.0
  1    Tuesday    21                13                   4             196.0
  2    Tuesday    22                39                   4             214.0
  3    Tuesday    23                 2                   8              46.0
  4  Wednesday     0                15                   8              18.0


texts: Help with type of ML problem: when training data is spread across different subgroups/categories

texts:
I've been searching around for a while without any luck - hopefully someone more knowledgable can give me some advice about the following ML problem that I've been thinking about:
Say you are trying to predict the Rotten Tomato "Tomatometer" score of a film before it's released. Typically you might approach this by compiling a list of features and labels for some existing films and input this into a supervised ML algorithm.
In this example, the feature list will be standard metrics that describe the film such as, budget, filming duration, number of actors, etc., while the label is the Tomatometer score of the film that is given as a value between 0 to 100. Every film can be expressed using this score, but individually they are spread across many genres, country of production, etc. meaning that there are natural subsets within the training data.
Let's say our training data contains only films belonging to five genres (e.g. Action, Thriller, Horror, Fantasy and Documentary), while we want our algorithm to be applicable to films outside of this genre (e.g. Sci-Fi or Animation), but for the sake of question, we do not have access to these entire categories. In this example, were also assuming that some features will be more important to certain genres than others, for example, a large cast size may correlate more with the score for Action films than for Animations.
What is the general way to transform the data to make it invariant to the subgroup (genre), or what ML algorithm can be used here (if any)? Is there a common name for this situation (some keywords I can search?)


texts: What is the difference between maximum likelihood hypothesis and maximum a posteriori hypothesis?

texts:
I am a student and I am studying machine learning. I am focusing on the concept of Bayesian learning and I have studied the maximum likelihood hypothesis and the maximum a posteriori hypothesis.
I have seen that the maximum likelihood hypothesis  is the hyposesis that maximizes the likelihood of seeng the data, and it is defined as:
$h_{ML}=arg_h max P(D|h)$
while maximum a posteriori hypothesis is the hypothesis that maximizes the posterir probability of seeng the data, and it is defined as:
$h_{MAP}=arg_h max P(D|h)P(h)$
I am really confused by these two definitions, since I can't grasp what is the difference between the two.
I have understood that the maximum likelihood hypothesis is the one that, given some observed data, finds the parameters of the distribution such that I am most likly to understand the data.
But I cannot unserstand what is the MAP hypothesis.
I have tried to read some interpretations and definitions, mìbut I can't understand the difference between the two.
So, what is the difference between maximum likelihood hypothesis and maximum a posteriori hypothesis?


texts: Dimensionality reduction without select components

texts:
I would like to use dimensionality reduction algorithm in my pipeline.
I have 2k features and I'm using xgboost.
My model is rebuilding each day (there are new records that should be involve to training set).

I'm looking for method for dimensionality reduction with out setting n_components.
I know that in PCA it shouldn't be set. But I'm looking for method that find something like clusters on my data and then I will use it to train my model. Of course the same flow I'll be using for prediction.

Do you have idea how should I do my data processing for this case?


texts:
I'm currently on week 4 of my Coursera course on ML, so I have much to learn about data science. However, I got the opportunity to apply what I've learned at work, and I'd like some guidance. Our company ships random objects to customers in boxes. We'd like to be able to estimate how big boxes will be, given the random objects inside.

Here's an example of the input data:

box # | contents                          | box size
----- | --------------------------------- | ---------
1     | a widget, a doodad, and a trinket | 20x12x8
2     | 3 widgets                         | 12x12x12

However, our list of items has a long tail. I did a count of total items shipped by object type, ordered by count descending. Here's the result:

rank | object count | object type
---- | ------------ | -----------
1    | 500,000      | doodad
2    | 350,000      | trinket
3    | 300,000      | widget
---  | snip         | ---
50   | 6,000        | whatyoumacallits
---  | snip         | ---
300  | 5            | quarts of blinker fluid

Etc. By item number 340, the count is 1, and there are 360 distinct items. I think one way to approach this at first would be to only consider the top 50 items, and try to do a simple polynomial regression with 50 features to estimate L, W, and H (assuming each variable is less than the previous one).

It won't be 100% accurate, but it will be better than wild guesses. But is there a better way to do this? Any advice is much appreciated.


texts: Choosing attributes for k-means clustering

texts:
The k-means clustering tries to minimize the within-cluster scatter and maximizing the distances between clusters. It does so on all attributes.
I am learning about this method on several datasets. To illustrate, in one the datasets countries are compared based on attributes related to their Human development Index. However some of the attributes are completely unrelated to this dimension, for example total population of countries. How to deal with this attributes? As mentioned before k-means tries to minimize the scatter based on all attributes, which would mean this additional attributes could hurt the clusters. To illustrate, I know the k-means cannot discern three clusters that are perfectly clustered around one dimension and are completely scattered around the other.
Should one just exclude some attributes based on prior knowledge? Is their perhaps a processes that identifies irrelevant attributes.