texts:
I use a Bayesian hyperparameter (HP) optimization approach (BOHB) to tune a deep learning model. However, the resulting model is not robust when repeatedly applied to the same data. I know, I could use a seed to fix the parameter initialization, but I wonder if there are HP optimization approaches that already account for robustness.
To illustrate the problem, let's consider a one-layer neural network with only one HP: the hidden size (h). The model performs well with a small h. With a larger h, the results start to fluctuate more, maybe due to a more complex loss landscape; the random initialization of the parameters can lead to a good performance, or to a very bad performance if the optimizer gets stuck in a local minimum (which happens more often due to the complex loss landscape). The loss vs h plot could look something like this:

I would prefer the 'robust solution', while the 'best solution' is selected by the HP optimizer algorithm. Are there HP optimization algorithms that account for the robustness? Or how would you deal with this problem?


texts: Why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?

texts:
Recently, I was working on a project and found my cross-validation error rate very low, but the testing set error rate very high. This might indicate that my model is overfitting. Why does my cross-validation not overfit while my test set overfits?

More specifically, I have about 2 million observations with 100 variables (n>>p). I randomly split the data set into 80/20 train and test. Then, I fit a model (i.e. XGboost) using a 5-fold cross validation on the training set and the estimated error rate is pretty low. Then, I used the same parameter setting and used the entire training set to fit the model. Surprisingly, when I used the test set to evaluate the performance of the model, the error rate is significantly higher than the CV error rate. Why?

Edit:

(About the error rate)

The error rate is actually multinomial logloss. I achieved a CV error rate of 1.320044 (+/- 0.002126) and a testing error rate of 1.437881. They might seem close by staring at these two numbers, but actually, they are not. I don't know how to justify this, but I am sure that they are different within the scale of performance of this project, which is from ~1.55 to ~1.30.

The way of 5-fold cross validation is like following,

divide the train set into 5 sets.
iteratively fit a model on 4 sets and test the performance on the rest set.
average the performance of all the five iterations.

I mean, if my parameter settings make model overfit, then I should see it at this cross-validation procedure, right? But I don't see it until I use the test set. Under what circumstances on the earth this could happen?

Thanks!

Added:

The only reason I could think of that why CV error rate diffs from test set error rate is

Cross-Validation will not perform well to outside data if the data you do have is not representative of the data you'll be trying to predict! -- here

But I randomly 8/2 split the 2-million-sample data set and I believe that the train set and test set should have a same distribution of variables.

(Pardon me that I post the same question here also.)

Edit:

(About the data leakage)

I got one interesting commend from @darXider in cross validated. He says,

  Sometimes when feature engineering, you have to be careful to avoid any data leak between training and test sets. For example, if you do a
  PCA on your original, untouched data, use PC1 and PC2 as "new"
  features, and then split your dataset into train and test, you are
  leaking information from training set into test set. That will boost
  your score up. You mentioned that after some feature engineering, your
  CV score and test score started to disagree. That could suggest some
  sort of information leak between training set and test set.

I wonder what is exactly the "data leakage" and why feature engineering BEFORE splitting can still cause "data leakage"??


texts: How to get a percentage of similarities of new data added based on model trained using a huge training set in machine learning

texts:
I need to find similarities in newly added data using machine learning.

We have a huge data set with more than 200 000 rows having following information:

Name
D.O.B
City (From predefined list)
Phone number (Predefined number format)

And many others.

The idea is that lots of people are registering each day for multiple services provided by our NGO.

What we really want is when a patient/user is coming to register, to check if similar data already exists in our database, so data is not duplicated, which will help us later to generate a unique list of beneficiaries.

If a user already exists with the following info:

Name: ABC

D.O.B: 1990-10-01

City: Lome

Phone number: 123456789

Service: Medications

And after a few months, he came to apply for another service without having his ID card, and while data entry errors do exist in our field of work and name could be written in different ways, the beneficiary will ask for a new registration, so instead of having 1 beneficiary registered in 2 services, we will have 2 beneficiaries with 2 different enrolled services. Which is not good for donors reporting:

Name: ABc (Assuming here ABC is different than ABc in a specific language)

D.O.B: 1990-10-01

City: Lomeh

Phone number: 1234567894

Service: Shelter

I need to run the new data into a model that will return a percentage of similarity, which means:

These data are 95% similar to already existed data (training set)

Any idea on how to do it? Which machine learning techniques should be used assuming that we have programmers who knows python and the basics of ML?


texts: What is the current state of the art solution for Movielens 100k / 20M?

texts:
I found Basic recommendation system for Movilens dataset using Keras which has a solution which works ok (MAE 0.84).

What is the current state of the art for this dataset?


texts: How to best estimate the coefficients of a confusion matrix in case of strong class imbalance?

texts:
I have a trained binary classifier (forget about how this was trained and think of it as a magical black box) and I would like to measure its classification performance (e.g. compute a confusion matrix) on a strongly imbalanced population/test dataset.
Suppose the composition (in terms of the two classes) of the population / test set is also known - say 99% of class $A$ vs 1% of $B$.

I also would like to reduce as much as possible the uncertainty in the estimate coefficients of the confusion matrix computed on the entire (large) test set under the constraint of doing predictions for a max of $N$ examples of the test set (suppose that the predictions are expensive to run).

What's the best approach to do this?

The best thing I can come up with is doing something like "stratified sampling":

I create from my test set (which is arbitrarily large for simplicity)  a balanced sample with $N/2$  examples belonging to $A$ and $N/2$ examples belonging to $B$.
I run the classifier on the balance dataset, and compute the resulting confusion matrix.
I "normalize" the so-obtained confusion matrix so that its rows sum to 1, obtaining $\hat{\mathbf C}$.
To estimate what would be the confusion matrix on a large representative sample (say of $M$ items taken from my large test set) I compute

$$
\mathbf C =
\left(
\begin{matrix}
0.99M & 0 \\
0 & 0.01M
\end{matrix}
\right)
\cdot \hat{\mathbf C}
$$

P.S.: To be really happy, I would like to estimate also the uncertainty of the components of the final confusion matrix $\mathbf C$.
To do that, can I just compute e.g. the credible intervals of the coefficients of $\hat{\mathbf C}$ starting from the balanced sample, and "rescale" them using the definition of $\mathbf C$ given above?

Bonus track: if somebody could send me with a reference about how to do some kind of "power test" (e.g. "how large should $N$ be to "bracket" the values of $\mathbf C$ within given tight intervals?") I'd be glad to review it.

Bonus$²$ track: any tips on how to extend the estimation of the uncertainty to nonbinary classifiers (more than 2 classes) are also welcome! $\ddot\smile$


texts: Can you interpolate with QLearning or Reinforcement learning in general?

texts:
I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding.

When there are not many states QLearning seems to be working well, but as soon as the environment gets bigger and the amount of states gets bigger it is performing rather bad. Since the convergence of QLearning is so slow I am wondering if it is possible with QLearning to interpolate the QValue of unexplored states since QLearning does not use a model? Is it possible with reinforcement in general or does it require to learn all possible states?


texts: K-means clustering on the data frame having only one column

texts:
unsup_df is a DataFrame which has only one column: review.

I want to form 2 clusters of the reviews. One positive and one negative.

from sklearn.cluster import KMeans

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(unsup_df)
num_clusters = 2
km = KMeans(n_clusters=num_clusters)
km.fit(tfidf_matrix)
clusters = km.labels_.tolist()

The above piece of code is throwing an error:

  ValueError: n_samples=1 should be >= n_clusters=2

on the line km.fit(tfidf_matrix)


texts: Training images for gender and and age detection from face

texts:
Can someone please tell me if it is feasible to train my own set of facial images to detect gender and age without using any cloud architecture or paying some amount of money ?
And , on an average how many images do I need to train my model on ? This question arises since I have to take into account my computational facilities . I use a Windows OS ( Thinkpad)  with 8 GB RAM .
Is it feasible for me to train my own model or I am  better off to use a pretrained model ?


texts: Classification method when idea conditions are known?

texts:
Dataset: Concrete measured on 8 sets of properties. ~4000 data points.

Known: under ideal condition, value of 8 properties for 10 different types of concrete.

The objective is to find: in 8 dimension space, what is the 'type of concrete' to which the given data point is nearest to.

I think image well explains the question if my words are confusing. Black = idea condition. Red =  points who's category need to be identified.

Clarification:

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=8)
knn.fit(X_train, y_train)
pred = knn.predict(X_test)

If i understand correctly, it would be acceptable to: X_train will be array of 8 by 10 and Y_train will be array of 1 by 10. Is this correct.


texts: install_tensorflow() keeps overwriting virtualenv, making it impossible to install tensorflow and keras

texts:
I would like to use the Keras & Tensorflow package for R in RStudio.

Everytime I use the command install_keras() or install_tensorflow(), I get the following error:

Collecting keras
Could not fetch URL https://pypi.python.org/simple/keras/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement keras (from versions: )
No matching distribution found for keras

I discovered that every time I tried to install Keras within R, it created a virtualenv in the following directory (basically overwriting it):

/users/username/.virtualenvs/

Where it has Python version Python 2.7.10 (default, Jul 14 2015, 19:46:27). When I directly activate the virtualenv and try pip install keras, I receive the same error. I figured out on Stack Exchange that I could solve this issue by using pip install requests[security]. This seemed to fix the problem but nevertheless, R won't use it.

How can I fix this problem, realising that every time I use install_keras(), I overwrite the virtualenv and the same problem occurs? I even tried using the tensorflow URL directly, but I get the same error when it tries to install other important packages.

Important information:

MacOS: 10.10.5
R: 3.3.3

I'm new to this, so my apologies if I don't use the terminology correctly. I did manage to make Tensorflow & Keras work in Anaconda and using Jupyter, but I can't seem to make it work in R in that environment as well.


texts: What is the advantage of using RNN with fixed timestep length over Neural Network?

texts:
More often than not, I see RNNs being used with fixed length timesteps. So what is the difference between the following two networks?

RNN with timestep length of 3 over sequence Xt.
NN with inputs x(t-2), x(t-1), x(t)

They both get 3 timesteps of sequence at each turn. Thus, (I know I am wrong but) these two networks have the same capability. They both use previous 3 samples to predict the next sample. What is the difference, then?


texts: Moving to pytorch from tensorflow: practical considerations regarding inputs

texts:
As TF 2.0 looms and with it the certainty of having to rewrite or throw away most of my scripts, I am considering switching to pytorch. I initially liked TF for its low-level API — I think it is becoming a complete mess by trying to become everything at once. Also, the library is a hell to compile on Windows. Using a trained model from C++ on Windows has become virtually impossible because of that.

I have looked a bit at pytorch, I like that its API is simple and consistent, and Windows users are greeted with C++ librairies straight from the website. Frankly I feel they are flirting with me. I thought pytorch was a python wrapper around a Lua library but I have clearly been mistaken.

One of the strength (maybe the selling point to me) of TF is its input pipeline. Whether from the old tf.queue days or the more recent tf.data framework, TF always tried to propose an input pipeline that is able to preprocess and prefetch input data to device memory concurrently to training. It is all pretty well explained on this page.

This efficient input pipeline is important in practice to achieve a high utilization of GPUs, esp. when I/O operations and data augmentation is a bit heavy. Is there an equivalent in pytorch? I have not seen such an example in the tutorials. I saw a Dataset class in the "Data Loading and Processing" tutorial but it looks like standard python, so I am assuming it cannot run concurrently to training.

If anyone with experience with both could weigh in on that topic, and also perhaps on other topics such as using a trained model from C++, I'd be glad to hear it.

I know I would be flamed on stackoverflow for asking this kind of broad question, I hope it's okay here.


texts: Cox proportional hazards with multiclass dependent variable

texts:
Is it possible to perform a multiclass Cox proportional hazards model? I'm interested in finding the probability of self cure on consumer loans, or just study how self-curing clients behave so as to identify future customers that behave the same way. My dependent variable of interest has 3 classes: -not cured (0) -cured after being contacted (1) -self cured (2)
I was thinking maybe two regressions have to be run in this case (similarly as in the multinomial logit), one where it's 0 vs 1, and another where it is 0 vs 2, maybe another 1 vs 2?
I tried looking for this but I didn't find anything, so I'm wondering if it's possible at all?


texts: pass variable length argument to mstats.kruskalwallis

texts:
I am trying to run kruskawallis test on multiple columns of my data for that i wrote an function

var=['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']

def kruskawallis_test(column):
    k_test=train.loc[:,[column,'SalePrice']]
    x=pd.pivot_table(k_test,index=k_test.index, values='SalePrice',columns=column)

    for i in range(x.shape[1]):
        var[i]=x.iloc[:,i]
        var[i]=var[i][~var[i].isnull()].tolist()

    H, pval = mstats.kruskalwallis(var[0],var[1],var[2],var[3])

    return pval

the problem i am facing is every column have a different number of groups so var[0],var[1],var[2],var[3] will not be correct for every column. mstats.kruskalwallis() take input vector which contain values of each group to be compared from a particular column.(as per my knowledge).

is there a better way to do this?

or what can i do pass different number of variable for every column for example:

if a column x have a, b, c, d, e levels how can i pass 5 vectors?


texts: Classification using xgboost - predictions

texts:
I was trying to build a 0-1 classifier using xgboost R package. My question is how predictions are made? For example in random forests, trees "vote" against each option and the final prediction is based on majority. As regard xgboost, the regression case is simple since prediction on whole model is equal to sum of predcitions for weak learners (boosted trees), but what about classification?

Does xgboost classifier works the same as in the random forest (I don't think so, since it can return predictive probabilities, not class membership).


texts:
I read a lot of articles online about how regularization works and most of them just show the equations with regularization terms but did not use example numbers to explain how the coefficient values change as lambda increase.

For example:
L1 Regularization

theory states that When Lambda is large, coefficients tend to 0.

X  =

How is the coefficients(beta i assume) value changing when lambda changes?
Is it because X a fixed value?

Example, does beta have to decrease when lambda increase to ensure that X is the same?


texts: How are per-layer-detected-patterns in a trained CNN plotted?

texts:
In the case my question is not clear, I am talking about the patterns that are detected in each of the layers of an image-trained Convolutional Neural Network (CNN). Take the following image as an example (by Honglak Lee). I think I understand the concept: different layers start coding for different features with increasing complexity. The first layers code for edges, the middle layers code for simple features (e.g. nose, eyes), and the later layers code for whole faces.
However, I do not see the equivalence betweence each of the patterns in the picture and the network elements. Do each pattern inside a pattern-class correspond to one neuron of that layer? How are these cool pattern detection pictures even plotted?


texts: For a square matrix of data, I achieve $R^2=1$ for Linear Regression and $R^2=0$ for Lasso. What's the intuition behind?

texts:
For a square matrix of random data, N columns and N rows. I am fitting two models, linear regression and Lasso.
For the linear regression, I achieve a perfect score in train set, while in the Lasso I achieve a score of 0.
import pandas as pd
import numpy as np
from sklearn import linear_model

N = 100
X = pd.DataFrame(np.random.rand(N,N))
y = np.random.randint(20, size=N)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Linear regression score 1.0
Lasso score 0.0
My questions will be:

Can someone give me an intuitive idea why for a square matrix, Lasso regression achieve different results from Linear regression?

What is the intuition behind the Lasso shrinking?

Note: In this case, the matrix is squared and the data/target is created randomly.


texts:
I am working on "community detection" on a big network graph and I've been using the Label Propagation Algorithm from GraphFrames (Spark package).

However, my graph is weighted and I was wondering whether I can leverage some community detection algorithms which take into account weights.

My main limitation is that I should run the algorithm in parallel

Any ideas which algorithm should I look at?


texts: I am looking for an example or tutorial of a system predicting numeric values by the use of various classifiers like SVM, Decision trees, ANN or KNN which optimises its choice of algorithm and parameter settings at runtime. Assuming there is unlimited input data and the system can train itself nonstop, the system should be able to adapt to different input data and choose the most accurate prediction setting as possible. Examples in Python, C++ or JAVA would be preferred. Thank you in advance!


texts: How to wisely choose machine algorithm for the given data set?

texts:
I have a data sample of RSSI value from three different devices and based on the RSSI sample, it should tell from which location the data sample is arrived. Below are the sample data set,

 device_1  device_2  device_3  location
 -45       -56       -78       drawing_room
 -48       -51       -82       drawing_room
 -41       -59       -73       drawing_room
 -71       -59       -59       conference
 -69       -60       -65       conference
 -73       -60       -52       conference
 -33       -68       -64       kitchen
 -32       -66       -63       kitchen
 -37       -67       -61       kitchen
 -63       -48       -48       lab
 -62       -48       -46       lab
 -59       -48       -54       lab

I will have "n" No.of data samples for "m" No.of location. The actual data set can be find here.

I want to predict "location" from the d1 d2 d3. Based on what parameters (i.e., correlation matrix or visualization), the machine learning algorithm can be chosen for the given data set?

The visualization chart for the considered data set is

From the visualization chart, Does it mean only the column "device_1" and "device_2" has a good separation among the location compared to other columns?

Note: If required, the data samples can be considered as positive.


texts: Designing an (RL) agent for a graph-based music improvisation system

texts:
I am trying to implement a simple agent that creates sounds by building up signal building blocks (e.g. sound generators, filters etc.) that can generally be connected in the form of a directed acyclic graph, and each module has quasi-continuous parameter inputs (defined intervals of floating point numbers, e.g. for a frequency, an amplitude etc.). This agent should be able to operate in real-time along with me as a human agent.

My idea is to use something like reinforced learning as it allows to operate in an exploratory mode where the state space is very large if not infinite, something that I understand is a disadvantage in a learning-by-example (supervised) mode.

Do you consider RL applicable in this scenario?
How would I go about defining the state space? Is every module (generator or filter) a cell in that space? How about their connections and parameters? Should I introduce a partial ordering of the graph, or radically simplify the state? Perhaps the state should be a feature vector of the actually produced sound instead of the graph structure?
I am thinking that I have to provide the rewards manually depending on whether the current result is musically meaningful or bad. Does that make sense? Is there any literature for this "human evaluated" RL? I know there are human fitness functions in genetic programming, so I am thinking this would be an analogous situation.
I am worried about the convergence rate. In automatic learning, I can easily perform thousands of steps, but if I have to assign manual reward and I have to listen to the actual sound production, the agent can only act at a fraction of the speed.


texts:
I'm coming from Stata and struggling to get used to group_by in dplyr. Perhaps using group_by is the wrong approach, but if you know Stata, you'll understand why I'm trying to use it (I think).

I'd like to transform the data as shown below:

Essentially, I'm trying to collapse each NAICS code into a single row, and to create columns for the 5 statistics under CONC and HHI for each NAICS code (the 4 values for CONC and one for HHI, ignoring the Xs)

The data is here. The code I'm trying is as follows:

ManufMktConc <- read.csv("Manufacturing.csv",
                            stringsAsFactors = FALSE)

ManufMktConc %<>%
  tbl_df %>%
  slice(-1) %>%
  filter(CONCENFI.display.label!="All companies") %>%
  select(-GEO.id,-GEO.id2,-GEO.display.label,-COMPANY,-RCPTOT,-YEAR.id)

ManufMktConc_byInd <- group_by(ManufMktConc,NAICS.id)

ManufMktConc_byInd %<>%
      arrange(CONCENFI.id) %>%
      mutate(FourFirm = ManufMktConc_byInd$CCORCPPCT[ManufMktConc_byInd$CONCENFI.id=="856"])

That gives me the following error:

> ManufMktConc_byInd %<>% arrange(CONCENFI.id) %>% mutate(FourFirm=ManufMktConc_byInd$CCORCPPCT[ManufMktConc_byInd$CONCENFI.id=="856"])
Error: incompatible size (651), expecting 4 (the group size) or 1

So I can tell that I am asking R to stick the 651 instances of ManufMktConc_byInd$CONCENFI.id == "856" into the group of 4. I guess this is where my understanding of group_by falls apart. Why isn't my logical statement only applied within each group?

Thank you.


texts: If the entire production dataset is known is advance, should I use it when fitting normalization parameters?

texts:
I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage.

I presume this recommendation assumes prediction data would flow to the model online, and the entire data is not known in advance.

Does this recommendation also apply to when the entire data is known in advance?
And even if not, I guess it is possible to refit the model every time we get new data, with normalization parameters acheived from the entire dataset seen until now?

What are the pros and cons of normalizing over the entire dataset before fitting (as opposed to only on the train set)?


texts:
I have a dataset consisting of products, clients, price policy, discounts, quantities, and net sales. The task as put in words by the business is quantity vs price. I have noted a few observations from looking at the dataset :

Discounts: Discounts nullify the effect of any change in the Price policy. So in the end the net sales don't follow this variation. And i observe this for so many client-product pairs.

Seasonality : Variation of quantity for client-product pairs simply follows a seasonality pattern and its not driven by any of the timeseries variables in the dataset. ( I should statistically verify this for now i just did a visual check).

Because at this point i dont see any logic behind how discounts are decided for the clients. Hence there is literally no affect on net sales vs price changes.
How should I model this ? Is this even a machine learning problem because there is simply no causal relation between the variables. If not Price vs Demand then what other things can I propose to the business ?
Edit : 1. Product-client scatter plots UNITARY_NET_SALES VS QUANTITY

The first column of the plot shows products are demanded at the same quantities across varying net sales. So no price vs demand effect here.

Timeseries plot for a particular product

Price_list and Discounts have the same behaviour. So whenever the business increases the prices they increase the discounts too, hence the overall affect on net sales is none. And Quantity simply follows a seasonal pattern.

Describe on the columns

75% of Quantity is less than 8 units !
Thank you !


texts: What is the difference between and Embedding Layer and an Autoencoder?

texts: I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what are the differences between them?


texts: Kmeans on mixed dataset with high level for categ

texts:
My retail dataset contains 3 numeric attributes and two categorical attributes  Time and ID with 50,000 records. Both categorical attributes have more than 20 thousand levels and their format is 1/11/2011 11:54 and 1TD10051 respectively.

How do I do kmeans on these dataset? Converting categorical to binary will give very sparse dataset?

How to proceed?


texts: How to evaluate the performance based on rate data

texts:
I have the following data:

Goal       Achieved

100        90
150        130
200        175

...

The first column "Goal" is the number which should be done that day, and the column "Achieved" is the number that was actually done that day. Each row stands for a day.

I want to use this data to assess the performance of a worker. Here below I have two ideas:

Method 1

Method 2

Which one of the methods is reasonable and mathematically right?

Or if neither is good, what should I do to evaluate the performance of a worker?


texts: General approach to work with text of phone calls (topics, promises, sentiments, etc.)

texts:
I have an NLP task. There is a text (telephone conversations). Voice is already converted into text and is divided into agent and customer paragraphs. I need to understand what approach is the best one for the next tasks:

Who is the customer and who is the agent?
Customer Name
The topic of conversation
Promises made by the operator to the customer (for example, "I call back tomorrow")
Negative Sentiment (if there is something in the conversation that the subscriber is not happy with)

I am just trying to understand how to handle it. Is it possible to create some kind of general approach for this? If yes, for which packages/publications/books could I pay my attention?


texts: Reframing multilabel classification with imbalance in "both" directions

texts:
Consider the multilabel problem when asking "does the sample belong to this class" with, for example, a movie label dataset where almost every movie is labelled "drama" because of this label's vagueness. It also has a few rare labels such as "French late 60s noir".
To both boost recall, as well as maintain compatibility with most classifiers, we can both resample and train a binary classifier for each label individually (drama: yes/no, noir: yes/no).
But while "French late 60s noir" might be a 0.05:0.95 imbalance that requires serious effort for the model to guess, the "drama" category is so widely applied that the problem is reversed, i.e. 0.95:0.05.
How should one logically best proceed to maintain recall over precision, now that the "target class" is 95% of the labels (kind of breaks with the definition)? Upsample the non-target class? Downsample the target class? Reframe the problem as "not drama" to maintain underbalance for every label?


texts: How do Ioffe & Szegedy obtain the equation $\frac{\partial \text{BN}((aW)u)}{\partial u} = \frac{\partial\text{BN}(Wu)}{u}$?

texts:
In the paper that introduced Batch Normalization, on page 5, the authors write the equation
$$\frac{\partial \text{BN}((aW)u)}{\partial u} = \frac{\partial\text{BN}(Wu)}{u}$$
Here $W$ is the matrix of weights connecting the layer $u$ to the next, batch-normalized layer, so the conclusion is that scaling the weights by a constant doesn't affect this partial derivative.
This seems false to me. Let $b$ be the value of some output neuron and $u$ the layer above, so that:
$$b=\sum w_iu_i$$
Now let $\hat b$ be the batch-normalized version of $b$:
$$\hat b = b - \frac1N\sum b^i$$
where by $b^i$ I mean the value of the neuron $b$ for the $i$-th training input in the batch. We have
$$\hat b = \sum w_iu_i - \frac1N\sum_j b^j= \sum w_iu_i - \frac 1N\sum_j (\sum_i w_ju_j^i)$$
Since the values $u_i$ never appear in the second sum we simply have
$$\partial_{u_i} \hat b = w_i$$
Which very much does scale with $W$. Am I making a mistake, or misinterpreting the original equation?


texts: Credit scoring using scorecardpy with XGBoost

texts:
I used XGBoost for scoring creditworthiness. At first I thought I could use predict_proba for scoring but then I saw that there was a module scorecardpy based on WOE to claculate code scoring. I tried to use it with my XGBoost like in an exemple but my ROC AUC fell to 0.5 and I don't see what I am doing wrong. Thanks for your help.

data = pd.read_csv('data.csv')

train_index = data['date'] < '2018-04-01'
test_index = data['date'] >= '2018-04-01'

data_final = data.drop('date', axis=1)

df_train = data_final[train_index]
df_test = data_final[test_index]

data_final_vars = data_final.columns.values.tolist()
y=['label']
X=[i for i in data_final_vars if i not in y]

# woe binning ------
bins = sc.woebin(data_final, y="label")
sc.woebin_plot(bins)

# binning adjustment
# # adjust breaks interactively
# breaks_adj = sc.woebin_adj(dt_s, "creditability", bins)
# # or specify breaks manually
breaks_adj = {
    'age': [26, 35, 40, 50, 60]
}
bins_adj = sc.woebin(data_final, y="label", breaks_list=breaks_adj)

# converting train and test into woe values
train_woe = sc.woebin_ply(df_train, bins_adj)
test_woe = sc.woebin_ply(df_test, bins_adj)

ytrain = train_woe.loc[:,'label']
xtrain = train_woe.loc[:, train_woe.columns != 'label']
ytest = test_woe.loc[:,'label']
xtest = test_woe.loc[:, test_woe.columns != 'label']

print("shape of xtrain: {}".format(xtrain.shape))
print("shape of xtrain: {}".format(xtest.shape))

from xgboost import XGBClassifier

XGB = XGBClassifier(n_estimators=100, n_jobs=6, verbose=1)
# List the default parameters.
print(XGB.get_xgb_params())

# Train and evaluate
XGB.fit(xtrain, ytrain, eval_metric=['rmse'], eval_set=[((xtrain, ytrain)),(xtest, ytest)])

# # Classifier

from sklearn.metrics import roc_auc_score

probs = XGB.predict_proba(xtest)
roc = roc_auc_score(y_true=ytest, y_score=probs[:, 1])
print("RF roc score: {}".format(roc))

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(ytest, probs[:,1])
plt.figure()
plt.plot(fpr, tpr, label='XGBoost Classifier (area = %0.2f)' % roc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.savefig('XGB_ROC')

from sklearn import model_selection
from sklearn.model_selection import cross_val_score
kfold = model_selection.KFold(n_splits=10, random_state=7)
modelCV = XGB
scoring = 'accuracy'
results = model_selection.cross_val_score(modelCV, xtrain, ytrain, cv=kfold, scoring=scoring)
print("10-fold cross validation average accuracy: {}".format(results.mean()))

# score ------
card = sc.scorecard(bins_adj, XGB, xtrain.columns)
# credit score
train_score = sc.scorecard_ply(df_train, card, print_step=0)
test_score = sc.scorecard_ply(df_test, card, print_step=0)

# psi
sc.perf_psi(
  score = {'train':train_score, 'test':test_score},
  label = {'train':y_train, 'test':y_test}
)


texts: Can I use the training rows multiple times while training with different labels attached to it?

texts:
If I have a data set where for every text message,  same but two labels are given. It could be that only one label has been filled. To visualise this scenario in real life, one may classify the accent to belong to 'US English' and 'UK English'.

Here is an example where fro every text, atmost 2 sentiments' values are given.

text - Sentiment - sentiment

Ron is very amicable person - 5 - 4
That is so not the place to enjoy - 1
Rita is not so caring - 1 - 2

Now, I have to predict SINGLE sentiment column for the given text column.

Can I create the training dataset as below for training purpose? What could be difficulties in doing so?

text - Sentiment

Ron is very amicable person - 5
Ron is very amicable person - 4
That is so not the place to enjoy - 1
Rita is not so caring - 1
Rita is not so caring - 2

Edit:
My focus is not on choosing a single class for given text but on the decision to include that text two times with different class attributed to it.


texts: I have a classification problem, where I initially started with 100+ class labels. My intuition told me this is too many labels for a model to predict with good accuracy. I thought grouping the labels together will help the situation and it did, there is now only 19 class labels. Is 19 still too many class labels? What are the typical signs that there are too many labels for a model to predict and how can this be addressed, if there is too many class labels?


texts:
In recent years, the term "data" seems to have become a term widely used without specific definition. Everyone seems to use the phrase. Even people as technology-impaired as my grandparents use the term and seem to understand words like "data breach." But I don't understand what makes "data science" a new discipline. Data has been the foundation of science for centuries. Without data, there would be no Mendel, no Schrödinger, etc. You can't have science without interpreting  and analyzing data.

But clearly it means something. Everyone is talking about it.  So what exactly do people mean by data when they use terms like "big data" and why has this become a discipline in itself? Also, if it is an emerging discipline, where can I find more serious/in-depth information so I can better educate myself?

Thanks!


texts: How can we handle cases were the quality of the prediction undermines the quality of the data?

texts:
Consider the following example:

You have a rare disease whose occurrence seems to depend on a certain number of variables. You build a model which tries to predict patients most likely to be effected by the disease which is partially successful, that is, it predicts likely onset of the disease with some accuracy, but less than desired. Ideally, you update your model as more historical data on patients with this disease comes in, and the accuracy starts to improve.

Eventually you start notifying high risk patients and provide them with steps to counter this disease. Because of this, more and more patients who would have been classified as high risk are actually not catching the disease and therefore decreasing the accuracy of your model. In a sense the model was a 'victim' of its own success.

Are there any strategies for dealing with such prediction scenarios: Where a model designed to predict an undesirable outcome looses accuracy due to it successfully averting the outcome in real world cases?


texts:
I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc.  The end goal is to predict churn rates for products in a large telco company, but the approach we use could apply to any similar problem.

The models produced in this way contain a few potentially useful performance measures such as variable importance, precision, recall and some others.  Each model has roughly 150 input variables.

The model scores have been used to group the customers by decile and measure the churn rate of each group.

The present situation is that the scores appear to be too good which suggests we may have a data leakage problem.  For instance, for one of the models the 1st decile captures 84% of the churn, with 99% of the churn captured by the 4th decile.

My task is to understand and report on potential issues with the model performance so we can improve the models and recommend action to the business.  What I would like to know is:

What are some basic analyses that I can perform to address the data
leakage issue.
How can I leverage the model metadata to better
understand model performance?
What other important questions should I
know to ask in order to fully address this task?


texts: Are there readily available models that can handle conditional correlation?

texts:
I've been working my way through the features of the Kaggle House Prices dataset (Note: this is a non-ranking entry, so this is just for exercises), and I've found a couple situations where there is a positive correlation between the feature and the house sale price, but only if the data exists. In one case (shown below) over 10% of the dataset had null (means "does not apply", not "missing", so I can't fill it in with imputation), but of the non-null values, a scatter plot showed a positive correlation. This looks like a conditionally useful feature and I'd like to keep it, but the nulls are tripping me up.
I experimented and replaced the null values with 0, and when I looked at the scatter plot again, I found that the values that the new 0s spanned the a good chunk of the price range and altered the trend.
Before replacing null values (blue trend line estimated by hand):

After replacing null values with 0 (~13% of the dataset):

Is there a readily available model in sklearn or some other python library that will perform a regression fit only if the data is non-null? If not, would it be best to just drop the column?
Note: This is a small dataset (<1500 entries). This is too small for neural network techniques.


texts:
My data consists of 2-dimensional arrays with shape (2,3).
The whole dataset (emp) consists of 12 items (I know that it is too small of a number for an NN training, but it is just a test), so it has shape (12,2,3)

truevals has shape (12,)

When I try to add Keras layer:

model = Sequential()
model.add(Dense(4, input_shape=(2,3), activation='tanh'))
model.add(Dense(1, activation='sigmoid'))
model.compile(SGD(lr=0.5), 'binary_crossentropy', metrics=['accuracy'])
model.fit(emp, truevals, epochs=200)

and then when I fit the model I get the exception:

  Error when checking target: expected dense_30 to have 3 dimensions,
  but got array with shape (12, 1).

What am I doing wrong?


texts: Reducing training data to a smaller subset for neural network regression task

texts: I have a large dataset of several billion inputs of categorical attributes. The neural network performs a regression task. Since I want to train the model, but not always use all of the data, how can I find a representative subset of the data that is representative for the whole dataset, while capturing the dependencies and generalizing to the other data? Currently I found only approaches for classification and ordinal attributes, where they find centroids as representatives, but for categorical data this is not useful.


texts: Why perceptron does not converge on data not linearly separable

texts:
This is how I understand the perceptron algorithm.

The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$.
Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with $y_i \in \{-1,1\}$.
The perceptron criterion is
$$
E(w) = \sum_i \ell(w, x_i, y_i) = \sum_i \max(0, -y_iw\cdot x).
$$
The perceptron algorithm tries to minimize $E(w)$ using gradient descent.
The gradient of $E(w)$ with respect to $w$ is
$$
\nabla E(w) = \sum_i \left\{
\begin{array}{ll}
\mathbf{0} & \text{ if } \ell(w,x_i,y_i) = 0\\
-y_ix_i & \text{ otherwise }
\end{array}\right..
$$
So the perceptron updates the $w$ according to
$$
w^{(t+1)} = w^{(t)} - \eta \nabla E(w^{(t)}) = w^{(t)} + \eta \sum_{i\in M(t)} y_ix_i
$$
where $M(t)$ is the set of points misclassified by $w^{(t)}$. Since we can define another equivalent sequence of weight vectors $v^{(t)} = (1/\eta)w^{(t)}$, we can just take $\eta = 1$.

I know that $w^{(t)}$ will converge to a seperating hyperplane if one exists and $E(w)$ will converge to $0$.

My question is about when the data set is not linearly separable. In that case, does $E(w)$ converge to some minimum?

If $E(w)$ attains some minimum, its gradient there has to be $\mathbf{0}$. One possibility is that all the terms in the sum of $\nabla E$ are 0, which means that the dataset is linearly seperable and this is contradiction. The other possibility is that the sum of $-y_ix_i$ over the misclassified points is somehow $\mathbf{0}$.


texts: What are the most popular data science application use cases for consumer web companies

texts:
The most popular use case seem to be recommender systems of different kinds (such as recommending shopping items, users in social networks etc.).

But what are other typical data science applications, which may be used in a different verticals?

For example: customer churn prediction with machine learning, evaluating customer lifetime value, sales forecasting.


texts:
I am new to deep learning. I am working on training an SSD model on a set of small objects. I am using Adam gradient descent for optimization and a large input (800x800), but I seem to only get an improvement of 0.010 after every 20 or so epochs(350 steps).

What can I do or look for to speed up convergence on this model?


texts:
Say I have a scatter plot like this:

Since I have many scatter plots like this I want to do feature transformation i.e. squash (x,y) in a single term to be input to a NN. Which transformation like x/y or (x/y)^2 or any other transformation will work best in this type of graphs i.e. increase the separation more while squashing it to a single term.


texts: Predict when a customer would buy from previous buying history (Mining Buying patterns)

texts:
I'm trying to predict when a customer would buy a product(which month of the year) based on the historical transactions. I have 3 years transactions data and good amount of customers are frequent ones.
Or basically want to mine the historical transaction data, and see for a set of customers (in a Geo/type of customer etc.) which months are likely period of buying a product X.

Can someone please advice which Machine learning technique or statistical approach would be best in this case.

I am trying
- Decision tree classification/Logistic regression model taking month as predictor variable.
- Statistical analysis to see which month is more significant for a set of customers.


texts:
I want to know how does one predict day to day weathers. Like what are the factors that must be known to predict day to day weathers using a NN. Few factors I could think of are:

Humidity
Weather history
Wind speed

What are some factors that a meteorology ignorant data scientist would require to correctly predict the weather?


texts: Identifying Waveform Segments Using Training Waveforms

texts:
Problem : There are several events (eventA, eventB,....) represented by waveforms. For each event there are several csv files (eventA1.csv, eventA2.csv,...eventAn.csv) having the points(x,y) from which the corresponding waveform can be generated.

Using these waveforms as training data, waveform segments on (test data) larger data should be identified i.e. the interval during which an event occurs is needed to be labelled on the test data (See image for reference).

What would be the best strategy, tools and algorithms to address this problem?
If classification model is to be generated then,

The points are just to describe the nature of the waveform i.e. the test waveform will not be on same points but with similar nature e.g. the concavity, convexity, slope etc.
Different events in training graphs are represented using different number of points i.e. event A can be determined by 20 points, while event B has 200 points.

So,

How do I create feature vectors to include all these things for classification?
How would the test data be analysed, as it has large number of points on the large waveform i.e. how to segment it to generate features and give it to classification model?


texts: What exactly is a step in Tensorflow prebuilt architectures?

texts:
When running the object detection tutorial, you can use train.py which is supplied. On the console, the following is printed :

INFO:tensorflow:global step 1595: loss = 2.5473 (46.045 sec/step)
INFO:tensorflow:global step 1596: loss = 2.9730 (37.213 sec/step)
INFO:tensorflow:global step 1597: loss = 2.4274 (26.814 sec/step)
INFO:tensorflow:Recording summary at step 1597.
INFO:tensorflow:global step 1598: loss = 2.2615 (38.410 sec/step)
INFO:tensorflow:global step 1599: loss = 2.5232 (26.713 sec/step)
INFO:tensorflow:global step 1600: loss = 2.3982 (28.633 sec/step)
INFO:tensorflow:global step 1601: loss = 2.4807 (29.261 sec/step)
INFO:tensorflow:Recording summary at step 1601.

This information is also displayed on Tensorboard while monitoring the performance of training :

At first glance, I believed that one step is one epoch, but apparently this isn't true.

What exactly is a step in Tensorflow / Tensorboard?

Example came from here.


texts: How to calculate temporary/periodic similarities of an increasing series in real time?

texts: Considering there are two series over time and new data is added in the series over a gap of n second . The series might have periodic similarity/dis-similarity within themselves. How to calculate correlation among the series value in real time?


texts:
I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records in the dataset are 3000. All e-mails are not in proper English or format, there are a lot of spelling mistakes and unwanted features as Indian Names, addresses etc.. The dataset is very imbalanced, the frequency of top 3 categories combines upto 2000 and there are 2 categories with frequencies ranging from 40-80.
I managed to pre-process the data and remove most of the unwanted stuff and spelling mistakes.

I have created a supervised learning model using LinearSVC which gave around 78% on validation dataset, after which I used XGBoost classifier which increased the accuracy to 80%. Both of these models use Tfidf Vectors with unigrams and bigrams of the pre-processed(stop words removed and stemmed text) as input.
I have tried SMOTE, TruncatedSVD and SelectKBest with chi2 but they decreased the accuracy.

In terms of Machine Learning Models I have used Multinomial Naive Bayes, Random Forests, Decision Tree Classifiers. I have even tried Feed Forward Neural Networks(MLP) and Recurrent Neural Networks for the classification but still XGBoost gives the best accuracy. I have also used Topical Modeling using LDA. I generated top words for each category and tried finding cosine similarities of each test set record with all the generated top words belonging to each category, still no improvement in accuracy.

So, my question is what kind of features should I look for? How do I extract more sophisticated features? My main objective is to increase the accuracy as the product is going to be an in-house software for the company. I am looking forward to context aware feature extraction or somehow engineer the features(add more or enhance them) to increase accuracy

PS: In the emails the customer ask for multiple actionable items, but still in the training set they are pre-classified as only one label corresponding to one of the actionable item asked in the e-mail. Plus, the data is also manually  wrongly classified by the previous workers. Please suggest and approach to capture the actionable items. I Have tried using class probabilities but they don't seem like giving the right answer, plus if the person has asked for only one actionable item then also the class probabilities will indicate other. I want to capture the actionable items as per asked by User. I have tried POS-tagging for this but due to bad quality of data, it is not giving good results.

PPS: The Email data which I have is in HTML format and has threads. I tried removing previous conversation threads before training but it resulted in bad accuracy. It is very strange that the more I try to refine the data by removing unwanted contents like disclaimers, auto-response emails, replies from customer-care representatives the more the accuracy falls.

Looking forward to get some insights on how to change my approach and specifically the features.

Thank You


texts: What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?

texts:
I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.

One approach is to use the following code:

distortions_2.append(sum(np.min(cdist(data
                              , kmeanModel.cluster_centers_
                              , 'euclidean')
                              , axis = 1)) / data.shape[0])

Another is to use inertia_ from sklearn.cluster.KMeans:

distortions_3.append(kmeanModel.inertia_)

When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?


texts:
I have a dataset with 20 variables and ~50K observations, I created several new features using those 20 variables.

I compare the results of a GBM model (using python xgboost and light GBM) and I found that it doesn't matter what are the hyper-parameters of the model the 'thinner' version leads to better results (AUC) even that all 20 original variables are included in the wider version.

when I compare the same using Lasso model - the wider version is better (~1% higher) as expected.

I guess it can be related to to the randomness in the GBM but I was surprised to see that GBM doesn't fix it along the way.

Any explanation of the phenomena will be appreciated.


texts: Logistic regression, where is my mistake

texts:
I am doing assigment on Logistic Regression on Andrew Ng DL course, and can't understand where is my mistake,

# FORWARD PROPAGATION (FROM X TO COST)
### START CODE HERE ### (≈ 2 lines of code)
A = sigmoid(np.dot(w.T,X) + b)                       # compute activation
cost = -1/m * np.sum(np.dot(np.log(A), Y.T) + np.dot(np.log(1-A), (1-Y.T)))  # compute cost

### END CODE HERE ###

# BACKWARD PROPAGATION (TO FIND GRAD)
### START CODE HERE ### (≈ 2 lines of code)
dw = 1 / m *(np.dot(X,(A - Y).T))
db = 1 / m *(np.sum(A - Y))
### END CODE HERE ###

assert(dw.shape == w.shape)
assert(db.dtype == float)
cost = np.squeeze(cost)
assert(cost.shape == ())

grads = {"dw": dw,
         "db": db}

return grads, cost

w, b, X, Y = np.array([[1.],[2.]]), 2., np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1,0,1]])
grads, cost = propagate(w, b, X, Y)
print ("dw = " + str(grads["dw"]))
print ("db = " + str(grads["db"]))
print ("cost = " + str(cost))

Giving this error

ValueError: shapes (2,) and (3,1) not aligned: 2 (dim 0) != 3 (dim 0)

Please help
Thanks a lot


texts: Anomaly Detection in Highly Variable Time-Series Data

texts:
I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this:
datetime         |  count
_________________|________
2021-03-31 00:05 |  25
2021-03-31 00:10 |  13
2021-03-31 00:15 |  0
2021-03-31 00:20 |  3
...              | ...
2021-04-15 22:10 | 111
2021-04-15 22:15 | 0
2021-04-15 22:20 | 9

However, the variance on the count column is huge, as a results of which when I am trying to use rolling z-score technique with a window of 288 (every 5 mins, means 12 data points in an hour, 24*12 = 288, 1 day window) and a threshold of -3, +3 as per central limit theorem, its failing because for most of the data points the scores are way outside these ranges as a result of which around 50% points are being declared as anomaly. Similarly I have already tried Isolation Forest, One Class SVM and Elleptic Envelope technique but nothing seems to be working. When I try to plot the anomalies, the graph gets filled with anomaly points, and the plot also doesn't make any sense.
I haven't worked with this kind of data earlier and I think I am going wrong somewhere. Can anyone suggest me what steps should I do to detect anomalies in this type of data and what might be some good algorithms which I can try?


texts: How to account for rare events at different time intervals while using LSTM neural networks?

texts:
I'm working on an interesting sequence-to-sequence (regression) time series problem where some static features/rare events can change the behavior of future time series. The problem is a forecasting problem, where I use previous time step values to forecast the next time step values and I try to integrate static features + rare events into time step t=0.
In my problem, there is always a rare event at t=0 in addition to some static features that should affect the future behavior of time series.
For clarity, My definition of "rare events": an event that happens at a specific time step (for ex: t=0) and another separate event can happen at any time in the future as well (for ex: t=n) in addition to the event that happened at t=0 but, it happens only once at that time and both events can affect the future time series behavior starting from the time they occurred.
Even though most of the static features don't change over time, the rare events can be different from each other (has different characteristics/features). The time of each event is usually known because it will be applied due to outside human intervention to optimize the future behavior (increase profit) but, they do not necessarily happen at the same time step for every sample/example.
These events are so rare that it kind of makes sense to me to treat them as static features at time=0 but, I can't think of a way to include a rare event that happens n timesteps later in the future and has different characteristics than the event at t=0.
Below is an example schematic of the problem. There may be multiple samples with varying time steps affected by these unique rare events but, if I don't account for these events, I believe my predictions may suffer.
Can anyone suggest any sources to look at for these types of problems? I may also be missing key words that are usually used with these types of problems and that may be one of the reasons why I'm still having difficulties finding good sources. I call it "rare events" but, it may be called something else in the literature...  At this point, I appreciate any type of source that addresses this issue such as scientific papers/articles, github code or a code example provided by you, correct keywords to search for, etc.
Thank you.


texts:
I did small survey and get such data:

|-------------| Yes | No | Dont_Know |
|-------------|     |    |           |
| Employee    | 60  | 5  | 5         |
| Workers     | 17  | 0  | 1         |
| Businessmen | 71  | 5  | 10        |
| Jobless     | 4   | 30 | 0         |

R code

dt <- data.frame(workers = c("Employee",
                             "Workers",
                             "Businessmen",
                             "Jobless"),
                 yes = c(60,17,71,4),
                 no = c(5,0,5,30),
                 dont_know = c(5,1,10,0)
                )

What kind of test I must do, if I want to show, that the Jobless people are often choosing No answer?
Is the difference between Jobless and Businessmen answers significant?
And what is about other groups?
What another information I can get from such data or what kind questions I can ask from such data?


texts: Can I sum up feature vectors of a user‘s collection?

texts:
I want to find items that are similar to items users already have in their collection. Every item has attributes, so I created feature vectors where every element of the vector represents an attribute and is either $0$ or $1$ (if an item has that attribute).

For the user collection I summed up all vectors, creating one vector which I then used to calculate similarities with other items.

Is this a correct approach or should I make this "user vector", binary like the other ones? Or is it easier to just calculate $n \times m$ (I.e. user items and new items) similarities?

The set of new items will consist of $\sim1000$ items, while the user collections tend to be $<1000$. As similarity function I used cosine distance, but wanted to try Pearson coefficient as well.


texts:
I'm working on a sequence to sequence approach using LSTM and a VAE with an attention mechanism.

p = np.random.permutation(len(input_data))
input_data = input_data[p]
teacher_data = teacher_data[p]
target_data = target_data[p]
BUFFER_SIZE = len(input_data)
BATCH_SIZE = 64
embedding_dim = 100
units = 256
vocab_in_size = len(input_lang.word2idx)
vocab_out_size = len(target_lang.word2idx)

# Create the Encoder layers first.

encoder_inputs = Input(shape=(None,))
encoder_emb = Embedding(input_dim=vocab_in_size, output_dim=embedding_dim)
encoder_lstm =LSTM(units=units, return_sequences=True, return_state=True)
encoder_outputs, state_h, state_c =
encoder_lstm(encoder_emb(encoder_inputs))
encoder_states = [state_h, state_c]
 #################### Adding VAE #######################
latent_dim =256
# output layer for mean and log variance
z_mu = Dense(latent_dim)(encoder_outputs)  #remplacer h
z_log_var = Dense(latent_dim)(encoder_outputs)
def sampling(args):
     batch_size=1
     z_mean, z_log_sigma = args
     epsilon = K.random_normal(shape=(batch_size, latent_dim),
                          mean=0., stddev=1.)
     return z_mean + K.exp(z_log_sigma) * epsilon

z = Lambda(sampling, output_shape=(latent_dim,))([z_mu, z_log_var])
state_h= z
state_c = z
encoder_states = [state_h, state_c]

def vae_loss(y_true, y_pred):

    recon = K.sum(K.binary_crossentropy(y_pred, y_true), axis=-1)
    kl = 0.5 * K.sum(K.exp(z_log_var) + K.square(z_mu) - 1. - z_log_var,
       axis=-1)
     return recon + kl[:, None]

 ##########################

# Now create the Decoder layers.
decoder_inputs = Input(shape=(None,))
decoder_emb = Embedding(input_dim=vocab_out_size, output_dim=embedding_dim)
decoder_lstm = LSTM(units=units, return_sequences=True, return_state=True)
decoder_lstm_out, _, _ = decoder_lstm(decoder_emb(decoder_inputs),
    initial_state=encoder_states)

# Attention layer
attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([encoder_outputs, decoder_lstm_out])

When I execute this code I get this error at the last line:

  ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

I tried all solutions given for this error, no one solved my problem, if any one can help, I'll be so thankfull.


texts: How to combine two different embeddings in the best way possible?

texts:
I have two models which are giving two books embedding
Ml_model_a => book1_embedding [ 1, 200 ]

Ml_model_b => book2_embedding [ 1, 200 ]

I am building a third model which will take these two different embeddings to tell me which book to choose.
Now my final layer is the classification between 0,1 ( which book to choose ). How to learn these embeddings best way possible to classify better?
What I have tried yet :

If I am averaging those embeddings and then sending to one model, then embeddings are losing a lot of information, so I am using concatenation method.
But it's not classifying well, Is there any other model, a technique I could use to enhance the capability of learning to book embedding and predict which book to take?


texts: Object Detection scratch implementation or high level API?

texts: I am at an intermediate level in Deep Learning. I wanted to implement object detection from scratch. Is the scratch implementation possible by a single person or will i have to use an high level API to do so. I am confused and don't know where to start.


texts: including Zeros counts categories with pandas value_counts()

texts:
What i want is including zero counts categories while generating frequencies for categorical variables
example:

col1   col2
a       x
c       y
a       y
f       z

what i want is to generate a frequency table with counts and percentages including zero counts categories
results
  Counts   Cercentage
a  2          50.0%
b  0          0.0%
c  1          25.0%
d  0          0.0%
e  1          25.0%

what i have done is generating the frequency table with counts and percentages but i need to include also the zero counts categories like b and d as illustrated above
here is what i have tried
pd.concat([df.col1.value_counts(dropna=False),
           df.col1.value_counts(normalize=True,
                                dropna=False).mul(100).round(1).astype(str) + '%'],
          axis=1,
          keys=('Counts','Percentage'))

any help please


texts: How to convert DataFrame column to Rows in Python?

texts:
I have the following dataset in df_1 which I want to convert into the format of df_2. In df_2 I have converted the columns of df_1 to rows in df_2 (excluding UserId and Date). I looked up for similar answers but they are providing little complex solutions. Is there a simple way to do this?
df_1
   UserId       Date                   -7  -6  -5  -4  -3  -2  -1   0   1   2   3   4   5   6   7
    87      2011-05-10 18:38:55.030     0   0   0   0   0   0   1   0   0   0   0   0   0   0   0
    487     2011-11-29 14:46:12.080     0   0   1   0   0   0   0   0   0   0   0   0   0   0   0
    21      2012-03-02 14:35:06.867     0   1   0   1   2   0   2   2   0   1   2   2   1   3   1

df_2
day | count
-7   0
-7   0
-7   0
-6   0
-6   0
-6   1
-5   0
-5   1
-5   0
.    .
.    .(Similarly for other columns in between)
.    .
6   0
6   0
6   3
7   0
7   0
7   1


texts: Using GPS signal, determine is this person driving a cab

texts:
New York City provides tens of gigs of data of taxi routes all over the city. What I'd like to do, is use this data (or some other method), to come up with an algorithm that can take a persons GPS data over a short span of time (say an hour), and answer the question: Is this person driving a cab?

The algorithm should work in any location, not just NYC. The idea is that I'd like to be able to determine patterns, that signal that a route being driven by a person is the type of route a person driving a cab would take.

Ideally, I'd like to write this in Ruby. But I am open to other suggestions, approaches, and implementations. Links to projects I should research, suggestions on languages to use, approaches to take, etc are all appreciated.


texts: What is the difference between SVM and GMM classifier

texts: What is the difference between support vector machine and Gaussian mixture model classifiers?


texts:
in order to score features, in ORANGE, using ANOVA scoring, the features should have a normal distribution?
Thank you,
J


texts: How to compute a confidence interval for AUC?

texts: I found that in results of several binary classification problems, people report an AUC value together with a CI. I wonder how those CIs are computed. Is there a close-formed formula to compute them or they are computed via some method like bootstrap?


texts: What are the latest and most succesful classification algorithms at the moment?

texts:
I have googled this for some time with no luck. All i get are tutorials or articles explaining the classic algorithms like linear regression, random forest, etc.

I would like to know which are the most succesful recent algorithms at this moment for classification with an implementation in Python. I'm interested specially on ensemble methods, such like XG Boost, Catboost and Light Gradient Boosting Machines. Are there any other algorithms similar to these 3 mentioned above with proven sucess capabilities? I would appreciate any help from experienced people that have tried any.

I know there is not such a thing like the best algorithm for anything, what i want to know is if there are any other algorithms that perform the same or better than the ones mentioned above in general. The goal is to solve classification problems using tabular data and supervised learning.

Thanks in advance


texts:
I trained an SVM model with GridSearch
svc = SVC()
parameters = {
  'kernel': ['linear', 'rbf'],
  'C': [0.1, 1, 10]
}

cv = GridSearchCV(svc, parameters, cv=5)
cv.fit(v_train, y_train)

print_results(cv)

Here is the result I got:
BEST PARAMS: {'C': 1, 'kernel': 'linear'}

0.912 (+/-0.037) for {'C': 0.1, 'kernel': 'linear'}
0.763 (+/-0.027) for {'C': 0.1, 'kernel': 'rbf'}
0.942 (+/-0.045) for {'C': 1, 'kernel': 'linear'}
0.903 (+/-0.044) for {'C': 1, 'kernel': 'rbf'}
0.94 (+/-0.043) for {'C': 10, 'kernel': 'linear'}
0.928 (+/-0.046) for {'C': 10, 'kernel': 'rbf'}

What is the best way to plot this result? in one plot that both contain the C parameter and kernel with their corresponding accuracy.
Thank you.


texts: Classification procedure using Minimum Redundancy Maximum Relevance (MRMR)

texts: After selecting features with MRMR (by quantizing original feature space of training data), should we classify the test data using quantized values or original values?


texts:
Are there any machine learning packages for R that can make use of the GPU to improve training speed (something like theano from the python world)?

I see that there is a package called gputools which allows execution of code on the gpu, but I'm looking for a more complete library for machine learning.


texts: Cheat Sheet of UNIX commands for Data Science

texts:
I was looking for a cheat sheet of UNIX commands, which are specifically usable for data science. I mean an introduction to very basic commands (starting from cd, ls, pwd, to some still simple but usable for data - e.g. wc, a few simple things with pipes, ssh, maybe s3cmd, etc), with some minimalistic examples.

I couldn't find one; or at least, nothing close to:

Git Cheat Sheet
Regular Expressions Cheat Sheet
series of R Cheat Sheets

I am pretty much aware of datascienceatthecommandline.com, but my goal would be to have a 1-2 A4 pages printout, with really simple stuff.

If there is one (but I missed it), I would be grateful for posting it as an answer. If there isn't, then I am asking to post pieces of such list.

(If Moderator permits, we may create one list with Community Wiki.)

EDIT: The closest thing I've found is here: Useful Unix commands for data science (very nice, but missing the very introductory part).


texts: How to explain the connection between the input layer and H1 of this CNN Architecture?

texts: I am currently reading the paper proposed by LeCun et al. for handwritten zip code recognition. There is this figure below visualizing the CNN architecture. But I do not really understand how the connection between Layer H1 and input layer makes sense. If there are 12 kernels with size 5x5, shouldn't the layer H1 be 12x144? Or is there any downsampling taking place here too?


texts:
I'm having trouble graphing Pandas grouped data in Bokeh.

company_id    company_score    date_submitted    company_region
AA            .07              1/1/2017          NW
AB            .08              1/2/2017          NE
CD            .0003            1/18/2017         NW

I've successfully grouped the data by company_id, and calculated the simple moving average of company_score using 10 days / inputs.

score_by_company = df['company_score'].groupby(df['company_id']).rolling(10).mean()

company_id    index           SMA
BBG000B9WMF7  7613            NaN
              1911663         NaN
              1911664         NaN
              1911665         NaN
              1911666         NaN
              1911667         NaN
              1911668         NaN
              1911669         NaN
              1911670         NaN
              1911671    0.000002
              1911672    0.000002
              etc.       etc.

How can I translate this data into a time series graph, with each company_id being its own line?

I feel like I need to create a data dictionary off of the grouped data, but I'm not sure if that's the right approach.


texts: Can training examples with almost the same features but different output cause machine learning classification algorithms to perform poorly?

texts:
We usually filter out features (columns) that have low correlation or no significant impact on target variable. How would an algorithm, being trained with high dimensional data set (let’s say, more than thousands of features) contain rows with very high correlation but have different target variable, perform? Wouldn’t it make the ML algorithm confused in classification task?

Let me give a simple example to explain what I mean. Assume, we are given the price of a car and the task is to classify it as either of ‘Cheap Car’, ‘Budget Car’, ‘Luxury Car’, and ‘Elite Car’. Further assume, the distance between two rows is generally expected to be greater than 1000. For example, if a row describes a Car with price 1000, the next higher level car in our classification expected to be at least 2000. What if there is some anomaly in data set like a car with price 1000 classified as ‘Cheap’ whereas a car with price 1050 classified as ‘Elite’. That is grossly wrong. We eliminate irrelevant features. Shouldn’t there be something to eliminate confusing training examples?


texts: Why don't tree ensembles require one-hot-encoding?

texts:
I know that models such as random forest and boosted trees don't require one-hot encoding for predictor levels, but I don't really get why. If the tree is making a split in the feature space, then isn't there an inherent ordering involved? There must be something I'm missing here.

To add to my confusion I took a problem I was working on and tried using one-hot encoding on a categorical feature versus converting to an integer using xgboost in R. The generalization error using one-hot encoding was marginally better.

Then I took another variable and did the same test, and saw the opposite result.

Can anyone help explain this?


texts: Has anyone tried to search for new basic machine learning operations like convolution and pooling?

texts:
From Neural Architecture Search: A Survey, first published in 2018:

  Moreover, common search spaces are also based on predefined building
  blocks, such as different kinds of convolutions and pooling, but do
  not allow identifying novel building blocks on this level; going
  beyond this limitation might substantially increase the power of NAS.

I wonder now if somebody tried that since this was written or if anybody has some thoughts about the feasibility of this idea.


texts:
I am seeking to evaluate logistic regression against a rule-based (nested if-else statements) classification in Orange.
The steps I've taken to investigate this are as follows:

I could not identify any widgets that would implement this. The closest I could find was the interactive tree from version 2.7, but this widget seems to have been discontinued.
I've attempted to generate the results I'm looking for by editing the in_data object produced by creating a tree, sending the results to the "Test and Score" widget, and then editing the results in "Python Script" to create an out_data object. However, the out_data object I produce won't work within the confusion matrix.

Is there an easier way to implement this rule-based criterion in Orange? If not, are there some other steps I can use to implement it within the Python script?
Thank you,


texts:
I came across the concept of retraining of models a few days back.
So tensorflow recommends to retrain an already trained model instead of making a new one from scratch so that the whole process takes less time. But I can't understand how this is going to work.
Like if I take a pretrained model (trained to classify cats and dogs), how can I use its parameters to train a whole new model that I want to make (to classify different flowers).


texts: Disadvantages of hyperparameter tuning on a random sample of dataset

texts:
I often work with very large datasets where it would be impractical to check all relevant combinations of hyperparameters when constructing a machine learning model. I'm considering randomly sampling my dataset and then performing hyperparameter tuning using the sample. Then, I would train/test the model using the full dataset with the chosen hyperparameters.

What are the disadvantages of this approach?


texts: Fill the null values in the dataframe with condition

texts:
traindf[traindf['Gender'] == 'female']['Age'].fillna(value=femage,inplace=True)
I've tried to update the null values in the age column in the dataframe with the mean values.Here I tried to replace the null values in the age column of female gender with the female mean age.But the column doesn't get updated.why?


texts: How to visualize categorical data with numeric columns

texts:
I have data like as shown below

I would like to represent the above tabular data in a visual form.
However, the below graph may not work because my real data as 50K unique drug names.

So, is there any better way to represent this? Can you share some suggestions on how this can be represented?


texts:
I have few questions for which I could not extract answers from text books and online tutorials. Therefore, will be extremely grateful if the following points are clarified.

1) If I want to apply SVM, MLP and decision trees and combine the prediction results from these learners. So, I will have a mixture of ensemble of SVM, MLP and Decision trees OR I can use one learner only and have an ensemble of decision trees, an ensemble of SVMS and an ensemble of MLPs. Is my understanding correct?
How do I combine the prediction results from different models?

2) In ensemble learning be it homogeneous and heterogeneous, each of the learner is trained on the same data set? Is this technique known as Bagging  or simply ensemble learning? Would I be using the same subset of data for training each learner or different?

2) I have read in many tutorials that bagging and boosting is performed on similar learners. But ensemble learning can be homogeneous and heterogeneous. As a result, can bagging and boosting be performed on different learner types?

3) In Boosting, the learners are trained sequentially and therefore often only a learner of one kind is used as it is probably difficult to know if the sequence influences the learning procedure. Am I correct?

4) How is stacking different from the rest?


texts: Processing csv file with more than 700K rows of data

texts:
I have a .csv file (around 400MB in size) which contains 700K rows of structured data.
Table structure is:

+----+----------+-------+-----------+--------+--------+
| Id | Category | List1 |   List2   | Value1 | Value2 |
+----+----------+-------+-----------+--------+--------+
|  1 |        1 | A,B,C | Cat1,Cat2 |    100 |      5 |
|  2 |        1 | D,F   | Cat1,Cat4 |    120 |      4 |
|  3 |        2 | E,A   | Cat3      |    140 |      2 |
|  4 |        2 | E,A   | NULL      |    110 |      3 |
|  5 |        3 | B     | Cat2      |    100 |      6 |
+----+----------+-------+-----------+--------+--------+

For each row (CurrentRow) of the table I want to calculate:
CurrentRow.Value3 = SUM(Value1)/SUM(Value2) of all other rows (OtherRow) in the table where the following conditions are met:

Condition1 = CurrentRow.Category != OtherRow.Category
Condition2 = CurrentRow.List1 intersects OtherRow.List1
Condition3 = CurrentRow.List2 intersects OtherRow.List2 or CurrentRow.List2 = NULL or OtherRow.List2 = NULL

Also, I want to list Ids of rows which where involved into calculation of Value3.

Example for first row:
Condition1: first row has category value 1. As a result, rows 3-4 meet consition #1 because their category values are not equal to 1.
Condition2: column "List1" has values "A,B,C" which intersects with values "E,A" (row #3), "E,A" (row #4), "B" (row #5).
Condition3: Column "List2" has values "Cat1,Cat2" which intersects with values "Cat1,Cat4" (row #2), "Cat2" (row #5) and also we take row #4 as it has "NULL" value.

As a result, we take rows #4 and #5 as they both meet all conditions.
Value3 = (110+100)/(3+6) = 210/9 = 23.33
Ids = "4,5"

The result for table above would be:

+----+----------+-------+-----------+--------+--------+--------+------+
| Id | Category | List1 |   List2   | Value1 | Value2 | Value3 | Ids  |
+----+----------+-------+-----------+--------+--------+--------+------+
|  1 |        1 | A,B,C | Cat1,Cat2 |    100 |      5 | 23.33  | 4,5  |
|  2 |        1 | D,F   | Cat1,Cat4 |    120 |      4 | NULL   | NULL |
|  3 |        2 | E,A   | Cat3      |    140 |      2 | NULL   | NULL |
|  4 |        2 | E,A   | NULL      |    110 |      3 | 20     | 1    |
|  5 |        3 | B     | Cat2      |    100 |      6 | 20     | 1    |
+----+----------+-------+-----------+--------+--------+--------+------+

I have tried to do it using pandas in python:

data = pd.read_pickle('data.df')
for _index, _record in data.iterrows():
    category = _record["Category"]
    list1 = _record["List1"]
    list2 = _record["list2"]
    candidates = data.loc[(data['Category'] != category) & (pd.Series(list1).isin(data["List1"]).any()) & ((pd.Series(list2).isin(data["List2"]).any()) | (data["List2"][0] == 'NULL') | (list2[0] == 'NULL'))]
    value1_sum = candidates["Value1"].sum()
    value2_sum = candidates["Value2"].sum()
    ids = candidates[['Id']].to_numpy()
    if value2_sum > 0:
        data.loc[_index, "Value3"] = value1_sum/ value2_sum
        data.loc[_index, "Ids"] = ids
data.to_csv('result.csv')

It works for small number of rows, but takes forever on 700K rows.
Is there any method to optimize this algorithm?


texts: Image segmentation training set labeling

texts:
I am new to pytorch and Deep learning. I am trying to do image segmentation.
But , I am stuck at how to label training set images.

Can anyone please help me ?

This is one of my training image

I have two kinds of plants here - one is weed and another one is a good crop. I need to label them.

Can anyone tell me how can I do this ?


texts: Quality check for preprocessing of Text data

texts: I have developed a pipeline for text data preprocessing with different clean up techniques like Stemming , Lemmatization, Stop words removal etc. But now the ask from the business team is to quantify the quality of the preprocessing steps (or, the text data it produced). How can we develop some metrics to evaluate the preprocessing quality of text data?


texts: What is the “learning” step in Gaussian Naive Bayes classification?

texts:
For conditionally independent features $f_i$, Naive Bayes Classification gives me the classifier

$Classifier(f) := \arg \max_{k} P(C=k) · ∏^n_{i=1} P(f_i|C=k)$

for classes $k$. I understand that for Gaussian Naive Bayes, I can assume normally distributed features, yielding

$Classifier(f) := \arg \max_k P(C=k) · ∏^n_{i=1} \frac{1}{\sqrt{(2πσ_{k,i})}} e^{( -\frac{(f_i - μ_{k,i})^2}{2σ_{k,i})}}$

where $μ_{k,i}$ is the mean of class $k$ and feature $f_i$ (and similiar for variance $σ_{k,i}$).

But where is the "learning step" in this whole procedure?


texts: Cross-entropy via softmax: Mathematical derivation

texts:
I am trying to understand the derivation of cross entropy loss in the context of softmax. However, some steps are still not clear to me. Hence, I would appreciate if someone could please explain.
y = one hot encoding (representing true label) = selector
$y ̂= $value predicted by NN and the softmax.
Through the NN:
$X\xrightarrow{\smash{NN, θ}} Z \xrightarrow{\smash{Soft-max}} \hat{y}$
To derive the loss function, we start out from the likelihood function that a given set of parameters θ of the model can result in prediction of the correct class of each input sample. The maximization of this likelihood can be written as:     $$ arg max _θ L(θ|y ̂,Z) $$
The likelihood $ L(θ|y ̂,z) $ an be rewritten as the joint probability of generating t and z given the parameters θ, which can be written as a conditional distribution:
$$ L(θ│\hat{y} ,Z)=P(\hat{y} , Z│θ)=P(y ̂│Z,θ)P(Z│θ) $$
Since we are not interested in the probability of z we can reduce this (assumption 1):
$$ L(θ│y ̂,Z)∝P(y ̂│Z,θ) $$
Which can be written as $P(y ̂|z) $ for fixed θ (assumption 2).
Since each yc is dependent on the full z and only 1 class can be activated in the t we can write
$$ P(y ̂│Z)= ∏_{(c=1)}^C(P(y ̂_c |Z))^{y_c} =∏_{(c=1)}^C((SoftMax(Z))^{y_c }=∏_{(c=1)}^C(y ̂_c )^{y_c} $$
Assumptions 1 and 2 are not clear to me. Some explanation would be appreciated.


texts: Why some people add results from PCA and other dimensional reductions techniques as features

texts: I often see in a well-known datascience competition platform, that a lot of people apply some dimension reductions techniques, but instead of using it to reduce the number of features (complexity) of their models, they append the resulting features to their datasets. Is'nt that adding complexity to their model rather than simplifying it ?


texts: Attributing causality to single quasi-independent variable

texts:
Apologies if this isn't the correct place to ask - I'm not sure if this fits best with Stats or Data Science.

I'm using analytics to help marketers identify attributes of their users correspond to successful conversions (such as someone buying a product, signing up for a newsletter, or subscribing to a service). Attributes could be things like which site they came from (referrer), their location, time/day of week, device type, browser, etc.

What I'd like to say (although I'm not certain it's possible) is to isolate differences in conversion rate to an individual attribute, something like, '11% of your users from Facebook converted whereas only 3% of non-Facebook users converted', which would mean that the attribute 'referrer' and the level of the attribute 'Facebook' are responsible for driving conversions.

Given that I may have 100s of quasi-independent variables, is it even possible to isolate the effect to one variable and one level of that variable? As opposed to a combination of them that is more likely to be driving the difference? If so, what technique or conceptual paradigm do I use to identify which variable-level is responsible for the greatest lift in my dependent variable, conversion rate?


texts: How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python

texts:
Interested words are ['with', 'can', 'will']
gutenberg corpus is the corpus we would like to search on.
Expected output is
                           can  could    may  might should   will  would
        austen-emma.txt    270    825    213    322    366    559    815
  austen-persuasion.txt    100    444     87    166    185    162    351
       austen-sense.txt    206    568    169    215    228    354    507
          bible-kjv.txt    213    165   1024    475    768   3807    443
        blake-poems.txt     20      3      5      2      6      3      3
     bryant-stories.txt     75    154     18     23     38    144    110
burgess-busterbrown.txt     23     56      3     17     13     19     46
      carroll-alice.txt     57     73     11     28     27     24     70
    chesterton-ball.txt    131    117     90     69     75    198    139
   chesterton-brown.txt    126    170     47     71     56    111    132
chesterton-thursday.txt    117    148     56     71     54    109    116
  edgeworth-parents.txt    340    420    160    127    271    517    503
 melville-moby_dick.txt    220    215    230    183    181    379    421
    milton-paradise.txt    107     62    116     98     55    161     49
 shakespeare-caesar.txt     16     18     35     12     38    129     40
 shakespeare-hamlet.txt     33     26     56     28     52    131     60
shakespeare-macbeth.txt     21     15     30      5     41     62     42
     whitman-leaves.txt     88     49     85     26     42    261     85

Reference: https://www.nltk.org/book/ch02.html


texts: Adam optimizer for projected gradient descent

texts:
The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc.  The Adam optimizer is an improvement on gradient descent.

I have a situation where I want to use projected gradient descent (see also here).  Basically, instead of trying to minimize a function $f(x)$, I want to minimize $f(x)$ subject to the requirement that $x \ge 0$.  Projected gradient descent works by clipping the value of $x$ after each iteration of gradient descent: each negative entry is replaced with 0, after each step.

Unfortunately, projected gradient descent seems to interact poorly with the Adam optimizer.  I'm guessing that Adam's exponential moving average of the gradients gets messed up by the clipping.  And plain projected gradient descent has hyperparameters that can be tuned.

Is there a version of Adam that can be used with projected gradient descent?  I'm looking for a method that is an improvement on projected gradient descent, in the same way that Adam is an improvement on ordinary gradient descent (e.g., doesn't require hyperparameter tuning).  Is there any such algorithm?


texts: Structure of a multilayered LSTM neural network?

texts:
I implemented a LSTM neural network model in Keras. However, how the codes worked under the hood was not quite clear. I want to know if it worked the way I guessed how it worked?

For example:
Say there's a 2-layer LSTM network with 10 units in each layer.
The inputs are a sequence data $X_1, X_2, X_3, X_4, X_5$.

So when the inputs are entered into the network, $X_1$ will be thrown into the network first and be connected to every unit in the first layer. And it will generate 10 hidden states/10 memory cell values/10 outputs. Then the 10 hidden states, 10 memory cell values and $X_2$ will be connected to the 10 units again, and generate another 10 hidden states/10 memory cell values/10 outputs and so on?

After all 5 $X_i$'s are entered into the network, the 10 outputs from $X_5$ from the first layer are then used as the inputs for the second layer. The other outputs from $X_1$ to $X_4$ are not used. And the the 10 outputs will be entered into the second layer one by one again. So the first from the 10 will be connected to every unit in the second layer and generate 10 hidden states/10 memory cell values/10 outputs. The 10 memory cell values/10 hidden states and then the second value from the 10 will be connected and so forth?

After all these are done, only the final 10 outputs from the layer 2 will be used? Is this how the LSTM network works? Thanks!


texts: Precision, recall and accuracy metrics significantly different between training/validation and actual predictions

texts:
I have two sequential models built with Keras that train on data from a CSV file.
This is how they are built
# First model

model = Sequential()
model.add(Input(shape=X_1.shape))
model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(0.7))
model.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer=Adam(learning_rate=0.001),
              metrics=[recall, prec, ba])

# Second Model

model_2 = Sequential()
model_2.add(Input(shape=X_1.shape))
model_2.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))
model_2.add(Dropout(0.5))
model_2.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))
model_2.add(Dense(1, activation='sigmoid'))
model_2.compile(loss='binary_crossentropy',
              optimizer=Adam(learning_rate=0.001),
              metrics=[recall, prec, ba])

My dataset is split in half, one for training/validation and one for testing, also I have about 47k rows of data across 19 columns. Most of them are categorical, just one is numerical, all of the categorical data is one hot encoded and the numerical is normalized using MinMaxScaler,
When training the model I use the built-in metrics from Keras for Recall, Precision, Accuracy and I get decent numbers of above 0.7 for these. I calculate the F1 score of my results manually.
I then test the trained model on the resting data (the second half of the dataset) and calculate the metrics using Sklearn as I wasn't able to find some built-in functions for that in Keras. I use the below code.
# predict crisp classes for test set
yhat_classes = (model.predict(X_2) > 0.5).astype("int32")

# Calculate F1 score
accuracy = accuracy_score(y_2, yhat_classes)
precision = precision_score(y_2, yhat_classes)
recall = recall_score(y_2, yhat_classes)
f1 = f1_score(y_2, yhat_classes)

On these metrics, I get considerably lower numbers across all of them. Again my validation numbers from the .fit method of the model are exactly what I need. I also use EarlyStopping on my model, watching val_recall when training.
Why are my resulting numbers from the prediction so different than training and how can I improve them?
Is there a difference between how Sklearn and Keras calculate Precision and Recall?


texts: Tensorflow.js mapping to Tensorflow (python)

texts:
Despite having no understanding of what I'm doing, I decided to take a shot at porting rnn-nlu to tensorflow.js

A few lines in, and I'm really confused by the relationship between the python and javascript APIs.

Take the following code:

def create_cell():
  if not forward_only and dropout_keep_prob < 1.0:
    single_cell = lambda: BasicLSTMCell(self.cell_size)
    cell = MultiRNNCell([single_cell() for _ in range(self.num_layers)])
    cell = DropoutWrapper(cell,
                            input_keep_prob=dropout_keep_prob,
                            output_keep_prob=dropout_keep_prob)
  else:
    single_cell = lambda: BasicLSTMCell(self.cell_size)
    cell = MultiRNNCell([single_cell() for _ in range(self.num_layers)])
  return cell

For BasicLSTMCell there seem to be two very similar functions in the tensorflow-js:

tf.basicLSTMCell - identical in name, but needs forgetBias, lstmKernel and lstmBias which are all absent from the python call.
tf.layers.lstmCell - I can create one of these in a similar fashion as per the python API: tf.layers.lstmCell({units: this.cell_size}

The first option seems like a more likely candidate, but there are 3 extra parameters. If I could get past that, I could call tf.multiRNNCell(cells) - but that also requires data, c and h which are not required in the python API (which optionally takes state and scope).

If I go with the second option, it seems as though I would replace MultiRNNCell with let cell =tf.layers.stackedRNNCells({cells}) - but the difference in function names is not encouraging.

The only thing in the js API that looks remotely similar to DropoutWrapper is tf.layers.dropout() but it returns a Layer instead of a Cell and the parameters do not match at all.

I have also attempted to import a model which was learned by Python into JavaScript, but I could not figure out how to do that either.


texts: Predict datetime variable with orange canvas

texts:
Since Orange is able to read datetime variables, i tried to build a simple workflow to predict datetime variables via regression. Loading the data etc. is no problem, but when it comes to predict the data, the output of Predictions Widget (in this case with AdaBoost Regression) is something like:

1450396800.000;
1438905600.000;
1438732800.000;

whereas the original datetime is:

2015-12-22;
2015-08-07;
2015-08-05;

What does that mean? Did i make any mistakes? Is there any other way to predict datetime variables in orange?

The workflow does its job for any "common" continuous variable.


texts: How to handle sparsely coded features in a dataframe

texts:
I have a dataset that contains information regarding diabetes patients, like so:
    id   diabetes   diet   insulin   lifestyle
    0    No         NaN    NaN       NaN
    1    Yes        Yes    Yes       NaN
    2    No         NaN    NaN       NaN
    3    Yes        NaN    NaN       NaN
    4    Yes        Yes    NaN       Yes
    5    Yes        Yes    Yes       Yes

Features diet, insulin and lifestyle have a high percentage of missing data (around 95% each). So initially, I excluded these features from my dataset. However, after taking a closer look at the data, I found the values for diet, insulin and lifestyle to be associated with the value for diabetes feature. This makes sense as diabetes patients would be recommended treatment relating to diet, insulin intake and lifestyle changes.

So in cases where diabetes='No', values for features diet, insulin and lifestyle are missing.

And in cases where diabetes='Yes', I have found that in most cases, at least one feature from diet, insulin and lifestyle to have a value of 'Yes', and the remaining values are missing.

After some reading, I believe features diet, insulin and lifestyle are Missing at Random (MAR), and clearly not missing completely at Random (MCAR) as is explained here.
Anyway, so my question is, should the nature of the missing data here change my decision to remove these features from the dataset due to their high percentage of missing values. Or, should I impute the data for these features, by filling in missing values with "No", like so:
    imputer = SimpleImputer(strategy='constant', fill_value='No')
    x[:, 2:5] = imputer.fit_transform(x[:, 2:5])


texts: Decision Tree Optimize Deviation From Objective

texts:
I have the following problem: I have three classes/modes, let's call them car, bike, and walking. For any given test data instance with some environmental variables such as distance, road quality etc, I would like to predict the cheapest mode. In all instances in the training and test data set, each of the modes is associated with a cost. I would like to use a decision tree due to it being easy-to-understand. Is it possible to change the "performance metric" during training such that the final tree minimizes the cost deviation due to wrongful classification rather than maximizing the accuracy?

I know that the goal of minimizing the cost deviation can be achieved by employing three (linear) regression models, and then choosing the cheapest mode, but I would like to keep the "easy-to-understand" property of decision trees. Also, some of the explaining variables are non-linear (for example, walking is preferred, if it is neither too hot nor too cold).

Is this possible in general?


texts:
I have CNN architecture for object detection ( one object in image ) in KERAS.
It has 22 Convolutional layers ( layer includes max pool , LeakyRelu and Batchnorm )

The last layes are following

model.add(Flatten())
model.add(Dense(6))

Basicly i want to output vector that reresents ( probability that is is ObjECT , probability that it isnt , x , y , xmax, ymax )

I am feeding it data that has one object in it and labels

( 1, 0, xmin , ymin , xmax , ymax )

This has around 0.60% accuracy. However the sum of (isOBject + isntObject ) is greater than 1.

[[   1.02664185    0.34075582  231.69602966  241.59547424  280.6262207
   266.35855103]]

Hos is this possible.

Also when i feed network another data that does NOT have object in it and labl them

( 0, 1 , 0, 0, 0, 0 ).

I get 10% accuracy. How can it be? It has more data to learn from and added false data on top of that.

I appreciate all tips and tricks / explanation, basicly all help. Thanks