texts: How to Identify Repeating Data Entries when the Repeated Entries are Spelled or Constructed Differently

texts:
I have a dataset of entries and a variable for the owner of the entry. Some of these people occur more than once. However, the names are sometimes written differently. I want to eventually be able to aggregate the other data to the single owner. These are the names of business owners so sometimes it's a singular name, sometimes it's more than one name, and sometimes it's just the company name. Here's an example of some of the styles of names in the data:

DOE JOHN
DOE JOHN J
DOE, JOHN
DOE, JOHN + JANE
DOE JOHN + JANE
JOHN DOE J ETAL % JOHN J DOE
COMPANY CO

I've never done anything like this before. How could I go about identifying some of the same people? Is there a way to create an index to identify the similarity between these groups? Most of the ones I've seen are for longer text. Is there an index well suited for this?
I apologize if this is too basic a question. I'm new to doing things like this and I'm not sure if I know exactly what to search for. I'm most comfortable with Stata and R but I've used Python before and I could eventually figure out how to do something with that.


texts: How to train an image dataset in TensorFlow?

texts: As I am new to TensorFlow, I would like to do image recognition in TensorFlow using Python. For this Image Recognition I would like to train my own image dataset and test that dataset. Please answer me how to train a dataset and how to select the dataset..


texts:
I'm in the process of preparing to teach
an introductory course on data science using the R programming language.
My audience is undergraduate students majoring in business subjects.
A typical business undergrad does not have any computer programming experience,
but has taken a few classes which use Excel.

Personally, I am very comfortable with R (or other programming languages)
because I majored in computer science.
However, I have the feeling that many of my students
will feel wary of learning a programming language
because it may seem difficult to them.

I do have some familiarity with Excel,
and it is my belief that while Excel can be useful for simple data science,
it is necessary for students to learn
a serious programming language for data science (e.g., R or Python).
How do I convince myself and the students
that Excel is insufficient
for a serious business student studying data science,
and that it is necessary for them to learn some programming?

Edited in response to comment

Here are some of the topics that I will be covering:

Data processing and data cleaning
How to manipulate a data table,
e.g., select a subset of rows (filter),
add new variables (mutate),
sort rows by columns
SQL joins using the dplyr package
How to draw plots (scatter plots, bar plots, histograms etc.)
using the ggplot2 package
How to estimate and interpret statistical models such as
linear regression, logistic regression,
classification trees, and k-nearest neighbors

Because I don't know Excel very well,
I don't know whether all of these tasks can be done easily in Excel.


texts: Misclassification Rate for Random Forest Plateauing too Early

texts:
Using R, I have created 5 different random forest models using 5 different numbers of trees (3,10,30,100,300). My intention was to compute the misclassification rates of each of these models and plot the rates against the number of trees to illustrate the idea that generally, an increase in trees in a random forest model correlates with a decreasing misclassification rate.

I had a few colleagues run this same model in Python and with all of them, their model reached a misclassification rate of ~0.08 with the 300-tree model. However, When I run my models in R, the misclassification rate seems to level out around ~0.2 at the 100-tree model, and does not get any lower with the ~300 tree model. I'm curious as to what may be causing this discrepancy. I've provided my code below.

madelon_train <- data.frame(madelon_train_data, madelon_train_labels)
for(i in c(3,10,30,100,300)){
    assign(paste("madelonforest", i, sep = ""),
    randomForest(as.factor(madelon_train$V1.1) ~ ., data = madelon_train, ntree =
    i, mtry = sqrt(500), replace = FALSE))
}

modellist <- vector(mode="list", length=5)
for(i in c(3,10,30,100,300)){
    modellist[[i]] <- eval(as.name(paste("madelonforest", i, sep = "")))
}

#Use models to predict training data and compute misclassification error

classerrlisttrain <- vector(mode="list", length=5)
for(i in c(3,10,30,100,300)){
    err <-table(as.numeric(as.character(predict(modellist[[i]],
    madelon_train_data, type = 'class', OOB = TRUE))) - madelon_train_labels)
    classerrlisttrain[[i]] <- assign(paste("misclassification", i, sep = ""),
err[names(err)==0])
}

for(i in c(3,10,30,100,300)){
    classerrlisttrain[[i]] = as.double(classerrlisttrain[[i]])
    classerrlisttrain[[i]] = 1 -
classerrlisttrain[[i]]/length(madelon_train_labels$V1)
}

#Use models to predict test data and compute misclassification error

classerrlisttest <- vector(mode="list", length=5)
for(i in c(3,10,30,100,300)){
    err <-table(as.numeric(as.character(predict(modellist[[i]],
    madelon_valid_data, type = 'class'))) - madelon_valid_labels)
    classerrlisttest[[i]] <- assign(paste("misclassification", i, sep = ""),
err[names(err)==0])
}

for(i in c(3,10,30,100,300)){
    classerrlisttest[[i]] = as.double(classerrlisttest[[i]])
    classerrlisttest[[i]] = 1 -
classerrlisttest[[i]]/length(madelon_valid_labels$V1)
}

#Plot misclassification errors vs Tree Depth

plot(c(3,10,30,100,300), classerrlisttrain[c(3,10,30,100,300)], type = 'l',
xlab = 'Number of Trees', ylab = 'Misclassification Rate', xlim = c(1,300),
ylim = c(0,0.5), col = "red")
lines(c(3,10,30,100,300), classerrlisttest[c(3,10,30,100,300)], type = 'l',
col = "blue")
legend(1,0.1,legend = c("Train Data", "Test Data"), col =
c("red","blue"),lty=1, cex=0.8)


texts: I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?

texts:
This might make you feel like I am looking for a recommender engine, but I am not. A recommender engine works well if accuracy isn't an issue, but in my case, it is.

What I have proposed is to associate tags with certain question's answers, and map those tags to a particular video(s) and discern based on the mapping, whether the person should see it. Personalization. What do you think?

Here's the topic at hand to put this into context. Each video deals with a particular workout and is to be served depending on the answers in the onboarding screen. It then forms part of a program.


texts: Image mapping using a deep learning network while images have additional input parameter

texts:
I'm using a UNet to perform image mapping using medical images. Basically, I have a training set that contains original images and target images. However, the images have another parameter: angle. This parameter is related to the artifact in my training set. Basically, when the angle of this image is large, the artifact is strong. This angle parameter is already known.

My problem is I do not know how to add this parameter in the network. If I only use images with strong artifact to train and apply the net to strong artifact image, the result is good. But, I want to train all these images together since I have to cope with all these images with different angles.

How do I solve this problem?


texts: How to implement a GridSearchCV custom scorer that is dependent on a training feature?

texts:
I would like to code a custom scoring function using the make_scorer function, where my custom_function(y_true, y_pred)calculates the DAILY sumproduct of y_true and y_pred and outputs, say the mean, for example. The problem is that the timestamps are made available in my X matrix as a feature and I cannot access index of current folder.
Does anyone have a clue how to make this possible?
Thanks


texts: I have a time series graph that is segmented into a few parts based on the maintenance day. You can think of it like vertical lines appearing out of the x axis which symbolize maintenance at the date. I would like to find out the relationship between the segmentation to predict what would happen to the signal after a maintenance day. Is there any way to tackle that problem? How should i solve it?


texts: Different values from GridSearch estimation

texts:
I am using GridSearchCV in order to find best estimator with best hiperparameters. I also want to check it once against a small portion of data used in cross validation again.

Here's my code:

print 'Load data'
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target)

tree_param_grid = {
    'criterion': ['gini', 'entropy'],
    'splitter': ['best', 'random'],
    'max_depth': np.arange(2, 8),
}

tree = GridSearchCV(DecisionTreeClassifier(), tree_param_grid)

tree.fit(X_train, y_train)
tree_preds = tree.predict(X_test)
tree_performance = accuracy_score(y_test, tree_preds)

print '\nDecision tree:'
print 'Best params: ', tree.best_params_
print 'Best scores: ', tree.best_score_
print 'My score: ', tree_performance

best = tree.best_estimator_
best.fit(X_train, y_train)
best_preds = best.predict(X_test)
best_performance = accuracy_score(y_test, best_preds)

print 'Best score', best_performance

What I do not understand is this: best_performance and tree_performance have two different values. Why is that?

I thought that when I perform fit method on an object returned from GridSearch, I will automatically perform fitting on best estimator found in Grid Search. It turns out, it's not true - I have to manually get the best estimator from Grid Search. So, where does fit method in first usage come from? What is returned from GridSearchCV if not the best estimator itself?

EDIT:
I added a result from printing these two objects into console:

tree GridSearchCV(cv=None, error_score='raise',
       estimator=DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, presort=False, random_state=None,
            splitter='best'),
       fit_params=None, iid=True, n_jobs=1,
       param_grid={'splitter': ['best', 'random'], 'criterion': ['gini', 'entropy'], 'max_depth': array([2, 3, 4, 5, 6, 7])},
       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',
       scoring=None, verbose=0)
best DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=3,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, presort=False, random_state=None,
            splitter='best')


texts: K Nearest Neighbour with different distance matrix to each datapoint

texts:
I'm wondering if there is library support in python (such as sklearn) for doing KNN on a data set that has a custom distance matrix (positive definite) for each data point (x is a query point, $x_i$ is a data set point):
$$
d(x,x_i) = (x-x_i)^TQ_i(x-x_i)
$$

I know that for a fixed positive definite matrix for all data points, this is a metric that I can transform into
$$
Q = A^TA
\ \ \ \ \ \ \
d(x,x_i) = (Ax - Ax_i)^T(Ax - Ax_i)
$$
Which I can compute via normal KNN by first transforming the input space via multiplying $A$.

My problem of having a separate matrix for each data point came up because I have a covariance around the neighbourhood of each point. KNN can then be interpreted as what are the most likely neighbourhoods this query point lies in. If a neighbourhood doesn't vary along a dimension then we should penalize difference along that dimension highly in terms of increasing distance.


texts: Two different pytorch networks, combined loss, back propagation and optimizer step

texts:
So here is my network architecture as shown in below image.
I’ve two separate networks and loss_1 and loss_2 are coming from two separate nn.Modules networks and final_loss = loss_1 + loss_2.

Still only one final_loss.backward() would calculate grad in both networks?
I do have two separate optimizers to update grads in two separate networks respectively.

If not so, then what is the correct way to do back propagation in these individual networks on combined loss value.


texts: Does this data work with Convolutional Neural Network

texts:
I have a data set of 900k+ row with each row consisting of a pattern (e.g. 1421143) and a result (flawed, good).

Can I use the patterns in a CNN model to predict the defects occurrence ?


texts: How to find patterns in a series of timestamps

texts:
I have a series of timestamps that represent the time a user clicked a certain button.

My goal is to detect the automated clicks, so I need to find recurring patterns in the data that may point to an automated script. The majority of the data is by regular users. I don't need to detect them online, I just need to detect them in historical data. I also have user_id to make a distinction between users.

Is there a default approach to detecting recurring patterns in a largely unstructured dataset?

Most things I've found are about anomaly detection on a certain value paired with a timestamp but my problem is not really anomaly-detection as I need to find patterns amongst unpatterned data


texts: MSE loss in VAE reduces only KL divergence

texts:
I am trying to reproduce the two stage VAE from this repository in TF2 with Keras to learn MNIST digits. Unfortunately I am experiencing behavior that I can not explain to myself:
As far as I understand (correct me if I'm wrong) the objective in the VAE can be expressed as:
$$
E_{x\sim D} \left[ E_{z \sim Q} \left[ \log p(x|z) + \log p(z) - \log q(z|x) \right]\right]
$$
where $\log p(z) + \log q(z|x)$ is the KL divergence and $\log p(x|z)$ is the reconstruction loss with $z$ sampled using the reparameterization trick from a normal distribution.
There are two ways to calculate the reconstruction loss:

crossentropy as done in the Keras and TF2 tutorials.
MSE with $\log p(x|z) = -\frac{\log(2\pi)}{2} -\log \sigma - \frac{MSE}{2\sigma^2}$

One important aspect of the original code is that the $\sigma$ in $\log p(x|z)$ should be learned.
Now in my implementation when using the 2. form of calculating the reconstruction loss (with $\sigma=1$) only the KL divergence decreases.  Not only does the KL divergence decrease but it also becomes vanishingly small (I have already tried a $\beta$ weight scheduling).
When $\sigma$ is learnable it decreases and so does the reconstruction error (together with the KL-div).
When I use crossentropy for the reconstruction loss the behavior is as expected (the KL increases and the reconstruction loss decreases)
Questions

Is my formulation of the loss function wrong?
How can the KL-divergence be minimized when the reconstruction loss is multiple times higher?

Images
Losses (for crossentropy and MSE) from the original implementation (expected):

Losses for my implementation with crossentropy (expected):

Losses for my implementation with MSE as described above (problem):


texts: For what condition boosting work better than bagging in Ensemble Learner?

texts:
In boosting process we get a better accuracy for training data, but there is a lot of chance to over fit. For bagging ensemble method over fitting chance is lower than boosting.
Why we use boosting and for what condition ??


texts: What kind of Neural Network should I build to classify each instance of a time series sequence?

texts:
Let's say, I have the time-series dataset below-left. I would like to train a model in such a way that if I feed the model with an input like below-right, it should be able to classify each sample with the correct class label.
    Training Sequence:              Test Sequence:

Time,   Bitrate,   Class           Time,   Bitrate    Predicted Class
 0,       312,       1              0,       234      -----> 0
 0.3,     319,       1              0.2,     261      -----> 0
 0.5,     227,       0              0.4,     277      -----> 0
 0.6,     229,       0              0.7,     301      -----> 1
 0.7,     219,       0              0.8,     305      -----> 1
 0.8,     341,       1              0.9,     343      -----> 1
 0.9,     281,       0              1.0,     299      -----> 0
          ...                                ...

Any help is appreciated. Thanks!


texts:
Suppose I implement a supervised learning version of LSTM similar to this. Namely, I have these univariate time series data:

t Y
1 101
2 105
3 108
4 104
5 110
6 112
7 119
8 111
9 113
10 115

So Y at time 1 is 101, Y at time 2 is 105, etc.

The goal is to predict Y at time t+1 given Y at time t and t-1. That is, I can rearrange this data set to look like supervised learning:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110
104 110 112
110 112 119
112 119 111
119 111 113
111 113 115

Notice here that each Y value from t=3 to t=10 (max) gets predicted only once.

Now say I partitioned my data set into overlapping minibatches of size 5. We'll consider only the first two minibatches for simplicity:

t Y
1 101
2 105
3 108
4 104
5 110

2 105
3 108
4 104
5 110
6 112

I do the same trick of converting this to a supervised learning problem for each minibatch:

Y_(t-1) Y_t Y_(t+1)
101 105 108
105 108 104
108 104 110

Y_(t-1) Y_t Y_(t+1)
105 108 104
108 104 110
104 110 112

Notice: now two of the predictions from the first minibatch are repeated in the second minibatch.

I know that if I had wanted to have only one prediction per Y value, then I should have converted the entire dataset to supervised first and then created minibatches.

Questions:

Is this alternative approach incorrect? If so, why? If not, should I average the predictions, or is there another accepted way of combining the predictions from several time steps like this?

EDIT: I should mention that the learned parameters of the model are different for each position, even though the data points are the same. For example, the parameters that would be used to predict Y_(t+1) = 104 in minibatch 1 are different from the parameters that would be used to predict Y_(t+1) = 104 in minibatch 2.

EDIT 2: I'm thinking this approach isn't incorrect, just perhaps a little odd. It would be similar, in a way, to a bidirectional LSTM (reading the input backward and forward to increase the exposure of the network to the end points - i.e. boost the signal). This also boosts the signal, but perhaps increases the bias by quite a bit. Thoughts?


texts: First steps with Python and scikit-learn

texts:
I believe I have a simple if not trivial question. I have a background in statistics and I tend to use Stata and R quite a bit. I am interested in learning Python. I used it for a while now and recently came into contact with scikit-learn.

I am trying to reproduce a simple example.

from sklearn import tree
features = [[140, 1],[130, 1], [150, 0], [170, 0]]
labels = [0 , 0 , 1 , 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(features, labels)
print clf.predict([[150, 0]])

As you can see, the tiny script tries to predict  - by the means of a decision tree - wether a object with the properties [150, 0] is likely a type 1 or 0.

I run the script and get following error:

File "clf_decision_tree.py", line 6
    print clf.predict([[150, 0]])
            ^
SyntaxError: invalid syntax

I realy don't get what is wrong... Can you help me out? Best /R

PS: I not sure if Cross Validated or Stackoverflow are better places to ask. Let me know. Thanks.


texts: Understanding arcface, sphereface, and their differences

texts:
I'm a beginner in ml and I want to make a facial recognition system. While going through the research paper I realized that I'm losing the intuitive sense of the computations. I'm not from a statistical background but till now I can understand logits, cosine similarity, softmax functions.
Can someone please help me to get the core understanding of the computations that are taking place?
How/Why they utilized the weights of the last layer and input feature vectors.
How L2 normalization of the vectors help? Why the margin loss has to be in the range 0 to pi?
In general why softmax is paired with cross entropy loss?
I also read that arcface is much better than any other method including sphereface.
Can someone please formulate the difference?


texts: Please enlighten me with Platt's SMO algorithm (for SVM)

texts:
From A_Roadmap_to_SVM_SMO.pdf, pg 12.

(source: postimg.org)
Assume I am using linear kernel, how will I be able to get both the first and second inner product?
My guess, inner product of datapoint with datapoint j labelled class A for the first inner product of the equation and inner product of datapoint j with datapoints labelled class B for second inner product?


texts: What is behind "A. Grothendieck scheme theory" in Mondobrain?

texts:
Mondobrain proposes a "big data" technology with:

  a new generation of algorithms based on A. Grothendieck scheme theory
  (Field Medal) that extract knowledge and rules from data without any
  model or distance, and that can explore every part of
  multi-dimensional spaces independently.

What is behind this method that does not use "model of distance"? Are there relations methods like Topological Data Analysis (evoked at math stackexchange)?


texts: Differences in scoring from PMML model on different platforms

texts:
I've built a toy Random Forest model in R (using the German Credit dataset from the caret package), exported it in PMML 4.0 and deployed onto Hadoop, using the Cascading Pattern library.

I've run into an issue where Cascading Pattern scores the same data differently (in a binary classification problem) than the same model in R. Out of 200 observations, 2 are scored differently.

Why is this? Could it be due to a difference in the implementation of Random Forests?


texts: Modelling probability of switching of label from 0 to 1 in next time period for multivariate time series data

texts:
I have a data set consisting of unbalanced panel data, i.e. longitudinal multivariate measurements on multiple individuals. I want to estimate a probability that the individual will become 1 in next few time steps, if it was 0 by now, by modelling the history and checking for similarities with other individuals which became 1 in previous periods. How to approach this?
I was thinking about joint modelling of survival and longitudinal data, but I am not sure whether survival is OK for this, by the observation period only 5% of individuals become 1. I also have highly correlated variables, but they are correlated only when I check them as if each row was one observation, but it is not, as these are from various individuals. I also need it to be interpretable.


texts: Variational Autoencoder: Negative log likelihood not optimized

texts: I am using the auto encoding variational Bayes algorithm for one unsupervised object detection task. In the loss function, the reconstruction loss is calculated as the log likelihood of the original image pixel points by assuming the generated pixel points as the mean of one Gaussian distribution. In other words, if the decoder network is generating an image Y and the main image is X, then each pixel of Y is taken as a mean of a Gaussian distribution and the log of the probability density function for that corresponding pixel value of the original image is taken as the reconstruction error. I am using Adam optimizer. All the KL divergences associated with the distributions are decreasing but the negative log likelihood is not getting optimized. Can someone point out where I am going wrong? This is the line where I am using this log likelihood.


texts: What are common data augmentation techniques for nlp in general and for chatbot use specifically (in rasa)?

texts:
I am trying to build a chatbot using RASA
The process to build the chatbot can summarized in two step:

First identify the intent and the entities in the user's question
Second, based on a history of intents and their corresponding actions, augment the history of the data.
Use the augmented dataset to train a model (which can be an ML model) in order to use it to identify the actions from the intents.

My problem is to understand how the data augmentation is done in this case. So far and after reading the code for a whole afternoon, I concluded that one of the first steps in the process is the generation of a graph which represents the dataset.


texts: How to obtain the optimum model using Keras? Deep Learning

texts:
There are too many parameters while building an artificial neural network. Some of which that comes to my mind are:

Number of layers
Types of layers
Number of nodes in each level of layer
Activation functions in each layer
Ordering of the layers
Different types of optimizers
Different types of loss functions
Batch size while fitting
Epochs while fitting
A lot of other things I never heard of (I suppose)

The lone fact that the number of layers might be in the order of tens and the number of nodes might be in the order of thousands implies that we can end up with billions of different combinations. Considering all the parameters in the list above, how can one know that their model is close to the optimum ?
The cases I study on does not require me to be too precise in terms of r2_score. In other words, it would not make a huge difference if the r2_score is 0.85 or 0.87. But the question is how do I know if I am real close to the saturation point of the r2_score for the given data, i.e. the optimum model would yield an r2_score of 0.90 and I am not stuck at 0.80?


texts: How do I find entropy of features having numerical data?

texts:
I'm a newbie and I'm writing a decision tree from scratch using entropy and information gain.
I understand that entropy is the measure of impurity of a data set and also calculated entropy for categorical features but for continuous data, I do get an intuition that I have to take a range of values for each class label but how do I choose that range? Is my intuition correct?
What is the standard way of finding the entropy of a feature with numerical data.


texts: Data Scientist Consulting Interview Guide

texts:
Does anyone have any books or blogs that specifically sheds light on questions to ask your organization (from a consulting POV) as a data scientist? I am a new data scientist, which I have a background in consulting and predictive analytics.

There are several good reads such as "The Data Scientist's Field Guide" by Booz Allen, "The McKinsey Mind" from McKinsey, or maybe "Business Analytics for Managers" from SAS. I have come across interview guides to understand and execute strategy, but none that specifically structure questions for a preliminary assessment of analytic capability.

Any recommendations would be appreciated.

Thank you.


texts: Machine learning - Algorithm suggestion for my problem using NLP

texts:
I am looking for a machine learning algorithm for my problem.

I have a set of sentences like,

["The cat in the hat disabled", "A cat is a fine pet ponies.", "Dogs and cats make good pets.","I haven't got a hat."]

and the search-words like,

["cat","hat"]

I want to convert my sentence list and search-words to a vector space and find matching score between my sentence list and search-word list.

the type of output I am expecting is,

[("The cat in the hat disabled",0.9), ("A cat is a fine pet ponies.",0.5), "(Dogs and cats make good pets.",0.6),("I haven't got a hat.",0.49)]

Please suggest a machine learning algorithm for my task, if possible please share a reference link.

let me know if you have any queries or need more information.
I am currently using semanticpy for this https://github.com/josephwilk/semanticpy

I am getting key-error for many search-words. its not performing stemming and lemmatization for the sentence list but only performing for the search-words list.

Please help on this.


texts: What is the main difference between Hadoop and Spark?

texts:
I recently read the following about Hadoop vs. Spark:
Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop hours or days. Memory is much faster than disk access, and any modern data platform should be optimized to take advantage of that speed. Also, columnar data storage greatly reduces the amount of memory spent on empty or redundant data.
Can someone explain: 1) what Apache Hadoop and Spark are, 2) how they differ, and 3) how this relates to memory vs. disk access.


texts: What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare with precision/recall/F measure of  different algorithms ? Is this the best way for evaluation ? Or is there any other way to compare results of various recommendation algorithms ?


texts: What are towers in inception architecture and tensorflow?

texts:
My understanding of towers in inception architecture and in tensorflow terminology is that they are part of a neural network model for which separate computation can happen on forward phase and gradient computation phase of back-propagation, independently. My present understanding is based on another datascience.stackoverflow post What is a tower?, tensorflow documentation at https://www.tensorflow.org/tutorials/deep_cnn and blog post on inception architecture https://pseudoprofound.wordpress.com/2016/08/28/notes-on-the-tensorflow-implementation-of-inception-v3/.

However, when I read Xception paper (https://arxiv.org/abs/1610.02357) in introduction section "The Inception Hypothesis", it talks about average pooling tower. Does this mean it is referring to branches within a inception module as towers, even though independent backpropagation computation will not be performed on these branches?


texts: BLEU_SCORE gives bad scores - what am I doing wrong?

texts:
I want to calculate the BLEU_SCORE but it gives me bad results I don't know why?
For example, this is my reference and predicted sentences:
ref :
['electron', 'and', 'a', 'proton']
predicted :
['electron', 'and', 'a', 'proton']
ref :
['to', 'reach', 'the', 'nectar', 'at', 'the', 'bottom', 'of', 'flowers']
predicted :
['to', 'reach', 'the', 'nectar', 'at', 'the', 'bottom', 'of', 'flowers']
ref :
['during', 'the', 'summer', 'near', 'the', 'north', 'pole']
predicted :
['during', 'the', 'summer', 'near', 'the', 'north', 'pole']
ref :
['only', 'blue', 'light', 'is', 'reflected', 'by', 'the', 'block']
predicted :
['only', 'blue', 'light', 'is', 'reflected', 'by', 'the', 'block']
ref :
['between', '20', 'and', '40', 'degrees', 'latitude']
predicted :
['between', '20', 'and', '40', 'degrees', 'latitude']
ref :
['external', 'and', 'internal', 'combustion', 'engines']
predicted :
['external', 'and', 'internal', 'combustion', 'engines']
ref :
['cleaning', 'disinfecting', 'and', 'in', 'swimming', 'pools']
predicted :
['cleaning', 'disinfecting', 'and', 'in', 'swimming', 'pools']
ref :
['body', 'mass', 'index', 'bmi']
predicted :
['body', 'mass', 'index', 'bmi']
ref :
['they', 'put', 'nutrients', 'into', 'the', 'soil', 'that', 'plants', 'use', 'to', 'grow']
predicted :
['they', 'put', 'nutrients', 'into', 'the', 'soil', 'that', 'plants', 'use', 'to', 'grow']
ref :
['structure', 'of', 'earth', 'interior']
predicted :
['structure', 'of', 'earth', 'interior']

And here is the code I used to calculate the BLEU_SCORE :
from nltk.translate.bleu_score import corpus_bleu

print("Individual n-gram")
print("Individual 1-gram")
print('BLEU-1: %f' % corpus_bleu(ref, pre, weights=(1.0, 0, 0, 0)))
print("Individual 2-gram")
print('BLEU-2: %f' % corpus_bleu(ref, pre, weights=(0, 1.0, 0, 0)))
print("Individual 3-gram")
print('BLEU-3: %f' % corpus_bleu(ref, pre, weights=(0, 0, 1.0, 0)))
print("Individual 4-gram")
print('BLEU-4: %f' % corpus_bleu(ref, pre, weights=(0, 0, 0, 1.0)))

OUTPUT:
Individual n-gram
Individual 1-gram
BLEU-1: 0.015625
Individual 2-gram
BLEU-2: 0.000000
Individual 3-gram
BLEU-3: 0.000000
Individual 4-gram
BLEU-4: 0.000000

Anyone can help with this I don't know why it doesn't give me good results?


texts: Handling encoding of a dataset which has more than total 2000 columns

texts:
Whenever we have a dataset to be pre processed, before feeding it to the model we convert the categorical values to numerical values for which we generally use LabelEncoding, One Hot encoding etc techniques but all these are done manually going through each column.
But what if are dataset is huge in terms of columns(eg : 2000 columns), here it wont be possible to go through each column manually, in such cases how do we handle encoding?
Are there any specific libraries available which deal with automatic encoding of variable? I know of category_encoders which provides with different encoding techniques but how do we do it at in the above mentioned condition.


texts: What is the meaning of, or explanation for, having multiple tags in a Doc2Vec model's TaggedDocuments?

texts:
I've tried reading the other answers on this topic but I'm unsure if I understand completely.
For my dataset, I have a series of tagged documents, "good" or "bad." Each document belongs to an entity, and each entity has a different number of documents.
Eventually, I'd like to create a classifier to detect whether or not an entity's document is good or bad and to also see what sentences are most similar to the good/bad tag.
All that being said, does it make sense to label my data as following:
train_corpus = []
i = 0
for entity in entities:
    for doc_name in entity:
        for sentence in get_doc(doc_name):
           train_corpus.append(TaggedDocument(sentence, tags = [i, doc_name, entity, doc_name.good_or_bad])
           i+=1

From what I understand, this means that each entity is contextualized by all TaggedDocuments that have that entity's name, whereas each document is contextualized by each sentence that composes it. And the overall good/bad idea is composed of all the sentences that make up either the good or bad documents. Is this a correct interpretation? And if that's the case, could I then do something like:
unlabeled_data = [...]
model.infer_tag(unlabeled_data[0])
# return predicted good/bag tag
model.cosine_distance(unlabeled_data[0], "bad")
#get a numerical measure of how far some unlabeled data is from the "bad" tag


texts:
I want to assign a score to all points in a group that I cluster several time.
I want the score to indicate how much this point is grouped with the same individuals all time.
I suppose this idea allready exists, however I didn't found anything, only scoring on global clusters, as mutual information score.
I had an idea, for a point x, to count each point y that is in the same cluster of x in two clusterisations, or in different clusters in the two cases.
Do you know if this is a good idea ? If there is another scoring function that can do that ?


texts: Making Use of the Target Values for Regression

texts:
Problem:
I have a regression problem and I decided to useg Gradient Boosting Regression Trees to solve it. After all the preprocessing, I end up having around 130 attributes, 70K rows, and my cross-validated R-squared is 0.62.
Work So Far:
To increase my R-squared (and tighten my prediction intervals) I tried using a hint-feature that divides the target into linear groups (I manually decided for 4 groups and placed the target into 4 bins, - why 4? not too small not to big and not cleverly decided.).
I am aware that this is not a usual way to go, but this is doable for my application since eventually, I will be collecting these hints from the user. (I will be asking for the target value for that user and suggest him/her another target value depending on his/her data.)
So it would be possible for me to place each user into such groups since it seems like not all of the users behave the same and they need a separated modeling for the ones that are extremely high/low target values, plus the distribution is extremely right-skewed.
After using this cheating-hint-feature, my cross-validated R-squared became 0.81. However, obviously, dividing the target into linear groups like that, brings the disadvantage of having a step-like model that has a clear separation between those groups while this is not necessarily always the case (a user that gives his target value most probably but not necessarily should belong to that group), the boundary values for such binning should not be manually decided, and this model would definitely need to be smoothed.
Next Steps?
After seeing that such an approach can be helpful (and I could really make use of the user-given values), I am now thinking about smarter ways to separate between the users regarding these hints. I am thinking about applying hierarchical clustering and using the cluster information as a hint-feature but not sure if this is the best way to proceed.
Would you maybe have suggestions for me for such method (and papers would be great) or statistical approach to figure out such boundary values (other than density-based methods)?


texts: How to choose an optimal threshold for binary discretization

texts:
We know that we usually do discretizations to continuous features to remove extra information and unwanted regularities, which makes the model robust and well-predicted.
But I am wondering except based on the context of feature how to choose an optimal threshold for data discretization, binary or multi.


texts: I'm wondering in what situations it is advantageous to stack LSTMs?


texts: Logistics regression with polynomial features vs neural networks for classification

texts:
I am taking Andrew Ng's Coursera class on machine learning. He mentions that training a logistic regression model with polynomial features would be very expensive for certain tasks compared to training a neural network. Why is that though? I mean, when we talk about neural networks we're usually looking at a model with a very large number of parameters so why would logistic regression be more computationally expensive?

PS: Here's some context (at the beginning of an exercise on neural networks):

  In the previous part of this exercise, you implemented multi-class logistic regression to recognize handwritten digits. However, logistic regression cannot form more complex hypotheses as it is only a linear classifier (You could add more features, such as polynomial features, to logistic regression, but that can be very expensive to train).


texts:
I'm working with a dataset $X$ (of length $N$) of count data, which looks like:

I developed a statistical model which can be improved, so I'm asking for any suggestions, for instance, differnet likelihoods or prior selection, different approach, anything...
My model
I'm trying to get the parameters of the likelihood of the data, so thaht I can get a posterior predictive density function, credible intervals and so on. Let's say, I want to model the generative process of the data given some parameters, $f(X|\theta)$
This data shows a large overdispersion ($\bar X << var(X)$), thus a Poisson likelihood, $f(X|\lambda) \sim \mathcal{Poisson}(\lambda)$, is not a good choice.
Reading literature about count data with overdisperssion, I decided to model $f(X|\lambda)$ as a Negative Binomial distribution, thus $f(X|\lambda) \sim \mathcal NB(r, p)$
Parameter estimation
In order to not to end up with a very complex set-up, I've performed bayesian estiamtion of the hyperameter $p$, letting $r$ be computed from the data: in a Neagative Binomial distribution, $r$ is related to the first and second moments of the distribution following:
$
r = \frac{\mu^2}{\sigma^2 - \mu}, \text then
$
$
\hat r = \frac{\bar X^2}{var(X) - \bar X}
$
The whole set-up is:

Likelihood: $f(X|p) = \mathcal NB(\hat r, p)$
Prior: $f(p) = \mathcal Beta (0, 0)$ (non informative, improper prior)
Posterior: $f(p|X) = \mathcal Beta (0 + \hat rN, 0 + \sum X)$

which returned the following posterior predictive distribution:

The first and second moments of the predictive posterior distribution are very close to those in the data (I've let the data have a huge impact in the posteriors since I've choosen a non-informative prior). Also, the point estimate posterior predictive (using $\mu_p$) does not differ from an averaged predictive posterior distribution over all possible values of $p$.
Once again, any suggestions for improvement?
EDIT
What about a zero-truncated negative binomial distribution?


texts: Appropriate loss function for multi-hot output vectors

texts:
I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four values that are equal to 1:

[0 0 0 1 0 0 1 0 1 0 0]

These kinds of vectors are sometimes called "multi-hot embeddings".

I am looking for an appropriate loss function for outputs of this kind. Is there a published equation I should check out? Or should I implement a custom loss function? Any suggestions others can offer on this question would be greatly appreciated!


texts: Normalizing historical data in time-series LSTMs

texts:
I am currently trying to solve a sequence prediction problem using LSTMs in a keras architecture. To illustrate the problem I give the following example which resemble the problem I must solve.

Lets assume I am using climate data to model a very specific temperature phenomenon that only happens in a given week during the spring. I am using the past few days as well as historical data to predict the temperature each day for the next week. My network takes two inputs and one output.

Input 1 is the daily mean temperature & precipitation for the past 5 days (sequence length=5).
Input 2 are the weekly mean temperature & precipitation from the past 6 months (sequence length=24).
Output is the temperature tomorrow.

Furthermore, lets assume that my data comes from several cities around the world with very different climates such that normalizing the data across batches is very troublesome.

Let the batch size be 7 such that each batch predicts the temperature for the next 7 days for the given city.

I know that to get a model with robust prediction power I should use the changes over time instead of the values (e.g. "time n" to "time n+1" to predict the change from "time n+1" to "time n+2") but that leaves me with the following issues:

How should I normalize the data given that the values from Input 2 are likely to be VERY different from those in Input 1 for each batch (since they cover different seasons)?

I think a batch normalization layer right after each input layer would work. However, I am not sure if that would mean that the network would "disassociate" the two inputs so-to-speak. Using batch normalization, I fear that the model would no longer recognize that the sequence in Input 2 leads up to the sequence in Input 1.

Is this a real consequence of batch normalization or does the NN handle it automatically?
Can a control variable, like a constant that represents the difference between the means of each input, solve this?

Remember that the scenario is fake, it is simply meant as an illustration of this approach. To be clear the question is:

Is batch normalization after each input layer (and maybe subsequent layers) safe or should I normalize each batch before the input and not use batch normalization layers?


texts:
I'm looking for a place to find benchmarks against which to evaluate performance on public datasets.

In this instance, I'm interested in results on the MovieLens10M dataset.  It seems to be referenced fairly frequently in literature, often using RMSE, but I have had trouble determining what might be considered state-of-the-art.

I did find this site, but it is only for the 100K dataset and is far from inclusive:

http://www.recsyswiki.com/wiki/MovieLens_100k_benchmark_results


texts: Is it possible to compress a sequence of numbers through an autoencoder?

texts:
Specifically: I would like compress a set of coordinates, which map to the locations of 1's in a binary image, and then decode back to the original set. For instance, for a 16x16 image, the input might be something like the following:
[5, 4], [12, 5], [8, 7],....
I am not looking to recognize any spatial patterns, nor is this a time series problem because the input corresponds to just one static image. The trained autoencoder should be able to handle any array of arbitrary "coordinates", under the assumption that the data is scaled between 0 to 1 so the resolution (actual range of numbers) is inconsequential.  Is this doable? What would be a good way to train it?


texts: Can we select features by examining scatterplots?

texts:
Suppose, I have a data set with eight features in my hand. I want to find features to predict the diamonds, hearts, clubs, spades.

------------------------------------------------------------------------------
|   f1   |   f2   |   f3    |   f4    |    f5    |   f6    |   f7    |  f8
+--------+--------+---------+---------+---------------------------------------
|        |        |         |         |          |         |         |

f1 column is for class-labels and the rest of them are features.

First, I have rendered scatter plots by taking the features f2 and f3 and it looks like the following,

Later, I have rendered scatter plots by taking the features f3 and f4 and it looks like the following,

If $Red$ = Diamonds, $Blue$ = Spades, $Magenta$ = Hearts, and $Black$ = Clubs,

I have the following questions,

(1) Why are some plots cohesive and others are separated?

(2) What do these two plots tell about those four types of cards?

(3) Which two features, among f2, f3, and f4, would you choose for further experiments and why?


texts: Why does my model fail to predict on the whole dataset?

texts:
So I have about 3000 images with 6 classes and this is what I did:
1 - split into training set and test set prior to anything with 20% test size
2 - performed data augmentation on the under represented classes in the training set and ended up with 2700 training and 640 test
3 - did feature extraction techniques (haralick, dominant color, avg color, hist, etc) on both sets
4 - did normalization of features using standard scaler (fit_transform on training and after just transform on test)
5 - did a gridsearch with 5 fold cv to find best params just in the training set and got 91% accuracy average
6 - used the best estimator to predict on the test set and got 94% accuracy
7 - pickled the model and scaler and then uploaded on a new file
8 - create a predict function with all the transformations and then feed it a random image from the data set, in theory this is not new data so it should give the same results yet it fails miserably every time
what am I doing wrong?
I don't think its overfitting otherwise my test accuracy would fail
I presume it's something to do with the scaler?


texts:
I'm new to data mining and have been going through constraint-based query mining lately. I came across the concept of succinctness which basically details a constraint as succinct, if we can generate all the candidate item-sets precisely, based on an itemset satisfying the constraint.

A more formal definition is :

  Given A1, the set of items satisfying a succinctness constraint C, then any set S satisfying C is based on A1 , i.e., S contains a subset belonging to A1

Example, min(S.Price) <= v  is succinct
But, sum(S.Price) >= v  is not succinct

I understand why the former is a succinct constraint => as all the candidates can be generated by ensuring that one of the subsets satisfies that constraint.
But I fail to understand why the latter is not a succinct constraint. Any pointers on this would be helpful !


texts: How to drop row where all columns BUT the timestamp are zeros in python

texts: I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros


texts: How to do Feature Scaling for these ranges [0,1] and [-1,1]?

texts: I want to rescale the features of my data to be between [0,1] and [-1,1]? Is their a clear cut way that works every time for these ranges? I think the below equation works for [0,1] but when it is describe people say generally it works, so I am not certain it works every single time.


texts: Bigdata cluster compatible distributed predictive model

texts:
I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The reason I am confused is
1)The scikit learn python codes don't scale in big data.
2)The spark mlib is written in scala and not in python.
3)Although h2o is open sourced I don't think it is written in python.


texts: How to create an image with pixel values of another image?

texts:
I have the following situation:

I have an image file with each pixel having a specific value. Now I want to rearrange and distribute these pixel values so that the result image looks similar to a set of other images that share a similar pixel value distribution.

I wanted to know how I should tackle this challenge. I know basics in machine learning and would "just" use the training images as a csv file and create a model from it to apply to my other image. Any tips you could provide me?


texts: How does decision tree for regression makes a prediction?

texts: For classification, it is obvious how a decision tree is used to make a prediction.You just have to find the final leaf. However for regression problems, how can you find the prediction considering the continous aspect of the variable to predict?


texts: Rules/Methods to use for determining user groups with no review data

texts:
I’m working on an analytical problem where I have customers who deal with end consumers. End consumers can leave customers reviews based on their experience (Star ratings, text etc). I’m trying to segregate new/old customers into different quality categories: good, average, bad, for an online advertising platform. I’m looking for insights on rules/methods on how to do this

How to set a quantifiable threshold for calling a customer good, average, bad for business? There is consumer review data available (Imagine a table with customer, and the star rating they received, and/or textual reviews from consumer)
How to classify a new customer on the platform with no or minimal review data available about them particularly, while there’s also some customers who have many reviews available?


texts:
I am toying around with a clustering and churn prediction framework, cluschurn which they deployed in production at Snap, Inc. In their research paper, paper_link, they use 14 days of user data and treat it as a time series. They do some transformations and they have a have a 3-dimensional dataset which is 12x14xm users; each user, m, has 12 daily features for a total of 14 days. This set gets fed into an LSTM so that each LSTM timestep has receives a 12-dimensional vector. Hopefully, I worded this right so it's understandable. They were able to improve their results by adding a embedding layer between the input and the LSTM layer.

  We connect a fully connected feedforward neural network to the
  original daily activity vactors, which converts users' sparse activity
  features of each day into distributional activity embeddings...

I'm completely confused on what this looks like dimension-wise and am confused if there is a fully connected network before each LSTM timestep or is there one fully connected network before the feeding the LSTM? For example, is the 3-dimensional data set flattened and then fed into a fully connected network and then reshaped back into a 3-dimension dataset that has 14 timesteps?

I made these concepts to help clarify what I mean.


texts: What tools are available for programming language parsing for ML?

texts:
I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and constructing structures out of a document of Python (for example) code, similar in concept to NLP tokenization and higher level language processing, adjusted for programming languages.

While I'm able to find a lot of general NLP related material, it seems to me a better starting point would be previous research specifically focusing on programming languages.

I'm looking for tools, resources, academia articles and keywords to search for, basically any help is appreciated!


texts: How to extract entities from text using existing ontologies?

texts:
I am working on a entity extraction task and I am using Stanford CoreNLP NER. Here, I want to detect entities of type "Animal", "Building", "Imagery", etc., which are not covered in Stanford CoreNLP entity extraction method. Also, I want to detect organization entities which are not included in CoreNLP entity model. I have the data about the organizations with me.

How can I do this?


texts: What is the most efficient method for hyperparameter optimization in scikit-learn?

texts:
An overview of the hyperparameter optimization process in scikit-learn is here.

Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid search is slow.

Random search is faster than grid search but has unnecessarily high variance.

There are also additional strategies in other packages, including scikit-optimize, auto-sklearn, and scikit-hyperband.

What is the most efficient (finding reasonably performant parameters quickly) method for hyperparameter optimization in scikit-learn?

Ideally, I would like working code examples with benchmarks.


texts: Add ID information from one dataframe to every row in another dataframe without a common key

texts:
I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two.

The first dataframe (df1) contains the identifing information, and the other dataframe (df2) contains data about that identifying information.  They are in different dataframes since they come from different files.  I am sure there is an easy way to do this, but cant seem to find it.  The basic problem is there are no unique identifiers on df2 yet, and I need them from df1.

For example, I get df1 from a file containing the identifing information (this dataframe will always only have 1 row, some chance that the column names change as well):

df1:

              UID    Name       ID2
0           21YX4    Alan  21458200

while df2 comes from another file, but contains no base identifying information, like this (this dataframe can have very many rows and columns, and columns vary by file).  df2's keys in this case are c and d (the index is reset), but that is not the problem, the real ID is in df1:

df2
Out[1184]:
           a        b       c   d    e
0   1.427962       NaN  shark   0  0.0
1  -0.977258  7.000000  shark   1  NaN
2  -0.301779       NaN  python  0  NaN
3   0.393833       NaN  panda   1  1.0
4   0.494462       7.0  panda   2  0.0
5  -0.526740  5.000000  shark   2  NaN

Desired result

      UID    Name       ID2           a         b       c   d    e
0   21YX4    Alan  21458200    1.427962        NaN  shark   0  0.0
1   21YX4    Alan  21458200   -0.977258   7.000000  shark   0  NaN
2   21YX4    Alan  21458200   -0.301779        NaN  python  1  NaN
3   21YX4    Alan  21458200    0.393833        NaN  panda   1  1.0
4   21YX4    Alan  21458200    0.494462        7.0  panda   2  0.0
5   21YX4    Alan  21458200   -0.526740   5.000000  shark   2  NaN

I have tried concat, join as well as looping through the keys, but no luck so far.
A join like:

df2=df2.join(df, how='outer')

puts df1 into df2, but as a new row with no data, and no ID values for the data rows.  Is reshape or repeat necessary somehow or is there an easy way to do this?

Any suggestions?


texts: where does kmeans store its trained model parameters in scikit learn?

texts: Like in linear regression, there is model.intercept_ and model.coef_ but in kmeans i found  kmeans.inertia_ , I dont know what is that.


texts: Is there a way or tools available to generate or retrieve cypher query from a Neo4j database ? Should we need to store cypher quries along with the graph data for regeneration ?


texts: How to properly represent a tic tac toe board to a CNN?

texts:
I'm figuring out how to manipulate convolutional neural networks (CNN) in python and I want to apply this kind of NN to an agent player that plays tic tac toe. I know that's weird and the problem doesn't need some elaborate and complex solutions like CNN, but I'm using this environment just to learn better and compare with the training of a MLP that I implemented.

So, I'm trying to think about how to represent the board correctly as an input to the CNN. Unlike the chess board which we can think the input as 8x8x6 (8x8 as the 2d array representing the board and 6 channels which one representing the different pieces of the game), the tic tac toe is a bit more complicated for me because there are just one kind of piece. Is it possible and correct to represent the X and O as different pieces? Has anyone implemented something like this?


texts: When to One-Hot encode categorical data when following Crisp-DM

texts:
I have a dataset that contains 15 categorical features (2 and 3 level factors which are non-ordinal) and 3 continuous numeric features. Seeing as most machine learning algorithms require numerical data as input features, and actually automatically One-Hot encodes them on the fly (random forest, glmnet etc.), should you not perform One-Hot encoding during data pre-processing to allow exploration of the relationship of the encoded feature data? Or is it best to rather explore relationships between raw categorical data and then only encode before running algorithms?
Basically my question evolves around data exploration and data understanding, and whether this needs to be performed on the raw or encoded categorical features?


texts:
I have a labeled image dataset, where the images are in subfolders and there is one Pascal XML per image with the labels.

I would like to compute stats like: how many images have exactly two labels?

Or - what is the average size of the labeling rectangle?

Ideally also statistics on image resolution, file size etc, but mostly labels.

This is probably an easy question (many papers include that info), but did not see that function in labelImg and Sloth. How can I do that?


texts: Evaluating metrics F1, F2, Mean Average Precision for object detection

texts:
Up today in the company where I work we are using the F1 Score for evaluating the performance of our model, also our competitor's using the same metric.

I would like to understand what's the difference between F1,F2 and mAP?(Please do not explain me how to calculate them, also I know that F measure gives the same weight to the precision and recall while mAP choose the best precision from all recalls)
Why in competitions (e.g. PASCAL VOC) and articles for object detection I am reading it is always preferred to use mAP instead of F1 or F2 scores ?

Thanks !


texts: Doing a fine tuning after a transfer learning

texts: I read about fine tuning and transfer learning for CNNs and was wondering if we can do fine tuning after using transfer learning on the same CNN? If so, will this increase the performance of the model or decrease it?


texts: Data set size versus data dimension, is there a rule of thumb?

texts:
I am trying to collect some data for ML, specifically for training a neural network model, and I don't know how big the data set is enough.

So is there a rule of thumb on how many data of dimension DIM should one collect for training a NN-Model ? For example, it depends on the number of features or the kind of NN-Model or something else ?

Any help will be appreciated.


texts: Error of linear regression as a result of the coefficients

texts:
I'm trying to resolve a paradox. Suppose that I have a bunch of data points $\{x_i,y_i\}$ and calculate a slope and intercept, $m$ and $b$ such that $y=mx+b$. Also, I can calculate the error in $m$, $\sigma_m$ and the error in $b$, $\sigma_b$. Suppose further that the true model is a line $y=ax+c +\epsilon$ where $\epsilon$ is an error term that is normally distributed and has a standard deviation of $\sigma$. If I were to calculate the error in my calculated value as a result of the errors in my coefficients I would get $error^2 = \sigma_m^2 x^2 +\sigma_b^2$. This error increases with increasing $x$, a very strange property given that the standard deviation of y at any given point x is constant, i.e. $\sigma$. What am I missing? What is the resolution of this paradox?

Update: The standard error of the slope $\sigma_m$ is given by $\frac{\sigma^2}{\sum_i (x_i -\bar{x})^2}$ so $\sigma_m$ goes down inversely with the standard deviation of the X values. I thought that perhaps this resolved the paradox. However, x,and $\bar{x}$ are not constrained by the standard deviations. All of the x values can be within two units of each other and yet x may be a thousand, million, billion or more.


texts: Calculating possible number of configuration

texts: I am wondering how did they get the $19200$ possible configurations? Like, $5^6 = 15625$, where $6$ is the number of hyper-parameters:


texts: Filtering Pandas Dataframe Results without saving to a variable first

texts:
Is there a way to more efficiently filter a result on a data frame without having to explicitly save it in a variable and then filter? For instance, in the code below I would like to add something to line 3 to be able to achieve either df1 or df2.

import pandas as pd
df=pd.DataFrame({'A' : [1,2,3,4,5], 'B' : [0,0,0,0,0]})
df=df.var()
df1 = df[df!=0]
df2 = df[df > 3]


texts: How to improve the accuracy of my MLP (Current benchmark 77%)

texts:
My MLP code:

# In[1]:

import pandas as pd
from keras.models import Sequential #Sequential Models
from keras.layers import Dense #Dense Fully Connected Layer Type
from keras.optimizers import SGD #Stochastic Gradient Descent Optimizer
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint

# In[2]:

data = pd.read_csv("doom.csv", delimiter=",")
data = data.drop('id', 1)
data = data.drop('cid', 1)
data = data.drop('alerts', 1)
data = data.drop('hsalerts', 1)
data = data.drop('capacity', 1)
data = data.drop('duplicate', 1)
data.shape

# In[3]:

data.selectedoption.value_counts()

# In[4]:

feature = data.drop('selectedoption',axis=1)
label = data[['selectedoption']]

# In[5]:

from sklearn.model_selection import train_test_split
train_feature, test_feature, train_label, test_label= train_test_split(
 feature, label, test_size=0.3, random_state=42)

# In[ ]:

# In[ ]:

# In[6]:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

scaler.fit(train_feature)
train_feature_trans = scaler.transform(train_feature)
test_feature_trans = scaler.transform(test_feature)

# In[7]:

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

import matplotlib.pyplot as plt
def show_train_history(train_history,train,validation):
    plt.plot(train_history.history[train])
    plt.plot(train_history.history[validation])
    plt.title('Train History')
    plt.ylabel(train)
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='best')
    plt.show()

# In[11]:

from keras import optimizers

model = Sequential()

model.add(Dense(units=32,
                input_dim=10,
                kernel_initializer='uniform',
                activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(units=32,
                kernel_initializer='uniform',
                activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(units=16,
                kernel_initializer='uniform',
                activation='softmax'))
model.add(Dropout(0.5))

model.add(Dense(units=1,
                kernel_initializer='uniform',
                activation='sigmoid'))

print(model.summary())

adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

model.compile(loss='binary_crossentropy',
              optimizer='adam', metrics=['accuracy'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)
train_history = model.fit(x=train_feature_trans, y=train_label,
                          validation_split=0.8, epochs=20000,
                          batch_size=1000, verbose=2,callbacks=[es, mc])

show_train_history(train_history,'acc','val_acc')
show_train_history(train_history,'loss','val_loss')

# In[9]:

scores = model.evaluate(test_feature_trans, test_label)
print('\n')
print('accuracy=',scores[1])

My dataset is this:

doom.csv

I made a few changes like, changing the optimizer, changing the learning rate, changing activation functions, changing the units. Nothing seems to work. The accuracy isn't going above 80%. Is there something that i can do to improve the accuracy over 90%?  I am a newbie to this. Thank you.


texts: xgboost in R have different results compared to boosted decision tree in Azure ML

texts:
I have a small data set (4000 records with 10 features) and I used XGBOOST in R as well as Boosted Decision Tree model in Azure ML studio. Unfortunately the results are different. I like to optimize recall and I could pick that as a measure in Azure but I can not do so in R.

I used the same parameters in both platforms. I know seeds might be different but I tried many of them. I always have a much better recall on my validation dataset using the Azure model compared to the R one.

I wonder if there is a big difference behind the methodology used in these two platforms causing me the issues.

I also used cross validation which did not help. Any insight is appreciated.

Thanks


texts:
Karpathy's' LSTM batch network LSTM batch network operates with batches

def checkSequentialMatchesBatch():
    """ check LSTM I/O forward/backward interactions """
    n,b,d = (5, 3, 4) # sequence length, batch size, hidden size
    input_size = 10
    WLSTM = LSTM.init(input_size, d) # input size, hidden size
    X = np.random.randn(n,b,input_size)
    #...

def checkBatchGradient():
    """ check that the batch gradient is correct """
    # lets gradient check this beast
    n,b,d = (5, 3, 4) # sequence length, batch size, hidden size
    input_size = 10
    WLSTM = LSTM.init(input_size, d) # input size, hidden size
    X = np.random.randn(n,b,input_size)
    #...

What does batch applied for? I'm only familiar with feeding one-hot word representation vector and can't understand LSTM learning process with batches. Please explain in terms of text processing.

Thanks in advance.


texts: Can we access social media advertisements and information like how many likes, comments, shares they received using their APIs?

texts: I want to access social media advertisements and information about how popular they were ( how many likes, comments and shares they received and the comments). Can I get these information using these APIs ? Or else where can I get this data?


texts:
Being an IT engineer, I am exploring the Big Data specialization and wishing to earn a certificate in that field to help me get a new job later.

I've been looking around and found these two specializations:

Data Science Specialization from John Hopkins University
Big Data Specialization from UC San Diego

I was hoping someone with experience in the field to advise on which to pick as it will be a big investment in time (months) and effort.

Any other suggestion out of the list is welcome.

Thank you!

PS: Let me know if I'm posting in the wrong section as this is a really genuine question!


texts: Inference from text data without label or Target

texts:
I have a use case where I have text data entered by an approver while approving of some loan.
I have to make some inferences as to what could be the reasons for approval using NLP. How should I go about it?
It's a Non english language. Can Clustering of text help?? Is it possible to cluster TEXT OF non English language using python libraries.


texts:
Although I have seen a few good questions asked about data anonymization, I was wondering if there were answers to this more specific variant.

I am seeking a tool (or to design one) that will anonymize human names from a specific country: particularly first names in unstructured text. Many of the tools that I have seen have considered the wider dimensions of data anonymization; with an equal focus on dates of birth, addresses, etc.

An imperative aspect is that it needs to have near absolute recall. The major pitfalls, as far as I can see, are diminutive variants ("Tommy" instead of "Thomas", "Ben" instead of "Benjamin", etc.) and typos. These two factors prevent a simple regex based on a database of names (based on censuses, etc.)


texts: I've taken a tensorflow for beginner course and I'm wondering if Project Malmo can be use with Tensorflow, just a simple demonstration will do. Specifically, I'm trying to modify tutorial_6 of python-example to use tensorflow's tf_agent (https://www.tensorflow.org/agents/overview). This is the closest example I can find online explaining how to use tensorflow for reinforcement learning. In the course I've taken, my instructors only taught supervised learning using models like CNN,DNN and RNN, and hardly even mention RL. Any help or advice is appreciated. If there's an example code for me to get started with will be the best.


texts: using logistic:binary as objective in xgboost

texts:
i am using 'binary:logisitic' as my objective function in xgboost classifier. Therefore  when i predict probabilities i.e:
model.predict_proba(x_test)[:, 1]

will this be the true probabilities output or the 'score' i.e. the mean predicted value? When i plot the probability distribution function it does not look right so i am thinking perhaps i need to apply some calibration . i have read other forums but am unsure what to use.


texts:
are there any resources out there (book, blog, your own answer post etc.) that gives advise on modelling strategy of deep neural net?

I know how to fit a neural net, I know how to change settings like number of hidden layers, activation function, dropout etc. I know how to use cross-validation to validate models.

But what I need is advise on the actual modelling process. i.e. given a dataset (which has been cleaned and explored), where do you start? What type of neural net do you train first? How do you then tune it?


texts: Root Mean Squared Error (RMSE) - significance of square root

texts: What is the significance of the square root in root-mean-square-error? Essentially, my question is: what is the difference between (rms error) and (rms error)$^2$?


texts: Gender identification task on instance or user level?

texts:
I'm working on a task which is gender identification. Given a user account (e.g. Twitter account) with its documents (e.g. 100 tweets), the user should be classified as a male or a female.

The dataset that I have contains around 500 users for each class (label).

There are 2 different ways to approach this task, but I don't know which one is "more correct":

1) I aggregate the documents of each user into one large document, and then feed the final document into a classifier.

2) I project the user label (class) into her/his documents, and then feed each single document of the user into a classifier. At the prediction time, I apply averaging on the probabiltities of the users' documents to get the labels of the users (e.g. larger or smaller than 0.5).

Probably an answer to this question could be that both ways are task-dependent, but I want to know if there is a scientific explanation behind any of the solutions, especially in my task.

BTW, some of the documents for many users are not important (e.g. Hello all :D!).


texts: Scikit learn linear regression - learning rate and epoch adjustment

texts:
I am trying to learn linear regression using ordinary least squares and gradient descent from scratch.
I read the documentation for the Scikit learn function and I do not see a means to adjust the learning rate or the epoch with the sklearn.linear_model.LinearRegression class.
Is there a standard learning rate for the linear regression model?
Epochs I am assuming are determined by the change in the error function and are thus dependent on the dataset, with some predetermined maximum amount before convergence.
Are there any other good packages that have a linear regression model that I can test my own model against with adjustable learning rates and epochs?


texts: I want to implement a deep learning model in Keras, but I want to use my own loss function, i.e. custom loss. If I implement some loss function and use Keras Functional API for the model, do I need to change the way the optimizer works, because the optimizer will minimize my loss function? If I need to do that, what is the way to do that?


texts:
For example suppose I've data set which looks like:

[[x,y,z],
 [1,2,5],
 [2,3,8],
 [4,5,14]]

It's easy to find the theta parameters from those tiny data set. Which is
theta = [1,2,0]

z = 1*x + 2*y + 0

But if my data set are non linear. Suppose:

[[x,y,z],
 [1,2,6],
 [2,3,15]]]

If i choose the mapping function to be of:
z = xy+yy

It would return the theta parameter :

theta = [1,1,0]

So my deal is how to choose such mapping function for data sets which varies over time. As in recommender system  the user rating varies as per the time, to reduce the cost. I've recently gone through regularization. Is there any other ideas for reducing the cost.


texts: Display tested batch and its predicted label in Keras

texts: I am using Keras to create a deep learning model (CNN), how can I display all tested batch images and its predicted label?


texts: Semantic similarity between two or more sentences

texts:
I need to determine how similar sentences (in meaning) are to one another.
In order to do it, I have been considering an algorithm (cosine similarity) to determine the similarity between sentences. I have thought as appropriate Word2vec or wordnet to build features for similarity.
If you have used this (or similar) approach, could you please provide me an example of use of word2vec/wordnet for similarity analysis?


texts: How to make a popularity-based recommender system having data on posts and number of likes? Please review a code

texts:
I'm writing a popularity-based recommendation system, where I have data on posts and likes the posts have. I need to recommend posts to a user based on their popularity (obtained likes).
Packages and a recommender engine "recommender":

import pandas as pd
import numpy as np
import sklearn
from sklearn.model_selection import train_test_split
import sklearn.externals
import joblib
import recommender as recommender

Data on posts and likes:

df_posts

POST_ID LIKES
0   1   5
1   2   10
2   3   15
3   4   20
4   5   25
5   6   30
6   7   35
7   8   40

User data:

df_users

USER_UUID
0   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E
1   878B9C5F-5BB7-49F4-AA34-F1FD1F8CAD0E
2   678B9C5F-5BB7-49F4-AA34-F1FD1F8CAD0E
3   578B9C5F-5BB7-49F4-AA34-F1FD1F8CAD0E
4   478B9C5F-5BB7-49F4-AA34-F1FD1F8CAD0E

I tried to write the following script for a recommender engine called 'recommender':

#Class for PopularSity based Recommender System model
class popularity_recommender_py():
    def __init__(self):
        self.train_data = None
        self.USER_UUID= None
        self.POST_ID = None
        self.LIKES = None
        self.popularity_recommendations = None

    #Create the popularity based recommender system model
    def create(self, train_data, POST_ID, LIKES):
        self.train_data = train_data
        self.POST_ID = POST_ID
        self.LIKES = LIKES

        #Sort the posts based upon the number of likes
        train_data_sort = train_data.sort_values(['LIKES', self.POST_ID], ascending = [0,1])

        #Generate a recommendation rank based upon score
        train_data_sort['Rank'] = train_data_sort['LIKES'].rank(ascending=0, method='first')

        #Get the top 10 recommendations
        self.popularity_recommendations = train_data_sort.head(10)

    #Use the popularity based recommender system model to
    #make recommendations
    def recommend(self, USER_UUID):
        user_recommendations = self.popularity_recommendations

        #Add user_id column for which the recommendations are being generated
        user_recommendations['USER_UUID'] = USER_UUID

        #Bring user_id column to the front
        cols = user_recommendations.columns.tolist()
        cols = cols[-1:] + cols[:-1]
        user_recommendations = user_recommendations[cols]

        return user_recommendations

Training the model and giving a recommendation:

#Training data
train_data, test_data = train_test_split(df_posts, test_size = 0.20, random_state=0)

#Creating an instance based on popularity
pm = recommender.popularity_recommender_py()

pm.create(train_data, 'USER_UUID', 'POST_ID', 'LIKES')

#Selecting a user to give a recommendation
users_uuid = df_users.iloc[:,0]
USER_UUID = users_uuid[]
USER_UUID

#Giving recommendation
pm.recommend(USER_UUID)

I need to get the following output:

USER_UUID                               POST_ID LIKES
0   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    8   40
1   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    7   35
2   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    6   30
3   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    5   25
4   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    4   20
5   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    3   15
6   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    2   10
7   778B9C5F-5BB6-49F4-AA34-F1FD1F8CAD0E    1   5

Could anyone can help me with a NameError: name 'USER_UUID' is not defined after running pm.recommend(USER_UUID)?
Also, I would appreciate, if someone can help to check the entire code and help me to make the engine workable.


texts: How to extract particular key from the dict()?

texts:
I have a dict() with 1000 keys. First 4 entries of dict are like.

{
    'aaa': [1,0,6,8,0,5,9,1,1,0],
    'abc': [1,1,1,2,4,0,0,0,9,8],
    'cfg': [0,0,0,4,3,1,0,0,0,1],
    'cghjj': [7,8,9,2,3,0,0,0,0,0]
}

I want to create a dataset using each key one by one. I want to pick key1. Then create a dataset using function1 using the values of key1 and then pick key2. Creating another dataset using values of key2 and appending the result of key2 to key1 row-wise. Then with key3... and then append the result to the result of key1 and key2 and so on.. up to 1000 keys.

dataset=dict()

create_dataset:
    select values of key 1.
    b1=function1()
    b1=np.asarray(b1)

    then select key 2.
    b2=function1()
    b2=np.asarray(b2)

    np.append(b1,b2,axis=0)

...upto 1000 key.


texts: Pytorch: how to pass the hidden state between the samples in LSTM?

texts:
I am trying to boost the performance of a object detection task with sequential information, using ConvLSTM.
A typical ConvLSTM model takes a 5D tensor with shape (samples, time_steps, channels, rows, cols) as input.
as stated in this post, a long sequence of 500 images need to be split into smaller fragments in the Pytorch ConvLSTM layer. For example, it could be split into 10 fragements with each having 50 time steps.

I have two goals:

I want the network to remember the state across the 10 fragment sequences. I.e. how to pass the hidden state between the fragements?

I want to feed in the images (of the video) one by one. I.e. the long sequence of 500 images is split into 500 fragments with each one having only one image. The input should be like (all_samples, channels, rows, cols). This only make sense if the 1.goal could be achieved.

I found some good answers on Tensorflow. However I still need some help on implementing it on Pytorch.
https://stackoverflow.com/questions/38241410/tensorflow-remember-lstm-state-for-next-batch-stateful-lstm
https://stackoverflow.com/questions/49082088/the-best-way-to-pass-the-lstm-state-between-batches
(I am thinking of using a buffer to temporarily save the intermediate hidden states, don't know if it is a good idea.)


texts: Notation for features (general notation for continuous and discrete random variables)

texts:
I'm looking for the right notation for features from different types.
Let us say that my samples as $m$ features that can be modeled with $X_1,...,X_m$. The features Don't share the same distribution (i.e. some categorical, some numerical, etc.). Therefore, while $X_i$ might be a continuous random variable, $X_j$ could be a discrete random variable.

Now, given a data sample $x=(x_1,...,x_m)$, I want to talk about the probability, for example, $P(X_k=x_k)<c$. But $X_k$ might be a continuous variable (i.e. the height of a person). Therefore, $P(X_k=x_k)$ will always be zero. However, it can also be a discrete variable (i.e. categorical feature or number of kids).

I'm looking for a notation that is equivalent to $P(X_k=x_k)$ but can work for both continuous and discrete random variables.


texts: Finding groups of friends in social network data

texts:
I have some social network data in which I'd like to identify people who belong to a group of 5 friends.  These are users who are friends with 4 other people who all are also friends with each other.

In the graph sense, this task involves identifying components of the graph where each node has an edge between every other node in the component and where the component contains at least 5 nodes.

Is there a well known algorithm to do this?  I was looking at measures like centrality, clustering coefficients, and strongly connected components but none of those are quite what I want.

Thanks!


texts: How to use REST API to execute Map-Reduce Task?

texts:
Let's say I have one Text File (Size 1 GB). I want to search particular word from file and if it is found the Line number should be returned.

I can Execute my java program using command line in linux. But what I want is some Interface using REST API.

When I submit that word in GUI(My Interface) and submit request it should be able to call HDFS , commands should be provided by itself and it should return me the result.

How Can you approach this problem ?
(I have very primitive knowledge about REST API)

Referred

https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#ParameterDictionary[]

https://www.youtube.com/watch?v=7YcW25PHnAA


texts: How to Integrate/map Population Data with/over Sample Data

texts:
Problem Definition:
Our organization is conducting different type of surveys and Census in our country. The basic difference between Census and Survey is that, the target of Census is the Complete Population but the target of Survey is just a sample (sub-set) taken from that Population. Now the organization is willing to integrate the results of surveys and Census to map the data of surveys over the census data. Currently the organization is using some statistical approaches to integrate the data but these technique have loads of issues. I am interested to solve this issue by using some state of the art approaches from Data Science, Machine Learning or Deep Learning.
Consider population (census) data set contains 220 million rows, whereas sample (survey)  data contains 40 million rows approximately.
Now,  my question is for all the data scientists having sound background of statistics, how to do that?
Need a step by step guidance to achieve this goal. Kindly recommend me the algorithms to get this task done. Also suggest me some resources to read and understand this problem.
Example to Illustrate the Problem:
Let for Census the Population is P having attributes A,
Similarly, for Survey the Population (sample taken from P) is S having attributes  B
i.e. S⊆P and A⊆B
For Example
Census Data (Table 1)

Block Code
HouseHoldID
PersonID
Sex
Marital Status
Age

1
1
1
Male
Married
25

1
1
2
Female
Un-Married
30

1
1
3
Male
Married
22

1
2
4
Male
Married
40

1
2
5
Male
Un-Married
30

1
3
6
Male
Un-Married
17

2
4
7
Female
Married
50

3
5
8
Female
Married
52

3
5
9
Female
Married
45

4
6
10
Female
Un-Married
45

4
7
11
Female
Un-Married
42

5
8
12
Male
Married
36

5
9
13
Female
Married
33

Survey Data (Table 2)

Block Code
HouseHoldID
PersonID
Sex
Marital Status
Age
Employment Status
Education Level

1
1
1
Male
Married
25
Employed
Graduate

1
2
4
Male
Married
40
Employed
No Schooling

1
2
5
Male
Un-Married
30
Un-Employed
Primary

3
3
6
Male
Un-Married
17
Un-Employed
Middle

3
4
7
Female
Married
50
Employed
Middle

As per our policy we divide our country into Provinces, then provinces into districts, then districts into Tehsils and similarly following the pattern we reach to the lowest level known as BLOCKS. Each Block is composed of at-least 500 households (Families) and each family consist of members.
The above Table i.e. Table 1 showing Census data collected from 5 different blocks but in Table 2 the Survey data is collected from two different blocks. Here you can see that Survey data is subset of Census data. Similarly, there may be many blocks collected in Census may not be visited in Survey.
Furthermore, the attributes in Census data are subset of Attributes of Survey Data (Survey is short but detailed)
Now if I am supposed to map Survey data over Census data i.e. I want to see the percentage of un-employed members in Block with Code 2 (where as this block is not visited in Survey), then how would I do that?
We may need such type of mapping at block level or upper level.


texts: Best way to classify plots which are overlapping?

texts:
I have an experiment in which it was done under two conditions. For each condition, the experiment was performed 26 times. The output of the experiment is a plot with 70 time indices. I would like to train a classifier to predict, given a plot, to which condition it belongs. The image below shows the output of the conducted experiment under the two conditions recognised by different colors. The actual experiment begins at index 35, and thus it can be seen there is no difference in the outcome of the experiment before that regardless of the condition. The plots represent power spectral density of EEG from one channel (electrode).

I am trying to train an svm classifer ignoring the features below 35. The classifier is having hard time doing so considering the high variability of each condition. One thing is, averaging the red plots and blue plots yield a noticeably different behaviour, as can be seen from the second figure. I would like to improve the accuracy of my classifier, beyond 65%. Is LSTM suitable for this type of problem? Any other suggestions?


texts: What are main algorithmic differences between XGboost, lightGBM and Catboost?

texts: Also, what makes lightGBM run faster than XGBoost?


texts: Best way to deal with realistically imbalanced dataset for Regression problem

texts:
I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data.

NOTE: The labels are continuous between 0 and 1 but binned for visual purposes.

These objects are from a simulation which accurately predicts the population numbers which we see in reality. (Hence the "realistically imbalanced" title)

Unfortunately, an object with label = 0 exhibits random looking motion and therefore the number of ways in which you can construct these objects tends to infinity..whereas those with label = 1 are highly ordered objects with not many variations in their structure and so have a limited number of ways in which they can appear.

To me this means that, in order to explore all possible variations of these objects in their domain would require a dataset who's "number of each class" would be the complete opposite as we see in the figure above. i.e. you need to observe exponentially more label ~ 0 objects than label = 1 objects in order to explore the objects' domain.

Obviously this is impossible and impractical. I'm curious as to what people think is the best method of approaching this issue?

I have thought about:

Making this an object detection method whereby you discriminate between objects with label greater than or below a threshold such as 0.8. (I think this would be difficult given that the labels are continuous and so you would get muddying at the boundary).
Joining two CNNs: one which models features that describe the label such as velocity data, and one which models properties that somehow link the level of physical randomness in the images to the corresponding labels.

I have already tried upsampling with a CNN classifier but with so few label ~ 0 images, the CNN quickly overfits and defaults to predicting them with a higher label.


texts: Measuring uncertainty in an LSTM network using dropout in keras/tensorflow

texts:
I've created a simple LSTM network for testing

model = tf.keras.Sequential()
model.add(layers.LSTM(32, input_shape = (timesteps, data_dim), recurrent_dropout = 0.2))
model.add(layers.Dense(1))
model.compile(loss = 'mae', metrics = ['accuracy'], optimizer = tf.train.AdamOptimizer())

I am using dropout method as researched by Yarin Gal here and Lingxue Zhu here
and my function for dropout looks like this:

f = K.function([model.layers[0].input, K.learning_phase()],
               [model.layers[-1].output])

after training I'm using the function above in the "predict_with_dropout"

def predict_with_dropout(x, f=f, n_iter=100):
    result = np.zeros((n_iter,))
    #print(f([x,1]))
    for iter in range(n_iter):
        result[iter] = f([x, 1])[0]

    return result
results = []

for point in test_X:
    results+= [predict_with_dropout([point])]

results_avg = np.apply_along_axis(np.mean, 1, results)
variance = np.apply_along_axis(np.var, 1, results)

This code is working as expected and as I understand it the "predict_with_dropout" function is using the f-function to re-train the LSTM model 100 times and within those 100 times it is dropping out certain cells of the model.

Is this the correct implementation of the papers or am I missing something?
If it is correct - is there any way to speed this up?


texts:
I am reading "reinforcement learning - An introduction" by Sutton and Barto.

At pag. 59, there is the Bellman equation for the state-value function

$\begin{array}{ll}
v_{\pi}(s) &=
\mathbb{E}_{\pi}[G_t|S_t=s] \\&= \mathbb{E}_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s] \\&= \sum\limits_{a} \pi(a|s) \sum\limits_{s'} \sum\limits_{r} p(s^{'},r|s,a) \left[ r + \gamma \mathbb{E}_{\pi}[G_{t+1}|S_{t+1}=s'] \right]
\end{array}$

I didn't understand why the expected value survived in the last expression. The definition of the expected value is $\mathbb{E}[X] = \sum\limits x \cdot p(x)$, not $\mathbb{E}[X] = \sum\limits \mathbb{E}[x] \cdot p(x)$

I don't know whether my question is clear. In the last equation of the defition of $v_{\pi}(s)$, I would not have put the expected value inside


texts: Does traning the Word2Vec model multiple times affect `min_count` parameter?

texts:
In Word2Vec, If I train a set of sentences multiple times with change in order (as it increases the vector representations), will the frequency of a word get changed due to it.?

For example, if I have the word "deer" in my corpus 4 times and If I set the min_count to be 5, does training the model 3 times repeatedly count "deer" with frequency 12 and will be included in the model ?

If it knows it is the same corpus then how it is possible to differentiate, if I retrain the model with a new corpus.


texts: In a binary classification, should the test dataset be balanced?

texts:
I have a dataset with 4519 samples labeled as "1", and 18921 samples labeled as "0" in a binary classification exercise. I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class.

However, should the test dataset be balanced as well?

In other words, if train my model with 1000 random samples of "0" class, and 1000 random samples of "1" class, should I test the model with the remaining 3519 samples of "1" class, and randomly select another 3519 samples of the majority "0" class, or I can go with the remaining 17921?

What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Thanks


texts: Feature relevance in PCA + kmeans algorythm

texts: Working on the World Happiness Report dataset, i have N countries with M features and a happiness score. This is the parameter I built 3 classes from: happy, medium, unhappy (numerical intervals of happiness score). I have run a 3 component PCA (only the first two components show relevant separation) and kmeans on the result (with 6 clusters, chosen in function of distorion). Plotting the two components with clustering predicted colouring gives very clear separation between classes, and I should now deduce what feature makes "happy" countries happy. My idea would be to implement some form of Bayes' theorem, but don't really know how since there is strong (>0.6) correlation between some features. How can I approach the problem?


texts: How to handle and report patient characteristic statistic with missing data in essay?

texts:
I'm now working on an clinical trial essay with some doctors. When generating patient characteristic statistic form, I found basic patient characteristic data has some missing.

The missing datas are in features like "whether patient had hypertension" or "average blood pressure on exam". The missing rates are from 2% to 50%. We don't have a way to re-collect those data and they must be reported in essay.

Patient characteristic statistics are important evidence to prove efficience of experiment so we can't just remove statistic with missing data from the report table.

I know multiple ways to fill NAs in machine learning task and just report missing rates in ETL task. But how should I deal with missing data like that in an essay?

Thank you


texts:
I want to create a simple perceptron with three inputs.
$ \sigma(w_1x_1 + w_2x_2 + w_3x_3) $, where $ \sigma = \frac{1}{1 + e^{-x}.} $
$$ L(w) = \sum_i^3(y_i -\sigma(w_1x_1 + w_2x_2 + w_3x_3)); $$

$ \large \frac{dL}{dw} =2(y - \sigma(w_1x_1 + w_2x_2 + w_3x_3))\sigma(1 - \sigma)x.   $

I am trying to program reverse propagate the error. Help, what am I doing wrong.

def sigmoid(x):
    return 1/(1 + np.exp(-x))

data_train = np.array([[1, 0, 1], [0, 1, 1], [0, 1, 0], [0, 0, 1]])
train_res = np.array([[1], [0], [0], [0]])

weights = 2*np.random.random((3,1)) - 1

for i in range(10000):
    out = sigmoid(np.dot(data_train,weights))
    dE = 2*np.dot(data_train,np.dot((out - train_res),np.dot(out,(1 - out))))
    weights += dE

test = np.array([0, 0, 1])
print(sigmoid(np.dot(test, weights)))


texts: Is it reasonable to use the output of the sigmoid function as the win rate prediction?

texts:
I'm working on a project which is predicting the win rate of one team or one person. (could be any kind of sports like baseball, basketball or e-sport games) The data I have is more like a classification problem. Data looks like this, A v.s B, the winner is A. So, I trained a classification model with those data. The model should input two units(a team or a person) and output a probability from sigmoid, so I should set a threshold to determine the output is 1 or 0. But I'm curious about if the output of the sigmoid can be considered as the win rate? I think the answer might be yes? But I got next question. My goal is to predict the win rate of one unit(team or person), which means I don't care about opponents. So When using this model to predict, for example, I want to predict Laker's win rate, and I make the match-up with all the others teams(29 teams) as the opponents and make the 29 pieces data like Laker vs Hawks, Laker vs Mavericks,...and so on, then I got the 29 output from sigmoid. Here are two choices I have.
1. setting the threshold for the output of sigmoid and determine the result is 1 or 0. Then
   calculate the win rate. For example, 3 output, 1 0 0 ---> win rate=0.33
2. just get the mean of all the 29 games' output from sigmoid(no threshold setting). For example,
   3 output 0.6, 0.3, 0.4 ---> win rate = 0.433

I think the first one is more reasonable, but the second one is outperforming the first one in my experiment. So, does the second one make any sense? For the second one, is it a regression problem? or still a classification problem? I
I'm not fluent in English. Please bear with me... I would be grateful for your response.


texts: Standard correlation coefficient of various datasets

texts:
I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for second and third line ?. Can someone help me to gain an understanding of below picture

  The second question is related to the correlation between attributes. The intention behind asking this question it to gain more understanding from StackOverflow data science users. The very one thing I learnt from the below picture is

if you plot attribute against the same attribute, you will end up with lines, which is of no use. For instance, if you notice median_house_value in y-axis against median_house_value in the x-axis, they are basically histograms.

Is there any valuable feedback that you can comment on the below picture ?. If so, please let me know.


texts:
I've been reading the sklearn documentation on outlier detection, and even the examples provided by the documentation. Once I have fitted my dataset, all I can do is apply the predict function to the estimator in order to find outliers. However, I would like to get the probabilities that the point is an outlier. Can this be done in sklearn? Is there an R package to do it?

I don't even know if there is a theoretical foundation of the outlier detection methods used by sklearn that allows you to give probabilities. If not, what is the criterium that tells you what is an outlier and what is not? Does it consider probabilities or some kind of non-probabilistic scores?

Any piece of information will be appreciated.

Edit

I would like the outlier method to consider the multivariate distribution of the data. I think that univariate detection methods are rather poor.