texts: Interpreting random seed effect on correlation in regression with small sample

texts:
I am working on a project with a sample size of 30. I have 7 features predicting a continuous variable where I am aiming to optimize the r-value. If I change the random seed of my train_test_split() 100 times, I get correlation coefficients that range between .6 and .9 with an occasional negative r-value.

How should I interpret this? Is this variability normal for such a small data set and a random shuffling of my training data? Or is something else going on? How should I decided which seed to use?


texts: Divide a column by itself with mutate_at dplyr

texts:
Hi I'd like to turn each non zero value of my selected columns to a 1 using mutate_at()

 BRAND  MEDIA_TYPE    INV1       INV2
   <chr>  <chr>      <dbl>     <dbl>
  b1     newspapers    2         27
  b1     magazines     3         0
  b2     newspapers    0         0
  b3     tv            1        145
  b4     newspapers    4         40
  b5     newspapers    5         0
  b1     newspapers    1         0
  b2     newspapers    0         28

The final result should be like follow:

 BRAND  MEDIA_TYPE    INV1      INV2
   <chr>  <chr>      <dbl>     <dbl>
  b1     newspapers    1         1
  b1     magazines     1         0
  b2     newspapers    0         0
  b3     tv            1         1
  b4     newspapers    1         1
  b5     newspapers    1         0
  b1     newspapers    1         0
  b2     newspapers    0         1

Do you have any suggestion on how to solve it?
Thank you!!


texts:
I've implemented a shallow FC feedforward neural net with 2 input nodes, 1 hidden layer with 4 nodes (tanh activation) and 1 outputnode with sigmoid activation function and binary cross-entropy loss. I'm trying to classify sklearns make_moons() dataset, but when i train it the loss just keeps increasing and the accuracy decreasing. It's like its doing the opposite of learning. Here is the code:
import numpy as np
from sklearn import datasets
from matplotlib import pyplot as plt
from math import log

def sigmoid(x):
    return 1/(1+np.exp(-x))

def dsigmoid(x):
    y = sigmoid(x)
    return y*(1-y)

def tanh(x):
    return 2/(1+np.exp(-2*x)) - 1

def dtanh(x):
    return 1 - (tanh(x))**2

class ANN:
    def __init__(self, n):
        self.n = n
        self.w1 = np.random.normal(0, (2/6)**0.5, (4, 2))
        self.w2 = np.random.normal(0, (2/5)**0.5, (1, 4))
        self.b1 = np.zeros((4, 1))
        self.b2 = 0

    def feedforward(self, x):
        self.a = tanh(np.dot(self.w1, x.reshape(2, 1)) + self.b1)
        self.out = sigmoid(np.dot(self.w2, self.a) + self.b2)[0, 0]
        return int(round(self.out))

    def backprop(self, x, y):
        y = y[0, 0]
        z1 = np.dot(self.w1, x.T).reshape((4, 1))
        z2 = np.dot(self.w2, self.a)
        delta2 = np.multiply(self.dbi_crossent_loss(self.out, y), dsigmoid(z2))

        delta1 = np.multiply(np.dot(self.w2.T, delta2), dtanh(z1))
        self.dw2 = np.dot(delta2, self.a.T)
        self.dw1 = np.dot(delta1, x.reshape(1, 2))

        self.w2 -= self.n*self.dw2
        self.w1 -= self.n*self.dw1
        self.b1 -= self.n*delta1
        self.b2 -= self.n*delta2

        return self.bi_crossent_loss(self.out, y)

    def bi_crossent_loss(self, y_hat, y):
        return (-y*log(y_hat)) - (1-y)*log(1-y_hat)

    def dbi_crossent_loss(self, y_hat, y):
        return (-y/y_hat) + (1-y)/(1-y_hat)

X_train, Y_train = datasets.make_moons(n_samples=500, noise=0.10)

network = ANN(0.01)

c = 0
l = 0
for j in range(5000):
    for i in range(np.shape(X_train)[0]):
        y = Y_train[i].reshape(1, 1)
        x = X_train[i].reshape(1, 2)
        y_hat = network.feedforward(x)
        c += 1 if int(round(y_hat)) == y else 0
        l += network.backprop(x, y)
    if j % 100 == 99:
        print('step: ', j*100, 'loss: ', l/50000, 'acc: ', c/500)
        c = 0
        l = 0

X_test, Y_test = datasets.make_moons(n_samples=1000, noise=0.10)

preds = np.zeros(Y_test.shape)
for i in range(np.shape(X_test)[0]):
    preds[i] = network.feedforward(X_test[i].reshape(1, 2))

# Draw decision boundry
x_min, x_max = X_test[:, 0].min() - 1, X_test[:, 0].max() + 1
y_min, y_max = X_test[:, 1].min() - 1, X_test[:, 1].max() + 1
h = (x_max-x_min)/400
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

grid_preds = np.zeros(xx.shape, dtype=int)
for i in range(xx.shape[0]):
    for j in range(xx.shape[1]):
        grid_preds[i, j] = network.feedforward(
            np.array([xx[i, j], yy[i, j]]))

plt.contourf(xx, yy, grid_preds, cmap=plt.cm.Spectral, alpha=0.8)
plt.scatter(X_test[:, 0], X_test[:, 1], c=preds)
plt.show()

In the end I also plot the test data and the decision boundry. I previously tried using only sigmoid functions instead of tanh, and I've also tried using MSE. When doing this the accuracy usually didn't seem to decrease rapidly, but the network only modelled a linear decision boundry. Can someone please take a look at my code and see if you see a mistake? I would very much appreciate it, as I am in a great hurry.


texts: Encoding logic rules in machine learning

texts:
For some problems you have certain rules that you know for a fact are true. A very simplistic example would be -

"A grade below 40% is a failing grade"

There is no ambiguity in the statement and for that system it will always be correct. Encoding such rules within any machine learning model should allow it to avoid simplistic mistakes and theoretically improve accuracy.

I'm aware there are several niche libraries for this and an entire subfield as well. However how would I combine such logic rules within a typical model (CNN,LSTM ,etc) in a platform like TensorFlow, PyTorch ?

Edit : What I'm looking forward is to add explicit label rules (similar to what Snorkel [ https://github.com/snorkel-team/snorkel ] uses to generate noisy labels) to help the model learn. These relations while present in the data might be hard for the model to learn or require large amounts of training data/time. In some scenarios a combination of deterministic rules and ML would be better than using either alone.


texts: Policy Gradient custom loss function not working

texts:
I was experimenting with my policy gradient reinforcement learning algorithm, and I was wondering if I could use a similar method to the supervised cross-entropy. So, instead of using existing labels, I would generate a label for every step in the trajectory.

Depending on the value of the action, I would shift the stochastic policy (Neural Network) output to a more efficient output and train it as a label to a cross entropy loss function.

Example of an action:
Real output: [ 0.2, 0.8 ]; Value: [ -0.5 ]; Action taken: [ 1 ] (0.8 probability).
Created label: [ 0.3, 0.7 ] (Second action not that great, reduce it's probability by a little bit)

My method didn't work, and I am really curious to know why.


texts: Residual plot is tight to zero, but low R2 score

texts:
Hi I am a beginner at data science, and currently trying to use Gradient Boost Regressor to predict car price based on several attribute such as machine capacity, car brand, car type etc.

I am following this tutorial: https://github.com/PaacMaan/cars-price-predictor/blob/master/cars_price_predictor.ipynb

I got a low R2 score and plotted the residual vs predicted value, what i am confused with is even though my residual value is close to zero (as showed on the graph) my r2_score is low. Can anyone kindly explain this to me?

here is my r2 score

r2_score: 0.38

and the code i am using

car_df_encoded = pd.get_dummies(car_df_encoded, columns=['Store_Loc','Brand','Type','Transmission'], prefix = ['Store_Loc','Brand','Type','Transmission'])
price= car_df_encoded.pop('Price')

x_train, x_test, y_train, y_test = train_test_split(car_df_encoded, harga, test_size=0.25)

gbr = GradientBoostingRegressor(loss='huber', max_depth=5, max_features=None,min_samples_leaf=8, min_samples_split=6, n_estimators=150,random_state=42)
gbr.fit (x_train, y_train)
predicted = gbr.predict(x_test)
residual = y_test - predicted

Tried using cross validation as well

scores = cross_val_score(gbr, car_df_encoded, harga, cv=4, scoring='r2')
print('\nMean Score:')
print(scores.mean())

and my mean r2 score is

Cross Validation Scores:
[0.76621245 0.41771035 0.35744057 0.73955563]

Mean Score:
0.5702297501214788


texts: Model assessment for EXTREMELY imbalanced dataset

texts:
I am dealing with an extremely imbalanced dataset, with about 10,000 negative samples for each positive sample.
I am now trying to come up with an adequate measurement of model accuracy but none seem to fit.
Many places recommend the PR curve over the AUC curve for imbalanced datasets (e.g. The most informative curve for imbalance datasets) but it looks like all these recommendations are good for less imbalanced sets.

Since the denominator for precision takes into account the number of false positives, more negative samples in the dataset means smaller precision vale.

I noticed that my PR curve tends to look nice and informative as long as I keep the positive/negative ratio to around 1/10-20, but as more negative samples are taken into account the curve looks less and less like it should.

My question is whether there's a better way to assess model performance for super imbalanced datasets, or maybe I am missing something with my interpretation of the PR curve and its purpose.


texts: Anomaly Detection over multivariate data containing Nominal and numerical predictors

texts:
I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors.
Dataset has following pattern:

If we consider the below sample records, category_id, currency, and product_id are nominal predictors, whereas price is a numerical variable.
My model is able to identify the anomaly in the price for '_id=4' because the price range for different products for the particular combination of category_id-currency-product_id is between 10-500EUR.
But it is not able to identify anomalies for product_id=1, product_id=2 or product_id=3 as they have unusual category_id or currency or product_id.

Correlation matrix for dataset:

I am applying 'Multiple Correspondance Analysis(MCA)' over nominal predictors and reducing these 3 predictors to 2 predictors.
Algorithm:
I have tried with OC-SVM, Isolation Forest, and Autoencoders, but none perform as expected.
Some of the resources that I have checked but to no help:
Unsupervised Anomaly Detection with Mixed Numeric and Categorical Data
Anomaly Detection/Novelty detection
I am not able to understand where I am making the mistake: Data pre-processing OR Algorithm OR something else?
I would highly appreciate any help here.
You can review my code here.


texts: How to know the state-of-the-art recommended approaches for data science?

texts: Data science, AI, NLP, and visualization are changing so fast. I wonder if there is a way/blog that shares the latest updates and recommended using certain techniques or avoid using others. For example, many NLP books are old and they would provide examples using TF-IDF. However, nowadays there are much better approaches but they are also changing fast. I am hoping to find some source that would say use these new techniques and avoid using those old techniques. Searching the web can help, but will bring back a lot of noise.


texts: Is there a Hyperopt equivalent for optimization in R?

texts: I've used Hyperopt in Python, but I'm looking for a package with similar capabilities in R. Does a package like this exist?


texts: quest for online tutorials on CNN backprop

texts: I'm looking for video or text tutorial materials on CNN backprop that show how weights (convolutional kernels and last layer weights) are trained. Anyone know?


texts: Comparing word frequencies between two corpora to find keywords

texts:
I have a set of tweets that are either labelled as ironic or non-ironic, and have separated them into two different corpora: ironic and non-ironic. I want to compare these two to see which words are the keywords of each corpora, remove them from the tweets and see how this affects irony detection.

So far, I have simply counted the frequency of the words in each corpus and ranked them this way. I have also computed the relative frequency of each word in the ironic corpus, and then compared it with the same word's relative frequency in the non-ironic corpus (and vice versa). This was done by either taking the difference (e.g. the word "love" has a relative frequency of 12 in the ironic corpus, and 5 in the non-ironic corpus, which gives a difference of 7) or the ratio (using the same example: 12 / 7 = 1.7).

However, I have been reading some papers about this, which is making me rethink my method. They mention the following ways of comparing two corpora:

Pearson's chi-squared test
log-likelihood ratio test
Welch’s t-Test
Wilcoxon rank-sum test
Bootstrap test

Is it best to figure out how to implement these in Python and use one of them instead of comparing the relative frequencies? If so, which ones are suitable to my problem?


texts: Reference implementation of q-learning in Python

texts:
I'm a machine learning newbie, trying to learn Q-learning. I read a few texts and I get the general gist, but what I'd really love to see is a simple example of a Q-learning algorithm in Python that I could run and play around with.
It can solve the simplest of games, I'm not looking for anything fancy.
I've searched and found lots of examples that use the gym framework. This framework looks great, and I'll probably use it later, but I want the most no-frills version of a Q-learning algorithm, with nothing being done automatically for me. I think this will help me get a better understanding of everything that's going on.
Do you know of a simple Python implementation of Q-learning?


texts: Connection between cross entropy and likelihood for multi-class soft label classification

texts:
The connection between cross entropy and log likelihood is widely expressed for the case when sample multi-class labels are one hot binary vectors (basically the same). Cross entropy is defined when labels are not one-hot but any valid probability distribution.

What is the relation between the general cross entropy and log-likelihood?

One-hot case

Both log likelihood and cross entropy come down to:

$-\sum\limits_{i}\sum\limits_k1\{{y_i^{(k)}=1\}}{\log{\hat{y}_i^{(k)}}}(\theta)$

When:

$i$ - the sample index
$k$ - the class index
$\theta$ - model parameters
$y$ - samples labels (one-hot vector)
$\hat{y}$ - predictions (probability distribution like vector)
$1\{\}$ - indicator function

General case ($y_i$ is probability distribution)

Cross entropy can be reduced to:
$-\sum\limits_{i}y_i\log\hat{y}_i(\theta)$

How does this relate to log-likelihood?


texts: Clustering method for 2-D data that self-detects number of clusters and takes care of outliers

texts:
Assuming I have data that looks something like that:

I'm looking for a method or algorithm that can perform the clustering (e.g. as shown in the picture), that automatically determines the optimal number of clusters while also accounting for outliers.
Would appreciate suggestions of methods or relevant implementations/libraries/pipelines, preferably in Python.


texts:
I was trying to plot the cluster result of DBSCAN clustering. I cluster data into two cluster and when i write code to plot those it shows 'name error'. But I couldn't understand whats the problem.
Here's my code with error

for i in range(0, reduced_data.shape[0]):
    if dbscan.labels_[i] == 0:
        c1 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='r',marker='+')
    elif dbscan.labels_[i] == 1:
        c2 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='g',marker='o')
    elif dbscan.labels_[i] == -1:
        c3 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='b',marker='*')
    plt.legend([c1, c2, c3], ['Cluster 1', 'Cluster 2','Noise'])
    plt.title('DBSCAN finds 2 clusters and noise')
    plt.show()

Edited:

Rest of my code:

feature_cols = ['age','workclass','fnlwgt','education','education num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']
X = train[feature_cols]
y = train['label']

# split X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)

X_train_scale = scale(X_train)
X_test_scale = scale(X_test)

reduced_data = PCA(n_components=2).fit_transform(X_train_scale)
reduced_data_test = PCA(n_components=2).fit_transform(X_test_scale)

from pylab import *
xx, yy = zip(*reduced_data)
scatter(xx,yy)
show()

dbscan = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)
labels=dbscan.labels_
print(labels)

n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print (n_clusters_)

The result of n_cluster is 2

and the main data before PCA reducing is like these:


texts: Tensorflow: how to look up and average a different amount of embedding vectors per training instance, with multiple training instances per minibatch?

texts:
In a recommender system setting: let's say I want to learn to predict future item purchases based on user past purchases using an approach inspired by Youtube's recommender system:

Concretely, let's say I have a trainable content-based network that receives as input an item and, based on its content, returns an embedding for such item. Now, let's say each user has purchased a variable number of items in the past (some users might have purchased 5 items, others maybe 1, others maybe 10, some outliers maybe 100, etc.). I want to generate a user vector, a candidate item vector and then a user-item match score as follows:

I map each item purchased by that user to its embedded item vector using the trainable content-based network
I calculate the average of all those embedded item vectors (as illustrated in the picture)
I apply a couple of ReLu layers on top of this average, thus obtaining a user vector
I map a candidate item (to be recommended) to its embedded item vector using the same trainable content-based network of step 1 (the weights of this network are always shared, like a Siamese network so to speak)
Finally, I compute the dot product between the user vector and the candidate item vector, apply a cross entropy loss during training, etc.

So my question is about the technical details of how to implement the embedding lookup and average of a variable number of embedded item vectors per user using Tensorflow, considering that during training each mini-batch may contain many training instances, where each training instance possibly consists of a different user with a different amount of purchased items in the past. Although the context is different, my question is very similar to this one, but unfortunately nobody has answered that question up to now.


texts: Product classification based on Description of a product in eCommerce website

texts:
I am searching for an eCommerce dataset which should have a product category and description for my academic research in Machine learning. Please suggest any public dataset available.

INPUT: UNIONBAY Men's Survivor Iv Relaxed Fit Cargo Pant-Reg and Big and Tall Sizes

OUTPUT: Clothing


texts: Why activation function is not needed during the runtime of an Word2Vec model

texts:
In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer.

Referring to this article, I understand that the reason we have the matrix $W'$ is basically to compensate for the lack of activation function in the output layer. As activation function is not needed during runtime, there is no activation function in the output layer. But we need to update the input-to-hidden layer weight matrix $W$ through backpropagation to eventually reach to the word embedding most suitable for our usecase. So there is this weight matrix $W'$ in the output layer.

But my question is why activation function is not needed during the runtime? Can anyone please explain?


texts: Neural network with multiple kinds of output

texts:
Introduction:

I am trying to build a neural network similar to the following one http://www.nature.com/articles/srep11476 to predict two different features at the same time:

Secondary structure: discrete label with three classes (Helix, Sheet and Coil)
Psi/Phi angles: continuous value (-pi to pi in rad or -180/180 in degrees)

To do so, I use an output layer with softmax activation for the secondary structure and a tanh layer for the angles to predict sin(phi), cos(phi), sin(psi) and cos(psi), which are subsequently converted back to angles phi/psi. Note that this is done to avoid problems with the periodicity of angles.

The cost for the individual features is a cross entropy for the secondary structure and a mean square error for the angles.

Problem

I want to predict both features at the same time by having two different output layers and then train the network with a cost that takes into account both features.

Question 1: the mean square error cost should be done at the sin/cos level or at the angle level (rad or deg)? sin/cos is constrained from -1/1, the rad from -pi/pi and the deg from -180 to 180

Question 2: how can I merge the cross entropy cost for the secondary structure with the mean squared error for the angles in order to make a single cost? I cannot use a simple sum since they have different ranges.


texts: How do I use the model generated by the R package poLCA to classify new data as belonging to one of the classes?

texts:
For example, in the election example from the documentation, if I create a new set of answers to the questions, how can I use the poLCA model to tell me what class (cluster) it's most likely to be in?

There doesn't appear to be a function to do this, though the model has a df within it that lists the probabilities of class membership for each value of each manifest variable. I'm tasked with converting some sql code that takes a second dataset and classifies the patients there as members of the clusters created from a first. Superficially this is a programming question. It seems like a function to do this would be a reasonable addition to the package. More deeply, if indeed there isn't such a function, it would become a question about how to use the table of probabilities to classify new data.

If readers aren't familiar with the R package poLCA, it's an LCA package that works with discrete/categorized data.

(full disclosure: I asked on cross-validated and a shorter version of this question was put on hold.)


texts: Adding Features To Time Series Model LSTM

texts:
have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to adding additional features to what is already a list of time series features. Assuming you have your dataset up like this:

t-3,t-2,t-1,Output

Now lets say you know you have a feature that does affect the output but its not necessarily a time series feature, lets say its the weather outside. Is this something you can just add and the LSTM will be able to distinguish what is the time series aspect and what isnt?


texts:
We have run an AB test at firebase which has the following results:

I was also building my own Bayesian AB-test suite and was wondering how they came to these conclusions.

What I was doing was querying the data of this test for the Control Group and Variant C:

Control Group: \$11943 Revenue from 900 payers of 80491 users.
Variant C: \$16487 Revenue from 894 payers of 80224 users.

I based my algorithm on this tool: https://vidogreg.shinyapps.io/bayes-arpu-test/. When I enter these inputs I get the following result:

This tool seems to be much more confident that Variant C is better than the control group then Firebase. It also seems like the Firebase distributions for Revenue per user are skewed while the Bayesian ARPU tool has very symmetrical distribution.

The code for the Bayesian ARPU tool is available. They used conjugate priors to get to these conclusions based on this paper:

https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf

Can anyone help me out which results are the best?


texts: What approaches are there to not display a search result that a user has no permission to see?

texts:
Question

What approaches are there to not display a search result that a user is not supposed to see?

Example

Suppose we have the following situation.

High-level view

In an enterprise, there are different repositories like websites, databases, filesystem folder etc. And there is a search engine that crawls all those repositories and creates an index. The user can then use the search engine UI to perform searches.

Let's say the search engine was Apache Lucene and the encompassing indexing and retrieval system Apache Solr.

User permissions are managed via Microsoft Active Directory.

The content in the different repositories are managed with all kinds of applications like MS Word, custom-made applications, database management tools etc.

Search-index and user permissions

The index contains the documents A1, A2, B1 and B2.
User A has only permission to see documents of type A*.

Performing a search

Now, user A performs a search for A* AND B*.

Displaying the results

A system administrator with permission to see everything should see all four documents in the search results: A1, A2, B1 and B2.

User A however, who only has permissions to see documents of type A*, should/will only see two documents in the search results: A1 and A2. So user A shouldn't even know that documents B1 and B2 exist.

Elaboration on the question

What ways are there to implement that requirement that users only see documents they are allowed to see?

I suppose it would be a bad idea to add the information to the index whether a user may see a document. I think that because of this use case: suppose all of a sudden user A may only see document A1 and not A2 anymore; those permissions are now set for example on the filesystem. Now it takes a while until the permissions in the index are updated and during that time user A would still be able to see document A2 in the search results even if he can't click on the result and access it anymore.

Also, the above approach probably would break down when there are hundreds of millions of documents and thousands of employees.

Where and how would the security aspect be implemented in a content indexing and retrieval system in the above scenario?


texts: Difference between rbfnn and svr with gaussian kernel

texts:
I try to understand the difference between radial basis neuron network and support vector regression with Gaussian Kernel. I watched Andrew Ng presentation about kernels in SVN, I have read also an introduction to RBFNN in Bishop's book and Wikipedia's article about SVR.

As I understand it correctly in both cases I try to find a function from features space to R by computation of gaussians of a distances between data from training set to some centers distributed in feature space (activation of rbfs). Next I try to find the best set of linear combination of coefficients, to approximate my data as a linear combination of activated RBFs.

However in some papers these methods are mentioned as two different, competitive methods.

So my question is what exactly is the difference?


texts: Time Series Classification for 1 hour blocks

texts:
I am doing some analysis on time series.
The time series would consist of 3 channels and contain 5 minute interval data.

What I want is to be able to give it a 1 hour block of 5 minute interval data and it will categorise it based on the entire one hour and picking up some patterns how the time series looks for each of the categories as per the training data.

I have many 1 hour series of 5 minute interval data which is classified to a particular category, and I want to be able to have a deep learning model which can detect the pattern between these samples and be able to determine for new samples which categories they belong to.

****Could you please recommend a type of deep learning model which is capable of this?****

Maybe I don't understand LSTM's but to my understanding they provide a prediction for each point in a time series based on the points that occur before it and would therefore give a series of predictions, where as I want 1 prediction for each hour.

I appreciate any help that you can provide to help me understand this better.

Thankyou.


texts:
I have been asked to unit-test my machine learning model(not the code that made the model). Since we wouldn't actually know what predictions models make, how to carry out the unit-testing to check the model's predictions against? How is this done?
EDIT 1:
The machine learning model I have is trained on tabular data of patients. let's take an example of cancer prediction(I am not allowed to disclose the actual one, but this example is very close). It takes multiple reading from various tests as inputs and outputs how close or how risky a patient is to get cancer.
EDIT 2:
Is there any way, like testing for range of value for every set of inputs (or) adversarial inputs(inputs that are sure model will fail on) or extreme input cases. What ate the best practices for this?


texts: Fully Convolutional Variational Autoencoder

texts:
I would like to make a neural network which uses black and white images as input and outputs a colored version of it. The important thing in that process is that the size of the images must stay the same.

Usually this is done by using a Fully Convolutional Network with GAN or AE architecture. Now I have decided to implement a VAE version, but when i looked it up on the internet I found versions where the latent space was Linear/Dense meaning it breaks the Full Convolution.

Is a VAE not a valid Approach for this type of neural network? Or is there a solution/some code someone can provide which would help in that situation? (Pytorch prefered)


texts:
I am working on an ML model for student churn prediction. It is a classification problem if some student will churn or not. I have a lot of data like the student data and the activities of the student. There are two problems which I would like to ask about:

The churn of the student in the first 6 weeks
The overall churn of the student after the 6 weeks

Would you split your work between 2 models: in 6 weeks and after 6 weeks?
How would you start such a project?


texts: Adjusting Probabilities When Using Alternative Cutoffs For Classification

texts:
I am reading Applied Predictive Modeling by Max Khun. I chapter 16 he discusses using alternate cutoffs as a remedy for class imbalance.

Suppose our model predicts the most likely outcome of 2 events, e1 and e2. We have e1 occurring with a predicted probability 0.52 and e2 with a predicted probability 0.48. Using the standard 0.5 for e1 cutoff we would predict e1, but using an alternative cutoff of 0.56 for e1 we would predict e2 because we only predict e1 when p(e1) > 0.56.

My question is, does it make sense to also readjust the probabilities when using alternate cutoffs. For example, in my previous example using 0.56 cutoff of e1.

p(e1) = 0.52;
p(e2) = 0.48

Then we apply an adjustment of 0.56 - 0.5 = 0.06.

So

p_adj(e1) =  0.52 - 0.06 = 0.46;
p_adj(e2) = 0.48 + 0.06 = 0.54

Basically we shift the probabilities so that they predict e1 when p_adj(e1) > 0.5.

I apologize if there is something obviously flawed with my logic but it feels intuitively wrong to me to predict e2 when p(e1) > p(e2). Which probabilities would be more in line with the real-world probabilities?


texts:
I am trying to implement shared layers in Keras. I do see that Keras has keras.layers.concatenate, but I am unsure from documentation about its use. Can I use it to create multiple shared layers? What would be the best way to implement a simple shared neural network as shown below using Keras?

Note that all the shape of input, output and shared layers for all 3 NNs are the same. There are multiple shared layers (and non-shared layers) in the three NNs. The coloured layers are unique to each NN, and have same shape.

Basically, the figure represents 3 identical NNs with multiple shared hidden layers, followed by multiple non-shared hidden layers.

I am unsure how to share multiple layers as in the Twitter example, there was just one shared layer (example in API doc).


texts: How can I create a table from an existing table in SQL but using cells from the old table as columns in the new table?

texts:
I have a table,

and I want to create a new table such as the one below (from the table above)

In SQL, I tried using the following commands. I am able to generate a table with only one column like this,
CREATE TABLE table2 AS
SELECT balance
FROM table1 WHERE balance='currency'

But if I try to do multiple WHERE clause's it doesn't seem to work.
I tried to do,
CREATE TABLE table2 AS
SELECT balance, category
FROM table1 WHERE balance='currency', category= 'date_valid_from'

I also tried using an AND statement but still no luck,
CREATE TABLE table1 AS
SELECT balance, category
FROM table2 WHERE balance='currency' AND category = 'date_valid_from'

This just returns empty columns of 'balance' and 'category' and not the specified Where clauses.

ALSO, the columns in the new tables, are the ones which I selected from the old table. i.e 'balance' and 'category' However I wan't the cells within those columns i.e 'currency' in 'balance' to be the new columns.


texts:
I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error.
The machine has 16G RAM, so I don't think this should have happened. I tried with some logging where the error came and  found that the data got loaded completely into numpy arrays. And it was the algorithm itself that caused the error. Can someone help me with this
or sugggest more memory efficient algorithm for feature selection, or is subsampling the
data my only option. Are there some deterministic good subsampling techniques.

EDIT:
Relevant code piece:

n=8;
y=mydata[:,0];
X=mydata[:,[1,2,3,4,5,6,7,8]];
#print y;
#print X;
print "here";
omp = OrthogonalMatchingPursuit(n_nonzero_coefs=5,copy_X = False, normalize=True);
omp.fit(X,y);
coef = omp.coef_;
print omp.coef_;
idx_r, = coef.nonzero();
for id in idx_r:
        print coef[id], vars[id],"\n";

The error I get:

File "/usr/local/lib/python2.7/dist-packages/sklearn/base.py", line 324, in score
return r2_score(y, self.predict(X), sample_weight=sample_weight)
File "/usr/local/lib/python2.7/dist-packages/sklearn/metrics/metrics.py", line 2332, in r2_score
numerator = (weight * (y_true - y_pred) ** 2).sum(dtype=np.float64)
MemoryError


texts: Why do I need pre-trained weights in transfer learning?

texts:
I am using a Mask-RCNN. I first chose the resnet50 backbone then downloaded COCO pre-trained weights.

Why do I need pre-trained weights for transfer learning?

The transfer learning approach is to train a base network and then copy its first layers of the target. My base network is resnet50 and I copied the first layers to Mask RCNN. So, why do I need pre-trained weights, for example, coco pre-trained weights?

I understand that I copy layers and pre-trained weights from resnet.

Thanks in advance.


texts:
I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix.

          Q1        Q2        Q3
Q1  1.000000  0.707568  0.014746
Q2  0.707568  1.000000 -0.039130
Q3  0.014746 -0.039130  1.000000

Why is it symmetric? Why is Q1:Q2, the same as Q2:Q1? Shouldn't they be inverses of each other? How do I read this and understand the relation?


texts: How to compare between two datasets of lexical density?

texts: I have two dataset from two different texts representing lexical density as a proportion based on a corpus. Both datasets are represented in the images below. Now, let's suppose I want to know which text has more uncommon vocabulary. How should I proceed? What statistics should I use? Should it be a t-students test or Wilcoxon signed-rank test? I'm lost on this one, and I don't wanna apply inference blindly. I am using the python library wordfreq to get word frequencies data.


texts: Is an image with 5 labels equivalent to 5 images with 1 label?

texts: I am collecting data to train an object detection model using and was wondering if 5 labels in the same image and 5 images with 1 label each provided the same quality of input training data. Example: an image with 5 labeled apples vs. 5 images with 1 apple each.


texts: What is the best way to match street addresses in a dataset in Python?

texts:
I have a dataset of land parcels owned by the government and I am attempting to match street addresses to an existing list of government agencies. I've used fuzzy matching and used a regex that ignores casing and distinctions between direction (for example North and N are treated the same).

However, the program ends up having a very poor matching rate, as a lot of the addresses are not getting matched. What are some other ways I should try to improve the matching rate?


texts:
My objective is to build an N-dimensional array in Python that has the mean of the intersection of a number of features.

For example, I have the number of customers in a store at any given time. I want to create a three dimensional array with axes [month, day_of_week, hour] where the values are the mean of the historical values.

One example data point, where the data covers several years but each axis here is just 12 months, 7 days, and 24 hours.

Array[January,Friday,1pm] = 3 customers on average

I've tried Pandas Groupby but I can't figure out how to translate that information from a Groupby object or Panel to a useful 3D array.

My closest attempt so far:

dataframe.groupby(["month","day_of_week","hour"])["count"].mean().unstack().to_panel()

This outputs a Panel object, but I can't find a way to translate this to just a 3D array.

Can anyone help me figure out a way to solve my problem and build this 3D array without for loops?


texts: How to know if classification model is predicting 1 or 0

texts: I have used logistic regression to predict whether customer is good(1) or bad(0). I got the accuracy .80 . How do i know whether model predicted 1 or 0 .Is it related to parameter of model1.predict_proba(X_test)[:,1] (the 1 in the end in square bracket).


texts: Pixel labelled image is must for semantic segmentation using deep learning in matlab?

texts: I have large set of CT images containing lung regions. But I don't have corrosponding pixel labelled images. So how I could do semantic segmentation using DL in matlab. Like a test program I created 6 pixel labelled images using image Labeler. But it's not practical. I have thousands of such images. Could someone help me with a solution?


texts:
For my Computational Intelligence class, I'm working on classifying short text. One of the papers that I've found makes a lot of use of granular computing, but I'm struggling to find a decent explanation of what exactly it is.

From what I can gather from the paper, it sounds to me like granular computing is very similar to fuzzy sets. So, what exactly is the difference. I'm asking about rough sets as well, because I'm curious about them and how they relate to fuzzy sets. If at all.

Edit: Here is the paper I'm referencing.


texts: GridSearchCV with Random Forest Classifier

texts:
I'm working with a supervised learning problem and trying to predict a binary label and using a Random Forest to do so.
I'm trying to tune my hyper-parameters to give me a best model based on my data.
I can do this with GridSearchCV(), but is this correct to do with a random forest?
If I'm using GridSearchCV(), the training set and testing set change with each fold. From my understanding we can we set oob_true = True in RandomForestClassifier(), we are already evaluating on the out-of-bag samples (so CV is kind of already built in RF).
What is the convention to hyper-parameter tune with Random Forest to get the best OOB score in sklearn? Can I just loop through a set of parameters and fit on the same training and testing set? Can I use GridSearchCV(), or does that make no sense with RF?


texts: Torch and conceptual question about neural nets backpropagation

texts:
First a theoretical question and then a practical one.

Neural nets backpropagation is the computation of the weights derivatives or the computation of the new weights (that is, the original weights minus  the weight derivatives times the learning rate - simplified)?

It may well be a semantics issue but important nevertheless.

Also, if anyone is familiar with Torch, nn class

gradInput = module:backward(input, gradOutput)

is gradinput the weight set for the next forward pass or is it the derivatives of the weights of the previous forward pass?

Thanks!


texts: Document similarity: Vector embedding versus BoW performance?

texts:
I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches:

A vector embedding (word2vec, GloVe or fasttext), averaging over word vectors in a document, and using cosine similarity.
tf-idf or its variations such as BM25.

Will one of these yield a significantly better result? Has someone done a quantitative comparison of tf-idf versus averaging word2vec for document similarity?

Is there another approach, that allows to dynamically refine the document's vectors as more text is added?


texts:
I'm new in this field and I started working with data by using R. Because of that, I find R much easier to approach a data project. However, apparently an employer wants you to know an object-oriented programing language (a language like Python).

So it would be smart to think I can use Python just when I need to deal with a complex programming process, like replacing the na's in Titanic/kaggle project with the average based on name and to use R for anything else? So use them both interchangeably?

Beside the fact that Python is more programming oriented I don't see why somebody would use it over R...


texts: Understanding a simple example of Restricted Boltzmann Machine (RBM)

texts:
I am trying to get an intuitive idea of RBMs out of curiosity, and using a simple example on youtube based on preferences for different sports, which denote profiles roughly corresponding to outdoor and indoor predilections. The idea is to target ads. Here is the model:

The examples in the sample are:

Example Football Hockey Cricket Table-tennis Chess Badminton      CLASS

P1          1       1       1         0        0      0           out
P2          1       0       0         1        1      1           in
P3          0       0       0         1        1      0           in
P4          1       1       0         0        0      0           out

The starting values are:

$W_i = 0.5 \quad A_i=0.2 \quad B_i=0.3.$

Given the first example,

$\begin{matrix}V_1 =V_2=V_3=1;\quad V_4=V_5=V_6=0 \end{matrix}$

Now the first step (updating hidden unit) with the first example $P1$:

$\begin{matrix}\text{Example}&\text{football}&\text{hockey}&\text{cricket}&\text{tt}&\text{chess}&\text{badm.}\\\text{P1}&\color{blue}1&\color{blue}1&\color{blue}1&\color{blue}0&\color{blue}0&\color{blue}0\end{matrix}$

is calculated (on the video) as follows:

$\begin{align}\small\text{net}(H_1)&=(V_1H_1W_{11} + V_2H_1W_{12} + V3H_1W_{13} + V_4H_1W_{14} + V_5H_1W_{15} + V_6H_1W_{16})+B_1\\[2ex]
&=\color{blue}1\times 1\times 0.5 +\color{blue}1\times 1\times 0.5 +\color{blue}1\times 1\times 0.5 +\underset{\text{whether the 0 term is } V \text{or} H \text{ or both is not specified}}{\underbrace{0+0+0}}+0.3
\end{align}$

and

$\begin{align}\small\text{net}(H_2)&=(V_1H_2W_{21} + V_2H_2W_{22} + V3H_2W_{23} + V_4H_2W_{24} + V_5H_2W_{25} + V_6H_2W_{26})+B_2\\[2ex]&=\small \color{blue}0\times 1\times 0.5 +\color{blue}0\times 1\times 0.5 +\color{blue}0\times 1\times 0.5 +\color{blue}1\times 1 \times 0.5+\color{blue}1\times 1 \times 0.5+\color{blue}1\times 1 \times 0.5+0.3\end{align}$

Here is the actual screen capture (slightly modified):

which seems to imply a change in the $V_i$ values for table-tennis, chess and badminton when transitioning from the update of $H_1$ to the update of $H_2$. Although I'm sure erroneously, I was expecting the following calculation, which would leave the $V_i$ values unchanged (blue for $V_i$). The  red integers corresponding to $H_i$:

$\begin{align}\small\text{net}(H_1)&=(V_1H_1W_{11} + V_2H_1W_{12} + V3H_1W_{13} + V_4H_1W_{14} + V_5H_1W_{15} + V_6H_1W_{16})+B_1\\[2ex]
&=\small \color{blue} 1\times \color{red}1\times 0.5 +\color{blue}1\times \color{red}1\times 0.5 +\color{blue}1\times \color{red}1\times 0.5
+\color{blue}0\times \color{red}{0}\times 0.5 +\color{blue}0\times \color{red}{0}\times 0.5+\color{blue}0\times \color{red}{0}\times 0.5 +0.3\end{align}$

$\begin{align}\small\text{net}(H_2)&=(V_1H_2W_{21} + V_2H_2W_{22} + V3H_2W_{23} + V_4H_2W_{24} + V_5H_2W_{25} + V_6H_2W_{26})+B_2\\[2ex]
&=\small \color{blue}1\times \color{red}0\times 0.5 +\color{blue}1\times \color{red}0\times 0.5 +\color{blue}1\times \color{red}0\times 0.5
+\color{blue}0\times \color{red}1\times 0.5 +\color{blue}0\times\color{red} 1\times 0.5+\color{blue}0\times\color{red} 1\times 0.5 +0.3\end{align}$

Can I get an explanation as to how the example $P1$ is not ignored in the first updated of the hidden unit?

Evidently following the example would be much more self-explanatory if $P1$ wasn't such a pure outdoor case...


texts: When being in a perfect "Long Valley" situation, does momentum help?

texts:
I am thinking of the following image where we have two weights and an error (so we can make a 3D visualization). In this case a "long valley" looks like $x^2$ in the plane of the gradient, but in a perpendicular plane the function can still be minimized:

According to my understanding, all optimization algorithms which are based on the gradient should be "trapped" at the saddle point like SGD is. Why does having a momentum term help in this case?

(I'm sorry, I don't know who made those images. More of them are available.)


texts: Find ID for the name in the list from csv file in python

texts:
The CSV file contains:
ID,Name
1,AS
2,er
3,rtf
4,addfs

The list contains the name for example (er,rtf)
I want to find the ID's corresponding to these names mentioned above
How to find the ID's using the Python code.

Thanks in advance


texts: How can we scale up the number of classes for deep learning after training a model?

texts: I have trained a deep learning model for a face recognition application. The model has been trained on the existing subjects (or call it classes) in the gallery. Now consider the real world problem, where the gallery size is dynamic as new subjects are to be enrolled, how can deep learning solve those type of problems? I have tried transfer learning, are there any other techniques?


texts: Making Factual drake work on Windows 7 64-bit

texts:
I have installed Drake on Windows 7 64-bit.

I am using JDK 1.7.0_51.

I tried both using the pre-compiled jar file and
compiling from the Clojure source using leiningen.
The resulting Drake version is 0.1.6, the current development version.

When running Drake, I get the current version number.

Next, I tried to go through the tutorial. The command:

java -jar drake.jar  -w .\workflow.d

results in the following Exception:

java.lang.Exception: no input data found in locations: D:\tools\drake\in.c
sv

Even though the file exists and has text inside it.
The same scenario works in a similar installation on Ubuntu 12.04.
Am I doing something wrong, or is this a Windows-specific bug?


texts: Explaining the similarities between instances in a cluster with KMeans

texts:
If I create clusters using the KMeans clustering algorithm in Python, is there any way I can find which attributes were used to group those instances in clusters?
Example: I have a dataset of cars with columns 1 - 10. I get 3 clusters, how can I know which attributes were used to make them? Is it because all instances in cluster 1 have the same value in columns 2 and 3?


texts: Shapley values without intercept (or without `expected_value`)

texts:
I have a model and I want to derive its interpretability by using feature contributions. In the end, I want to have some contribution per feature such that the sum of contributions equals the prediction of the model.
One approach may be to use shapley values. The issue with shapley values and its implementation in the shap python library is that it also derives and expected_value, which I'll call $E$. To obtain the prediction of the model, one must add all the shapley values and $E$. Is there a way to derive feature contributions without the need of $E$?

My solution
I've derived a solution but I'm not sure it makes sense. Let's say I have features $x_1, \dots, x_n$ with shapley values $\phi_1, \dots, \phi_n$ for the model $f$. Then, I have
$$
f(x) = \phi_1 + \dots + \phi_n + E \Rightarrow f(x) = (1 + \frac{E}{\sum \phi_i})(\phi_1 + \dots + \phi_n)
$$
(by using a simple math trick). Then, I claim that I have shapley values, $\hat{\phi_1}, \dots, \hat{\phi_n}$ where
$$
\hat{\phi_j}=(1 + \frac{E}{\sum \phi_i})\phi_j
$$
These values are suitable with
$$f(x) = \hat{\phi_1} + \dots + \hat{\phi_n}$$
But, does it make sense to call them shapley values?
Any criticism is more than welcome.


texts: Problem with two layers: how to predict?

texts:
Let me get straigt into it:

Think of an scheduled operation. There is a surgeon and, say, three other medical personel that carry out various tasks. For instance, a nurse, an anesthesiologist, and a backup.

Each one reports to duty at a certain time. However, with some probability p (to be predicted) one or many member(s) of the team won't show up for work. If an individual does not show up for work, but his particular skill set is still covered due to the backup, then the operation is done. Otherwise, it is cancelled. For instance, if the backup is a certified anesthesiologist, absence of the scheduled anesthesiologist does not prevent the surgery from taking place since a minimum required skill mix is given. If, however, the backup is a nurse, and the anesthesiologist does not show up, then the surgery has to be cancelled.

This problem involves two layers:

Does an individual team member show up for work?
Is an individual team member's skill critical for the execution of the surgery.

My goal is to predict cancellation of the surgery, which involves step 1 and step 2 simulataneously (alternatively, you could think of step 2 being conditional on step 1).

I have a good understanding of how to use standard machine learning algorithms to predict 1. But how do I generate final predictions taking both layers into account? The problem set up seems reasonably general so that I would assume there exists a "standard" method?


texts:
Hello data scientists (I'm not),
I adapted a multi-class-text-classification-cnn from https://github.com/jiegzhan/multi-class-text-classification-cnn (thanks Jie Zhang)
It works ! 90% accuracy !
Now my issue ;-)
I really need to load my model with java in order to allow my existing java service to predict categories.
According to documentations, I need to use SavedModelBuilder which is the new way of modele saving. I tried for hours but I misse a lot of things... About signature, assets.
If someone could help me with that ? => replace the old saver.save by tf.saved_model.builder.SavedModelBuilder into https://github.com/jiegzhan/multi-class-text-classification-cnn/blob/master/train.py
He or she will have my infinite gratitude


texts: What is the best way to understand changes made in new releases? For example, Orange 3.21 came out recently and I’m eager to understand any prominent fixes or modified features.


texts: While Merging image datasets which of the image parameters should be prepossessed/Normalized before giving to a CNN Model?

texts: When two datasets are merged or images of different parameters size, dimension, Format are combine which parameters of the datasets should be normalized/ pre-processed before giving it to a model?


texts: Time series classification using CNN model, 1D or 2D?

texts:
I have a multivariate time series dataset that has the same length for each observation but looking at a different time frame (eg. One might be from January to May and another one might be from August to December). Those variables are assumed to be correlated. I am currently working on applying CNN model for this data to do binary classification. I cannot say much about data but it is like classifying time series with anomalies.
What I know is that usually 1D CNN is used for the time series dataset. But since I have multivariate time series, I can basically format them into the 2D matrix and passed them through 2D CNN too.
Questions I have are

For 2D CNN, for example, the location of a dog on the image does not matter. Is it the same for 1D CNN? whether anomaly happens at the beginning of the time series or at the end for each variable?

If I were to have 5 variables (5 time series per observations) and they are assumed to have a correlation with each other regarding anomaly, is it more logical to use 1D CNN where kernel spans through whole 5 time series as it convolves or it doesn't really matter and 2D CNN also does the job.

Any suggestions on how the model should be developed? I read about the CNN+LSTM model or just LSTM being used for time series classification. Also, I saw a more "traditional" methods of extracting statistical features from time series and use random forest or other machine learning models.


texts: What is the mathematical definition of the latent space

texts: I have been seeking a mathematical explanation of the latent space of a neural network. The best I have gotten is "The latent space is the space in which the data lies in the bottleneck layer." However, what do we mean by "the space in which..." rigorously speaking?


texts:
Can some one advise me direction where to look in.Or some resources. Here is a task:

User leaves feed back-text with min 50 characters.
I need to check if it's normal human sentences/ word combination OR just bag of words and characters.

For ex ( 1-normal, 0-not normal):
"I wrote question.hope for answer" - 1(class)
"Bla bla goog goog goog gooo" - 0(class)
Maybe some dataset available.or some approach?
Thanks in advance!


texts: If I trained a network using Neural Network classifier, how can I know which feature was most important for predicting the target variable? I mean how to create a "feature ranking" among the features (from high importance value to low). I have seen some literature about decision trees/AdaBoost but I am typically interested in Neural networks especially for classification purpose. To make it more clear, an example is shown in the figure.


texts:
I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of the actions may repeat). The dataset looks like as below:

Datapoint 1    Datapoint 2 .............  Datapoint 100

Add wall        Add wall                   Add window
Edit wall       Remove Roof                 Add wall
Add wall        Add window                  Edit wall
.......         .........                   .........
........        .........                   .........
Remove door     Add door                    Remove door

My target is to predict the next design actions. However, when I used these sequences in the LSTM model, the prediction accuracy is not so high (35%). For this reason, I am thinking if I can use any embedding model. It is mention-worthy that actions in the sequences are correlated. It means each action has a certain relation with its previous actions and later actions. How can I represent these relationships using embedding? In short, I want to build my own embedding based on the sequence. If anyone help me to provide some reference, paper, it would be highly appreciated.


texts: Pearson correlation with data sets that have values on different scales

texts:
I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have values on different scales.
The features in both datasets are exactly the same (the actual samples are of course different). The range of values are as follows:
dataset1 = 3-20
dataset2 = 10-30

Now my understanding is that pearson correlation coefficient is not dependent on the change or difference in scale of the different datasets, however for all examples I have seen, although the scales are different, the lower value is always 0. As you can see though, the lower value on one dataset is 3 and the lower value on the other is 10. (in either case though they mean they same level of 'low', it is just how they are measured.
Can anyone confirm this is still fine?


texts:
I am dealing with a highly unbalanced data, so I used the SMOTE algorithm to resample the dataset.
After SMOTE resampling, I splitted  the resampled dataset to training/testing sets, using the training set to build a model and testing set to evaluate the model.
However, I am worried about that some data points in the testing sets might actually jittered from data points in the training set (i.e. the information is leaking from the training set to testing set), so the testing set is not really a clean set for testing.
Does anyone have similar experience? Does the information really leak from training to testing? Or the SMOTE algorithm actually took care of it and we don't have to worry about it?


texts: Data Generation for LSTM with variable time input

texts:
So, I have a dataset that observes entities for 60 time periods. At each time period, I collect some X observations per entity.
So I have an LSTM to predict some element in the 60th time period using observations in the first T periods. However, say I want my LSTM to be able to take as input any T from 1 to 59, I know that I can just specify None in the input dimension to have my LSTM accept variable inputs. The problem is getting this data. I have around 100000 entities with observations over 60 time periods for each. I have a function that, for each entity, duplicates the row and limits it. So for row 1, there are now 60 versions, showing observations for 1 period, then 2 periods, and so on. Easy enough to implement in pandas.
The issue is, this takes forever. Doing this to just 100 rows takes 10 seconds, since appending to a dataframe is slow. How can go I about making this faster?? See the code below
def multDat(dat):
    fnl = dat.head(0)
    for x in range(1,60):
        #return names of columns that are up to period x
        ntars = narrowScope(dat.columns, x)
        temp = dat[ntars]
        #mark for refrence the max availabiloity of the rows
        temp[str(x)+"available"] = 1
        #append temp dataset to master dataset
        if x == 1:
            fnl = temp
        else:
            fnl = fnl.append(temp)
        #print(temp.shape)
    return fnl

Thank you!


texts: mAP calculation for object detection yolov3

texts:
I'm currently working on yolov3 implementation in tensorflow 2.0 and currently I'm working on calculating the mAP(mean average precision) scores to evaluate trained models which is a popular metric in measuring the accuracy of object detectors like Faster R-CNN, SSD, yolo, ... Average precision computes the average precision value for recall value over 0 to 1.

Here's an illustrative plot that calculates mAP scores for 30 classes:

31.05 % is calculated by averaging the average precision values for all classes in the model

(9.32 / 30) * 100 = 31.05%.

As of my understanding the more the figure, the better and it is a good indicator to rely on when deciding to stop the training / train further.

The question is assuming for example I have a dataset of 2000 images with n bounding boxes and m classes and the validation data will be 0.2 of the dataset size(1600 training examples and 400 validation examples) should I be calculating the map scores on the whole training dataset after each epoch or just take some fraction of it?

I mean to calculate training and validation mAP scores should I make predictions at the end of each epoch and calculate mAP accordingly on 1600, 400 examples respectively? or just take a fraction maybe(100 examples) and predict and calculate on the 100 examples? I'm concerned about efficiency, for example if I'm working with a large dataset, let's say 100k images, wouldn't it be extremely inefficient to make predictions on 100k images after each epoch?


texts: Comparing Parameter Importance Across Models

texts:
I have a dataset with 20 features(columns that is). I create a few models pairs with a subset of these parameters.

For example: If I have 6 columns (named A, B, C, D, E, F) with 10k lines of data, one of model in the pair will have (A, B, C) with all 10k lines of data and the other (A, B, C) with maybe say 6k lines of data. Yet another pair will have the parameters (B, C, E), however, the number of lines is always fixed at 10k and 6k.

I am limited to using a sci-kit learn's decision tree and this cannot be changed. I use the following function to calculate my features:

model = clf_gini.best_estimator_
print(model.feature_importances_)

My question is: **Is there any formal, programmatic way to compare feature importance across model pairs using sci-kit, other than intuition? Also, does it make sense to compare to models trained on some similar some different parameters? **

Please let me know if the question is ambiguous, I will clarify.


texts: Sentiment Analysis: using a dataset (IMDB reviews) to train a neural-net and using it to predict entirely different datasets (Political articles)

texts:
We need to analyse a lot of articles relevant to political instability in a given country (things like the possibility of a coalition / a snap election etc). The problem is that I could not find any labeled datasets which could be plugged into a neural network (CNN/LSTM in TensorFlow) so as to supervise it for real-time events (news articles, tweets etc).

I believe we can't use publicly available big datasets - like IMDB film reviews - for training the models to accurately identify and predict the occurrence of such events (or can we?).

Are there other ways to solve this problem?

I also thought of using unsupervised learning - libraries like VADER - but that gives me a more generic sentiment-score, rather than attuned to the specific corpora relevant to the problem.


texts: GPU vs TPU for convolutional neural networks (NLP)

texts: I am testing ideas on IMDB sentiment analysis task by using word embeddings + CNN approach. What could explain a significant difference in computation time in favor of GPU (~9 seconds per epoch) versus TPU (~17 seconds/epoch), despite supposedly superior computational power of a TPU over GPU?


texts: Does flipping binary class labels impact performance metrics?

texts:
I am working on a classification problem with 0 and 1 class labels. I am wondering if I flip the class labels (i.e if I label all the formerly 0 classes as 1 and all the formerly 1 classes as 0) how machine learning performance metrics would change, if it all.
The metrics I am considering are: weighted F1 score, accuracy, balanced accuracy, AUC, precision, and recall.
My intuition is that all metrics would stay the same except precision and recall, which would be flipped (i.e. the precision and recall relative to the former 0 class would be the precision and recall for the new 1 class, etc). Is this accurate?


texts:
I would like to train a network to conjugate verbs and recognize verb conjugations. For example:

Given "To eat, 1st person, past continuous", the model would return "I was eating"
Given "I was eating", the model would return "To eat, 1st person, past continuous"

The benefit of training would be to have the model conjugate and recognize verbs it hasn't seen previously.

I would like to train the network on standard conjugation tables such as http://conjugator.reverso.net/conjugation-english-verb-eat.html.

Is this within the realm of possibilities with our present machine learning technologies?

EDIT: If yes (and it seems to be the case), hints for suggested architectures or actual implementations would be most welcome.


texts: How does a convolutional ply differ from an ordinary convolutional network?

texts:
I am currently working on recreating the results of this paper.
In the paper they  describe a method for using CNN for features extraction, and have a acoustic model that is Dnn-hmm and pretrained using RBM.

Section III subsection A states different ways the input data can be represented. I decided to vertically stack the spectrum plots of the static, delta and delta deltas.

So as such:

The paper then describes how the network should be.  They state that they use a convolutional network, but nothing about the structure of network?.
Further more is the network always refered to as a convolutional ply? which i am sure i see any difference in  compared to an ordinary network convolutional neural network (cnn).

The paper states this regarding the difference:

(from section III subsection B)

  A convolution ply differs from a standard, fully connected hidden
  layer in two important aspects, however. First, each convolutional
  unit receives input only from a local area of the input. This means
  that each unit represents some features of a local region of the
  input. Second, the units of the convolution ply can themselves be
  organized into a number of feature maps, where all units in the same
  feature map share the same weights but receive input from different
  locations of the lower layer

Another thing i was wondering is whether the paper actually states how many output parameter is needed to feed the dnn-hmm acoustic model. I can't seem to decode the number of filters, filters sizes.. in general details of the network?


texts: How to change the input of conv2d from gray scale to coloured image

texts:
I have CNN with Tensorflow for gray images with weights

  w1=[9,9,1,64], w2 = [1,1,64, 32] and w3 = [5,5,32,1]

Now I want to use RGB 3 channel data instead of gray scales. Can anybody help me how to change

  w1, w2, and w3

for RGB images?

Is it possible to use conv2d function for RGB images in Tensorflow?


texts: Data cleaning and data transformation before EDA?

texts:
What is the most common order of data cleaning, data transformation and exploratory data analysis?

For me it seems most logical to do data cleaning, then EDA and finally data transformation (encoding of categorical variables, and feature scaling).

Doing data transformation before the EDA, seems to make the EDA not that useful, as you cant ex. check for stuff like:

  Passengers in the age interval 0-18 has a higher chance of survival

(if feature scaling has been applied to age feature).

But then again, doing data transformation after the EDA, also miss out on chance of encoding categorical variables and thereby visualize correlations of those with the target variable.

What is the order of the mentioned processes? And is there even an order?


texts: Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

texts:
When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less uniformly distributed. This is also referred to as the curse of dimensionality.
However, why not to make lemonade out of this dimensionality lemon?
Why not to encode nominal categorical variables of high cardinality with randomly distributed points in some high-dimensional space but such that the number of dimensions is much less than we would get with the one-hot encoding?
As the distance between any two points is almost the same, no artificial ordering will be introduced. The encoding is very fast. No need to calculate embedding with some neural network.
Will this encoding work? Will it be meaningless or introduce any bias? Is anybody using it? Do you know any references?


texts: Should weights on earlier layers change less than weights on later layers in a neural network

texts:
I'm trying to debug why my neural network isn't working. One of the things I've observed is that the weights between the input layer and the first hidden layer hardly change at all, whereas weights later in the network (eg. the weights between the last hidden layer and the output) change significantly. Is this to be expected or a symptom of an error in my code?

I'm applying backpropagation and gradient descent to alter the weights.


texts: Multilabel image classification: is it necessary to have traning data for each combination of labels?

texts:
I want to train a CNN for a multilabel image classification task using keras. However I am not sure how to prepare my tranining data. More specifically, I am wondering if I need training images that show a combination of two or more labels or if it is sufficient to train the network on single labels and it will then be able to detect multiple labels within an image.

I think my question becomes clearer with an example:

Say I am using the dog vs cat classification dataset and I want to build a model that is able to classify images as either being a dog or a cat or seeing both animals in one image. In this case, do I need to train the model with images showing cats, dogs AND images that show both in one image or is it sufficient to only have training images that only display cats and dogs?


texts: Using multiple machine learning algorithms together

texts: I'm kinda new to machine learning and wanted to know if we could use multiple machine learning algorithms, for example, SVM and backpropagation together to solve a particular problem.


texts: Mapping one embedding to another using Deep Learning

texts:
I am trying to write a model that has the input vector of one embedding (say $E_1$) and predicts the corresponding vector in the second embedding $E_2$. Both are n-dimensional real dense vectors $\mathbb{R}^n$.

Concretely one is a skipgram word embedding, and the other is a node2vec graph embedding. I have approximately 30 000 training examples that provide a mapping between the two. Since they are both just real vectors, it seems like a trivial task to write a simple MLP that learns the non-linear transformation of one to the other (I actually don't really care about over fitting here, since the domain is closed).

However, I can't seem to get it to work properly.

In Keras, naively something like this should work:

in = Input(e1_dim)
hidden = Dense(some_value, activation="tanh")(in)
out = Dense(e2_dim)(hidden)

I have tried adding more hidden layers, but I think my problem lies with the fact that the input and output vectors are in the domain (-1,1), and so the choice of initializer, loss function, and activation function is critical.

I have tried setting the initializer to RandomUniform, but still no good results. For loss I have tried MAE en cosine_proximity, but both seem to produce terrible results. In particular cosine_proximity seems to not get above -0.5 which might be a sign. Any thoughts on the choice of architecture and loss function for mapping one embedding onto another (essentially a high-dimensional non-linear regression?)


texts:
I am working on a text-classification problem, and trying to understand how to work with the tensorboard projector for an embeddings layer in Keras. Borrowing an example from the Deep Learning with R book, I have a model set up like this:

library(keras)

max_features <- 2000
max_len <- 500

imdb <- dataset_imdb(num_words = max_features)
c(c(x_train, y_train), c(x_test, y_test)) %<-% imdb
x_train <- pad_sequences(x_train, maxlen = max_len)
x_test = pad_sequences(x_test, maxlen = max_len)

model <- keras_model_sequential() %>%
  layer_embedding(input_dim = max_features + 1, output_dim = 128,
                  input_length = max_len, name = "embed") %>%
  layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>%
  layer_max_pooling_1d(pool_size = 5) %>%
  layer_conv_1d(filters = 32, kernel_size = 7, activation = "relu") %>%
  layer_global_max_pooling_1d() %>%
  layer_dense(units = 1)

summary(model)

model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("acc")
)

When it comes time to configure the callback_tensorboard object, however, I'm a little lost. It seems the API has changed since the book was written, and I haven't found a good working example yet. The embeddings_data property is apparently required if the embeddings_freq parameter is set, and it needs to match the shape of the model inputs c(?, 500). I can satisfy this by simply passing all the word tokens as a matrix:

tensorboard("my_log_dir")

callbacks = list(
  callback_tensorboard(
    log_dir = "my_log_dir",
    histogram_freq = 1,
    embeddings_freq = 1,
    embeddings_data = matrix(1:max_features, nrow = 4)
  )
)

history <- model %>% fit(
  x_train, y_train,
  epochs = 5,
  batch_size = 128,
  validation_split = 0.2,
  callbacks = callbacks
)

The model trains OK, but when trying to look at the result in projector, it seems that the embeddings are being processed as 64,000 dimensions (2,000 words * 128d layer output?) with a single point for each 500 element vector in embeddings_data. This makes PCA computation stall out.

I expect a final result closer to the example at https://projector.tensorflow.org/, where Dimensions are equal to the layer output dimensions, and there is a single point for each word.

What am I missing? Are there any good working examples of visualizing a standard word embedding layer with current versions of Keras and Tensorflow?

I am using R 3.3.4, Keras 2.2.0, and Tensorflow 1.11.


texts: Please help me out with this Python error - 'invalid syntax'

texts:
Code:

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
house_price = [245, 312, 279, 308, 199, 219, 405, 324, 319, 255]
size = [1400, 1600, 1700, 1875, 1100, 1550, 2350, 2450, 1425, 1700]
size2 = np.array(size).reshape((-1, 1))
#fitting into the model
regr = linear_model.LinearRegression()
regr.fit(size2, house_price)
print("Coefficients: \n", regr.coef_)
print("intercept: \n", regr.intercept_)
#############################
#formula obtained for the trained model
def graph(formula, x_range):
   x = np.array(x_range)
   y = eval(formula)
   plt.plot(x, y)
#plotting the prediction line
graph('regr.coef_*x + regr.intercept_', range(1000, 2700))
print(regr.score(size2, house_price))
plt.scatter (size,house_price, color='black')
plt.ylabel('house price')
plt.xlabel('size of house')
plt.show()

Error

**Error Line:print regr.predict([2000])**
Error: File "<ipython-input-4-9afa91ca7f9e>", line 1
    print regr.predict([2000])
             ^
SyntaxError: invalid syntax


texts: Stuck on deconvolution in Theano and TensorFlow

texts:
I'm captivated by autoencoders and really like the idea of convolution.  It seems though that both Theano and TensorFlow only support conv2d to go from an array of 2D-RGB (n 3D arrays) to an array of higher-depth images.  That makes sense from the traditional tensor-product math, c_ijkl = sum{a_ijk*b_klm}, but means it's hard to 'de-convolve' an image.

In both cases, if I have an image (in #batch, depth, height, width form), I can do a conv to get (#batch, num_filters, height/k, width/k).  I'd really like to do the opposite, like going from (#batch, some_items, height/k, width/k) to (#batch, depth, height, width).

TensorFlow had the hidden deconv2d function for a while (in 0.6, I think, undocumented), but I'd like to know if there's a math trick I can use to get a bigger output in the last two dimensions after a convolution than the input.  I'd settle for a series of differentiable operations, like conv -> resize, but I want to avoid just doing a dense matrix multiplication -> resize like I've been doing so far.

EDIT: As of today (2016/02/17) TensorFlow 0.7 has the tf.depth_to_space method, which helps greatly in this endeavor.  (https://www.tensorflow.org/api_docs/python/tf/depth_to_space)  I would still love a Theano based solution, too, to complete my understanding of the material.


texts: Bidding strategy without some information (vertical search)

texts:
I am facing the following problem which I wish to do some classification. Suppose when a customer search for car insurance, the features one enters are

Currently insured or not (Yes, No, Unknown)
Number of vehicles: {1,2,3}
Number of drivers: {1,2}
Marital status: {Male, Female}

Given this information, multiple companies will bid on the price to have their ads first appear (the highest bid will appear first, and we only concern about the first 5 ads, ranked from 1 - highest to 5 - lowest). Say our company is X, always bid at $10 no matter what, and we get data as follows

The question is how can we better predict what customers will likely buy some policies? The data appears to have, for example, multiple customers with the same 4 features above, but the ranks are different (probably other companies bid differently, so even though we bid $10 at all times the rank does change).
My questions are:

How to take into account information from rank, click, and policies sold to classify (or predict) if a customer buys a policy (policies) or not?

Can we use a simple method (without any fancy Machine/Deep learning) for this problem?

I have tried to classify the ranking based on 4 features, but the accuracy is very low no matter what method I used (Random Forest, Regression, Gradient Boost, ...)


texts: What is the difference between Kafka Channel and Kafka Broker?

texts:
I am a beginner in Apache Kafka. I am reading now some architecture documents and I find in a diagram 2 components:

Kafka Channel
Kafka Broker

What is the difference between these 2 components?


texts:
I am trying to call a procedure for each group of data. For example, I have a list of customers and movie's features that they like. Is a relation one to many, I have many records for one specific customer in dataset. Therefore, what I am trying to do is for each specific customer to apply a clustering algorithm. Need some help with what statement should I apply this on SQL.

Thank you in advance!


texts:
I am on a binary classification problem with the AUC metrics. I did a random split 70%, 30% for training and test sets.
My first attempts using random forest with default hyper-parameters gave me auc 0.85 on test set and 0.96 on training set.
So, the model overfits. But the score of 0.85 is good enough for my business.
I also did a 5-folds cross validation with the same model and same hyper-parameters and the test set results were consistently something between 0.84 and 0.86

My question is: can I believe on the score 0.85 and use this model in production?


texts: Why is shuffling timeseries a bad thing?

texts:
I'm trying to understand precisely why it is a bad idea to shuffle time-series when splitting train and test data.
Like, what is false about shuffling time-series? How does it tamper with the model?


texts: What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

texts:
I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm.

Turns out my model is heavily under fitting, even after 1000 epochs. My loss never improves beyond 0.69321, i tried checking my weight derivatives and instantly realized they're very small ( as small as 1e-7 ), such small gradients are causing my model to never have bigger gradient descent updates and never reach the global minima. I will detail out the math/pseudo code for forward and backward propagation's, please let me know if I'm on the right track. I will follow the naming convention used in DeepLearning.ai By Andrew Ng.

Say we have 4 layer neural network with only one node at the  output layer to classify between 0/1.

X -> Z1 - > A1 - > Z2 - > A2 - > Z3 - > A3 - > Z4 - > A4

Forward propagation

Z1 = W1 dot_product X + B1
A1 = tanh_activation(Z1)

Z2 = W2 dot_product A1 + B2
A2 = tanh_activation(Z2)

Z3 = W3 dot_product A2 + B3
A3 = tanh_activation(Z3)

Z4 = W4 dot_product A3 + B4
A4 = sigmoid_activation(Z4)

Backward Propagation

DA4 = -( Y / A4 + (1 - Y /  1 - A4 ) ) ( derivative of output activations or logits w.r.t to loss function )

DZ4 = DA4 * derivative_tanh(Z4) ( derivative of tanh activation, which I assume is ( 1 - (Z4 ) ^ 2 ) )
Dw4 = ( dZ4 dot_produt A3.T ) / total_number_of_samples
Db4 = np.sum(DZ4, axis = 1, keepdims = True ... ) / total_number_of_samples
DA3 = W4.T dot_product(DZ4)

DZ3 = DA3 * derivative_tanh( Z3 )
DW3 = ( DZ3 dot_product A2.T ) / total_number_of_samples
DB3 = np.sum( DZ3, .. ) / total_number_of_samples
DA2 = W3.T dot_product(DZ3)

DZ2 = DA2 * derivative_tanh( Z2 )
DW2 = ( DZ2 dot_product A1.T ) / total_number_of_samples
DB2 = np.sum( DZ2, .. ) / total_number_of_samples
DA1 = W2.T dot_product(DZ2)

DZ1 = DA1 * derivative_tanh( Z1 )
DW1 = ( DZ1 dot_product X.T ) / total_number_of_samples
DB1 = np.sum( DZ1, .. ) / total_number_of_samples

After the above back propagation steps I updated the weights and biases using gradient descent with their respective derivatives. But, no matter how many times I run the algorithm, the model never improves it's loss beyond 0.69 and the derivatives of output weights ( in my case dW4 ) is pretty low 1e-7. I'm assuming that either my derivative_tanh function  or my calculations of dZ is really off, which is causing bad loss values to propagate back to the network. Please share your thoughts whether my implementation of backprop is valid or not. TIA. I went through
back propagation gradient descent calculus

and

how to optimize weights of neural network .. and many other blogs, but couldn't find for what I was looking for.


texts: How to deal with time series of multi-source energy in order of classifcation?

texts:
I would like to do classification of multi-source energy (wind/solar/teg) repersented in a time series data.
My questions are :
1- What are the most relevent feature that I should chose to do the classification (statistical ones (kurtoisis/means/ variance...) on each sliding window (for experimental purpose) or spectral ones (DWT/FFT)) and which feature selection/extraction method is the best in this case.
2-What is the best classification method should I chose?

Thank you


texts: A situation that validation accuracy is higher than test accuracy at its minimum loss region

texts:
This case would be an example.
Around epoch = 800, we have the validation loss(orange line) reaches its minimum region.

Should we record the accuracy of the model as an average value from epoch 780 to 800? Or shall we take the accuracy after it became an almost straight line (after epoch = 1200)?

If we record the accuracy at the minium point of validation loss. The accuracy may not be the highest. For example, at epoch = 800, val_loss = 0.627,val_accuracy = 0.783. At epoch=909, val_loss = 0.624, val_accuracy = 0.761

what is the best way to do it for such a situation?
Thanks


texts: Ground truth/label modification during training (with the data obtained from the

texts:
I'm working on an image segmentation algorithm with FCN (Long et al., 2015) as the backbone network.

One idea I have is to use the argmax binary mask obtained from the final score layer (250x250x1) to generate some data (e.g. number of blobs in the mask) to modify the ground truth (e.g. set some pixels in the gt mask to 'ignore' labels) or in some way (partly) extract from the features (similar to RPN layer in FasterRCNN).

Does this violate any deep learning or machine learning rules?


texts: Based on the question, what are they and what are their differences? I've seen a lot those terms but cannot find an appropriate answer.


texts: What is the significance of underflow during parameter update using stochastic gradient descent?

texts:
Background

I am using scikit-learn's MLPRegressor to learn a model with the following arguments:

hidden_layer_sizes=(16, 16),
learning_rate_init=1e-2,
solver='sgd',
momentum=0.9,
activation='tanh',
learning_rate='constant'

I use partial_fit() to incrementally fit the model. Occasionally I get a FloatingPointError on this line in the MLPRegressor's code:

updates = [self.momentum * velocity - self.learning_rate * grad
                   for velocity, grad in zip(self.velocities, grads)]

>>> FloatingPointError: underflow encountered in multiply

Problem

I am getting an underflow error when the model is calculating how much to update parameters by multiplying learning rate, momentum, velocity, and error gradients.

Does not an underflow mean that the updates are too small? Does it imply the model has reached some local minimum? Should I take this as a hint that:

I have exhausted my data and training has ended,
I should up my learning rate to keep updates happening
Something else?


texts: Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?

texts:
I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities from the models do not match the actual probabilities in the modeling data.

Is there a way to adjust the predicted probabilities from the class weighted models to match the actual probabilities in the data? I have seen equations for under-sampling (https://www3.nd.edu/~rjohns15/content/papers/ssci2015_calibrating.pdf) but they don't seem to work for class weights. I have searched online for an answer but maybe I am not using the right language?

Thank you!


texts: How can we convert time series data to supervised learning problem?

texts:
I am preparing a data for machine learning model. I want to deal with time series data as normal supervised learning prediction. Let's say I have a data for car speed and I have several cars models such as

+-----+---------+-------------+
| day |  Model  |   Speed     |
+-----+---------+-------------+
|   1 | Bentley | 20.47 km/h  |
|   2 | Bentley | 32.22 km/h  |
|   3 | Bentley | 23.11 km/h  |
|   1 | BMW     | 37.60 km/h  |
|   2 | BMW     | 27.90 km/h  |
|   3 | BMW     | 40.47 km/h  |

so I want to deal with several model in training so that predict the speed for Bentley and BMW.

I have converted the data for training like this :

+---------+------------+------------+-------------------+
|  Model  |   day_1    |     day_2  |    label == day_3 |
+---------+------------+------------+-------------------+
| Bentley | 20.47 km/h | 32.22 km/h | 23.11 km/h        |
| BMW     | 37.60 km/h | 27.90 km/h | 40.47 km/h        |
+---------+------------+------------+-------------------+

Is it a correct approach?


texts:
Given a CNN, say AlexNet:

How could one relate kernel locations at the 3rd conv block, i.e 13x13 filter size to the input image. Would that give a meaningful representation in terms of the input image.

I was thinking of just upsampling the location, from 13x13 to 224x224.


texts: real word dataset on company full name with commonly used short name

texts:
I am involved in a work where i have to recognize company when user does not provides its full legal name. Database only has full legal name which is rarely used by human user. Like no body calls google inc but they just call google. My job is to match that user provided name with database name. The approach i am thinking is to create short name of company from legal name and index that so it what user used could be directly matched.

I was thinking of writing a classifier that could be trained on real world data of full name and short names map and then used to create short names from legal names in database. Are there any such dataset currently available that i can use as input source for my trainer?


texts:
I am trying to setup a big data infrastructure using Hadoop, Hive, Elastic Search (amongst others), and I would like to run some algorithms over certain datasets. I would like the algorithms themselves to be scalable, so this excludes using tools such as Weka, R, or even RHadoop. The Apache Mahout Library seems to be a good option, and it features algorithms for regression and clustering tasks.

What I am struggling to find is a solution for anomaly or outlier detection.

Since Mahout features Hidden Markov Models and a variety of clustering techniques (including K-Means) I was wondering if it would be possible to build a model to detect outliers in time-series, using any of this. I would be grateful if somebody experienced on this could advice me

if it is possible, and in case it is
how-to do it, plus
an estimation of the effort involved and
accuracy/problems of this approach.


texts: How to generate plot of reward and its variance?

texts: I am new to reinforcement learning and I would like know how to generate a learning curve plot such as that shown below (taken from this blog post), that illustrates the reward (return) and its variance (shaded region). I would like to use Matplotlib or any other Python plotting framework.


texts:
In reviewing “Applied Predictive Modeling" a reviewer states:

  One critique I have of statistical learning (SL) pedagogy is the
  absence of computation performance considerations in the evaluation of
  different modeling techniques. With its emphases on bootstrapping and
  cross-validation to tune/test models, SL is quite compute-intensive.
  Add to that the re-sampling that's embedded in techniques like bagging
  and boosting, and you have the specter of computation hell for
  supervised learning of large data sets. In fact, R's memory
  constraints impose pretty severe limits on the size of models that can
  be fit by top-performing methods like random forests. Though SL does a
  good job calibrating model performance against small data sets, it'd
  sure be nice to understand performance versus computational cost for
  larger data.

What are R's memory constraints, and do they impose severe limits on the size of models that can be fit by top-performing methods like random forests?


texts:
I've been searching around the Internet for a while but I have not been able to find detailed instructions on how to install Polynote (the polyglot notebook
with first-class Scala support) for Windows with mixing multiple languages, Python and Scala.

Github Link for Polynote.
Official Website.

According to the official website:

  Polynote is currently only tested on Linux and MacOS, using the Chrome browser as a client. We hope to be testing other platforms and browsers soon. Feel free to try it on your platform, and be sure to let us know about any issues you encounter by filing a bug report

I would really appreciate it if anyone here can share his/her method if he/she
had successfully installed Polynote on Windows, either from Virtual Machines
(VMware/Virtualbox) or directly.


texts: Model and CV setup for structural timeseries forecast

texts:
I wrote this question on cross-validated, but no replies were yet made, so I'm posting here as well since the topic may overlap a little bit. I have no problem with deleting one off one of the sites, if it is in violation of community guidelines.
I'm attempting to use tensorflow-probability's structural timeseries forecast tools, but I'm running into memory allocation problems and the trainings seem to last quite long, so I'd like to check if I'm maybe using the framework in a way it's not meant to be used.
So here's the setup of the problem - I have hourly electricity consumption and meteorological data from 2013 to 2019. I would like to create a model that can perform day-ahead consumption forecasts. The model components I use are various seasonal effects (hour, weekday, month), autoregressive and a linear regression for the meteorological data (air temperatures). The solution I've set up so far looks as in this issue comment, so the build function is something like:
def build_model(observed_time_series, x_effect_design_matrix):

    hour_of_day_effect = sts.Seasonal(...)

    day_of_week_effect = sts.Seasonal(...)

    month_of_year_effect = tfp.sts.Seasonal(...)

    autoregressive = sts.Autoregressive(...)

    x_effect = sts.LinearRegression(
        design_matrix = x_effect_design_matrix,
        weights_prior = tfd.Normal(loc=0., scale=1.),
        name='x_effect')

    model = sts.Sum([hour_of_day_effect,
                   day_of_week_effect,
                   month_of_year_effect,
                   x_effect,
                   autoregressive],
                   observed_time_series=observed_time_series)

    return model

and it's used as:
# get data
training_time_series, training_x_effect = get_training_data()
forecast_time_series, forecast_x_effect = get_forecast_data()

# train model
training_model = build_model(training_time_series, training_x_effect)
q_samples_ = train_model(training_model)

# forecast
forecast_model = build_model(forecast_time_series, forecast_x_effect)
forecast_dist = tfp.sts.forecast(
    model=forecast_model,
    observed_time_series=forecast_time_series,
    parameter_samples=q_samples_,
    num_steps_forecast=num_forecast_steps)

(The training function just uses variational inference, but that part executes more-or-less well, a little slow, but it goes through it)
What bugs me is, after I've trained the model and gotten the parameter samples, what needs to be passed in the forecast_time_series and forecast_x_effect variables? For instance, I've fit the model to hourly data from year 2013 and want to make a forecast for date 2014-05-15. What data needs to be contained in the variables in question? Since the model has components that make use, for instance, the month when data was observed, does that mean that the data passed needs to be from the beginning of the year? (how else is the model supposed to "know" which month it is) But then I get out-of-memory errors and it makes me think that maybe this isn't the way?


texts: What is LSTM, BiLSTM and when to use them?

texts:
I am very new to Deep learning and I am particularly interested in knowing what are LSTM and BiLSTM and when to use them (major application areas). Why are LSTM and BILSTM more popular than RNN?

Can we use these deep learning architectures in unsupervised problems?


texts: Format for proposing a new algorithmic project?

texts:
I am working for an organization that is still new to implementing data science projects. I have an idea for a data spider/algorithm project that will require various pieces of code as well as datasets that I will need to gain access to.

I know that this is something that I should just start doing instead of proposing, but unless I come up with a formal document, the org might see it as a "waste of time". So are there any generally accepted formats and/or project templates for proposing a new data science effort?

To be clear: I am looking for something that I can present to management and be able to "sell" them on a new data science project idea. So the template I'm looking for isn't necessarily too technical, it's more high-level than that.


texts: How to convert vector values to fit k-means algorithm function?

texts:
I have a set of user objects that I want to group using a $k$-means function from their quiz answers. Each quiz question had predefined answers with letter values "a", "b", "c", "d". If a user answers the question #1 with letter "b", I put this answer into vector $(0, 1, 0, 0)$.  The $k$-means function I have to use takes a two-dimensional array of numbers as an input vector (in this case array[user][question]), and I can't figure out how to use it, because, instead of a number value representing a user's answer to question, I have a vector input.
How can I convert my vector values to numbers so that I can use my $k$-means function?


texts: Traditional Predictive Analytics vs Machine Learning Methods

texts:
What is the difference between traditional predictive analytics done using statistics and its tools and, one using machine learning and deep learning?
 How are we leveraging machine learning and deep learning to make predictive models better?
How to decide the boundary for when to use traditional models and, when to use ML and DL?
We have a lot of efficient statistical tools available Stata, SPSS. They are easier to use. Also, computationally efficient.

Thanks

I am unable to find some satisfactory and convincing answer with strong points on google.


texts: Trying to use term document matrix as input to orange 3

texts:
I have a CSV file that has tokens as columns and documents as rows, where the rest of the cells are ints that represent term frequency. I'm trying to use this as input into Orange 3, but Orange 3 doesn't seem to be prepared to handle a term-document matrix as an input. Is there something that I'm missing?

I'm trying to perform topic modeling (hierarchichal dirichlet processing and latent dirichlet allocation, to be specific).


texts: Class label prediction in keras sequential model showing different results in confusion matrix

texts:
With Keras Sequential Model Prediction
To get Class Labels
we can do
yhat_classes1 = Keras_model.predict_classes(predictors)[:, 0] #this shows deprecated warning in tf==2.3.0

WARNING:tensorflow:From <ipython-input-54-226ad21ffae4>:1: Sequential.predict_classes (from tensorflow.python.keras.engine.sequential) is deprecated and will be removed after 2021-01-01.
Instructions for updating:
Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).

or
yhat_classes2 = np.argmax(Keras_model.predict(predictors), axis=1)

With the first class labels if i create confusion matrix, i get
matrix = confusion_matrix(actual_y, yhat_classes1)
 [[108579   8674]
 [  1205  24086]]

But with the second class labels with the confusion matrix, i get 0 for True Positive and False Positive
matrix = confusion_matrix(actual_y, yhat_classes2)
 [[117253      0]
 [ 25291      0]]

May I know whats the issue here?


texts: Magnification factor in image classification

texts:
If a CNN is trained on images focusing on an object, will it also recognize when multiple such objects are present in the image?

For example can a network trained on single flower images also identify image having many such flowers in a bunch?

Is the answer same if situation is reversed? That is, can a CNN trained on bunch of flower images also identify single flower image?