texts: CountVectorizer vs HashVectorizer for text

texts:
I'd like to tokenize a column of my training data (n-gram word-wise), but I'm working with a very large dataset distributed across a compute cluster. For this use case, Count Vectorizer doens't work well because it requires maintaining a vocabulary state, thus can't parallelize easily.
Instead, for distributed workloads, I read that I should instead use a HashVectorizer. My issue is that there are no generated labels now. Throughout training and at the end, I'd like to see which words were the most important for my model, but this isn't possible with Hash.
Is there something I can do to maintain the human-readable labels generated from CountVectorizer but take advantage of a parallel distributed cluster like HashVectorizer can?


texts:
I am trying to duplicate this papers feature engineering for user activity. They take 14 days of accumulated user activity and keep the parameters (2 parameters) that fit a sigmoid to it. I would like to do the same except with 7 days of activity. http://hanj.cs.illinois.edu/pdf/kdd18_cyang.pdf

They use the formula below and keep the parameters x0 and k as features.

from scipy.optimize import curve_fit
import numpy as np

def sigmoid(x, x0, k):
y = 1 / (1 + np.exp(-k*(x-x0)))
return y

I used scipy curve_fit to find these parameters as follows

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata)), ydata, maxfev=20000)

When I had a user that had the values below, I had the following error:

ydata1 = [0,0,0,0,0,91,91]

RuntimeError: Optimal parameters not found: gtol=0.000000 is too small func(x) is orthogonal to the columns of the Jacobian to machine precision.

I noticed that if I add the method 'dogbox' I know longer get the error.

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata1)), ydata1, maxfev=20000, method='dogbox')
print(ppov[0], ppov[1])
5.189237217957538 11.509279446215949

However, I played around with other values and noticed that the resulting parameters can have very different values.

For example. If I have values for that are

ydata2=[0,3,5,30,34,50,91]

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata2)), ydata2, maxfev=20000)
print(ppov[0], ppov[1])
-24.681668846480264 118.77183210605865

However, if I add the method='dogbox' I get very different k and x0 parameter values.

ppov, pcov = curve_fit(sigmoid, np.arange(len(ydata2)), ydata2,  maxfev=20000, method='dogbox')
print(ppov[0], ppov[1])
0.28468096463676695 8.154477352500013

Can anybody help me with 2 things:

I read the doc about 'dogbox' and don't really understand it. Can anybody explain it more simply?
The curve_fit scipy function is looping through about 100,000 users and I need to set the parameters of the curve_fit so it does not throw an error. Is using the 'dogbox' method okay for my purposes knowing that the parameter results seem very different between the 'dogbox' and default 'lm' method? Or, are there other arguments in the curve_fit function that I could set instead that will help me get past this error?


texts: How do we go about imbalanced data for prediction problem?

texts: As in classification we have imbalanced classes, we use up-sampling or down-sampling and other techniques, what do we do when we have imbalanced data in prediction problems, for example, I have distribution of outputs like 90% value1, 5% value2, 3%value3, 2% value4?


texts: How to create a multi-dimensional softmax output in Tensorflow?

texts:
I came across this research paper released by YouTube, on how they use deep learning neural networks for recommendations. It's located here: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf

In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an "output embedding" of each of the 1M video classes.

How is this possible to implement in tensorflow, for example? Isn't softmax supposed to be only 1-Dimensional. If the model outputs an "embedding" like this, as they say it does, how would the training data's labels be formatted as 256-dimensional? In other words, how do they compute the 256-dimensional vector for each of the videos in their training dataset?

Thank you so much for your time and help, guys!


texts: Representation options of strings (keywords/topics) in models

texts:
What are all the possible ways to represent keywords in a machine learning model?

The two I am aware of are:

one hot encoding, using a static index.
vector representation, using
an embedding layer.

We have a specific problem where we are doing client side (browser) ml and need to convert text data into something the model can consume without sending it over to the server.

EDIT: (comment clarification)
The text data is extracted from the page on which our script loads, we then want to run a model locally on the browser - using text based features from the page (ideally). We are minimising in all elements any data needed to sent to the server.

In terms of model, that is not defined at this stage, also this question primary concerned with representation of text based features.


texts: How to encode a job description for machine learning

texts: I'm working on a sample project and one of the features is the job description of a person (categorical, for example: blue-collar, retired, unknown, unemployed, student, etc.). Since in the future more job descriptions could be used, I don't think one hot encoding it's the best approach. How would you encode it without using one hot encoding??


texts:
I am working for a large food retail company and we are using ML models to predict the demand of certain products for the weeks to come.
Of course, looking at the sales distribution, 2020 was a completely irregular year and therefore, it really messes with our forecasting model.
Are you also experiencing this in the data you are analysing? I am really interested how you adapt your data analytics solutions to this situation. Do you exchange features? Do you exclude certain time periods?


texts: Should a Cluster Validity Index contain the same measure(s) as the Clustering Algorithm?

texts:
I'm currently trying to use cluster analysis as a tool for time-series aggregation for a project of mine. The dataset is high-dimensional (386-d), so no chance in assessing the cluster validity visually.

I'm using three different clustering algorithms (k-means++, k-medoids PAM, fuzzy c-means) to find representative periods. As I do not know how many periods (thus, how many centers/medoids k) are present in the dataset, I want to use an internal cluster validity index (cvi) for it. (Basic procedure: run the clustering with multiple k's and plot the cvi against the k; choose highest/lowest k dependent on cvi optimum).

Let's stick with k-means as an example. It is non-deterministic, thus I initiate it multiple times with varying starting points. It then tries to reduce the intra cluster variance. The result with the lowest intra cluster variance of the various initiations with the same k is then kept.

My question is: should the cluster validity index that I use consider the intra cluster variance as a cohesion measure? So to speak: should the cvi use the objective function of the clustering algorithm as a measure to assess "goodness" of clustering?

On the one hand I think, that this would be a good idea, as it can assess, where the clustering algorithm was "most successfull" in its objective function. On the other hand I think, that for a good clustering it shouldn't depend on the same (cohesion) measure. Thus, using the same measure would increase the likelihood of identifying a "bad" clustering, that might represent an unnatural cluster, drawn from random points.

What's "the truth" here?


texts: Removing constant from the regression model

texts:
I am trying to calibrate two variables $(X,Y)$ of different measuring techniques from two instruments, the result of the linear regression analysis appears as shown in the image.
The result shows the regression constant is not statistically significant but the model is significant. I have tried to remove the regression constant (it is a very small value close to zero) and $R$ of the new model is raised to 90%. Is it correct to remove the regression constant?


texts:
Going through this book, I am familiar with the following:

  For each training instance the backpropagation algorithm first makes a
  prediction (forward pass), measures the error, then goes through each
  layer in reverse to measure the error contribution from each
  connection (reverse pass), and finally slightly tweaks the connection
  weights to reduce the error.

However I am not sure how this differs from the reverse-mode autodiff implementation by TensorFlow. As far as I know the above algorithm first goes through the graph in the forward direction and then in the second pass computes all partial derivatives for the outputs with respect to the inputs. This is very similar to the propagation algorithm.

How does backpropagation differ from reverse-mode autodiff ?


texts: dividing Mean by standard Deviation meaning

texts:
I have played around with logistic regression a little using movement data intervals that are prelabeled as either resting or active.
I now found that if I divide the mean movement of the individual intervals by the intervals standard deviation, the outcome is quite a good predictor of whether the interval is a resting interval or or active, with an average auc = 0.93 in a 20 fold cross validation.
Does someone have an idea of what I have created dividing the mean by the standard deviation? Its like a flipped coefficient of variation, or semi-normalization?!.
I want to report this in an essay, so I am asking myself whether there is a name to this statistic.


texts:
When i look at the available datasets in https://www.openml.org i often see a BNG dataset with no further information about it.

Can someone explane what BNG means in this context?

I am especially interested in this dataset: https://www.openml.org/d/1389

Has anyone more information about where this data set comes from?


texts: How to calculate time difference in between rows using loop in panda python

texts:
I have a CSV file with columns date, time. I want to calculate row-by-row the time difference time_diff in the time column. I wrote the following code but it's incorrect.
Here is my code and at bottom, my CSV file:

#def time_diff(x):

date_array = []
date_array.append(pd.to_datetime(data['date'][0]).date())
start = []
end = []
temp_date  = pd.to_datetime(data['date'][0]).date()
start.append(pd.to_datetime(data['time'][0], format='%H:%M:%S').time())
for i in range(len(data['date'])):
    cur_date = pd.to_datetime(data['date'][i]).date()
    if( cur_date > temp_date):
        end.append(pd.to_datetime(data['time'][i-1], format='%H:%M:%S').time())
        start.append(pd.to_datetime(data['time'][i], format='%H:%M:%S').time())
        date_array.append(cur_date)
        temp_date = cur_date
end.append(pd.to_datetime(data['time'][len(data['date'])-1], format='%H:%M:%S').time())
if start <= end:
    return end - start
else:
    end += timedelta(1) # +day
    assert end > start
    return end - start

for i in range(len(date_array)):
    s_time = datetime.datetime.combine(date_array[i],start_time[i])
    e_time = datetime.datetime.combine(date_array[i],  end_time[i])
    timediff = (e_time - s_time)

My CSV file:

First column and second column are date and time. Third column shows expected time_diff in hours.


texts:
I will evaluate classification models I made. That's logistic regression and decision tress.
1. What I use standards for comparison?
2. Suppose model selection's standard is ASE. One is high ASE of training data, low ASE of test data, and the other is ASE of training data is low and ASE of test data is high.. If you select a model, which models do you choose?


texts:
As presented in the first article of Google Wavenet (https://arxiv.org/pdf/1609.03499.pdf) the model can approximate the joint probability of the whole sequence (raw audio waveform) using the chain rule. They implement this using some stacks of dilated-causal convolution.
However in 2017 they presented a new article for neural audio synthesis of musical notes (https://arxiv.org/pdf/1704.01279.pdf).
In this article the formula of the joint probability for the sequence(using chain rule) is different from the previous article.
I believe that the chain rule in the first article is correct but I can't understand the same rule in the second one.
For completeness I report the formula from the article:

I believe it should be given x1...x(i-1).

Is it correct?


texts: How to make a Power Spectral Density Plot in R

texts:
I have a time series point process representing neuron spikes. I have computed and plotted autocovariance using acf but now I need to plot the Power Spectral Density.

Power Spectral Density is defined as the Fourier Transform of the autocovariance, so I have calculated this from my data, but I do not understand how to turn it into a frequency vs amplitude plot.

I have used the following code

# X is some set of Wait times between spikes, below is just an example
X <- c(56, 3, 4, 119, 3, 4, 121, 3, 3, 121, 3, 4, 120, 3, 4, 4, 115)
acf <- acf(X,type="covariance")
psd <- fft(acf$acf)

Now psd is a complex valued array across the default 24 lags from the acf function.

How do I turn this array into a PSD plot?


texts: Solutions for Continuous Online Cluster Identification?

texts:
Let me show you an example of a hypothetical online clustering application:

At time n points 1,2,3,4 are allocated to the blue cluster A and points b,5,6,7 are allocated to the red cluster B.

At time n+1 a new point a is introduced which is assigned to the blue cluster A but also causes the point b to be assigned to the blue cluster A as well.

In the end points 1,2,3,4,a,b belong to A and points 5,6,7 to B. To me this seems reasonable.

What seems simple at first glance is actually a bit tricky - to maintain identifiers across time steps. Let me try to make this point clear with a more borderline example:

The green point will cause two blue and two red points to be merged into one cluster which I arbitrarily decided to color blue - mind this is already my human heuristical thinking at work!

A computer to make this decision will have to use rules. For example when points are merged into a cluster then the identity of the cluster is determined by the majority. In this case we would face a draw - both blue and red might be valid choices for the new (here blue colored) cluster.

Imagine a fifth red point close to the green one. Then the majority would be red (3 red vs 2 blue) so red would be a good choice for the new cluster - but this would contradict the even clearer choice of red for the rightmost cluster as those have been red and probably should stay that way.

I find it fishy to think about this. At the end of the day I guess there are no perfect rules for this - rather heuristics optimizing some stability criterea.

This finally leads to my questions:

Does this "problem" have a name that it can be referred to?
Are there "standard" solutions to this and ...
... is there maybe even an R package for that?

Reasonable Inheritance of Cluster Identities in Repetitive Clustering


texts: How to set and check normal distribution on a data set?

texts:
Sorry if my question is simple, I have a data set with two class and want to check and set normal distribution on it(if it was necessary, in MATLAB).

But the question is that I should use it on every rows or every column, it's 200 rows and 30 columns.


texts: MCMC algorithm -- understanding some paremeters

texts:
I am trying to understand an MCMC program. I manage to run it, but I am trying to understand the meaning of the some parameters in the analysis.
The code is something like this
#Nsamples
nsamp   = 50000

#Burn-in
skip    = 300

#temperature at which to sample
temp    = 2

#Gelman-Rubin for convergence
GRstop  = 0.01

#every number of steps check the GR-criteria
checkGR = 500

#1 if single cpu , otherwise is giving by the nproc-> mpi -np #
chainno = 0

This is for the  metropolis-hastings algorithm.
How my results depend on these parameters or what these parameters mean ?
I understood the nsamp and skip, I also kind of know what GRstop and checkGR means but I have no idea what temp and chainno means. How does changing these parameters can effect the analysis ?


texts: Apparent Paradox between Principle of Maximum Entropy and Perplexity

texts:
As I found the principle of maximum entropy said:

  the probability distribution which best represents the current state of knowledge is the one with the largest entropy

And the definition of perplexity said:

  perplexity is a measurement of how well a probability distribution or probability model predicts a sample. It may be used to compare probability models. A low perplexity indicates the probability distribution is good at predicting the sample.

As perplexity is $2^{H(p)}$, the question is "does higher entropy show the better probability distribution or not?". If it is, it means high perplexity is better. how should we resolve the paradox?


texts: predict future value in every one hour using (t+60 minutes) LSTM neural network in python

texts:
I have a data csv file including with three inputs and two output with time series. Here data took an every one hour one hour. So I need to predict my next future value at t+60 according to the previous input value and at that time period if having new input value using regression neural network. So I choose LSTM neural network to predict next future value. But I don't know how to give time period to predict my future value.
Can anyone suggest me how to solve this problem?
Can anyone give me any examples to clear out this problem?
Here that prediction value will come as input value (g).

subset of my csv file

here I upload my code;

def create_data(data, look_back=1):
dataX, dataY = [], []
for i in range(len(data) - look_back - 1):
    a = data[i:(i + look_back), :]
    dataX.append(a)
    dataY.append(data[i + look_back, 2])
return numpy.array(dataX), numpy.array(dataY)

data = pd.DataFrame(data,columns=['g','p','c'])
numpy.random.seed(7)
data = data.values
scaler = MinMaxScaler(feature_range=(0, 1))
data = scaler.fit_transform(data)
train_size = int(len(data) * 0.67)
test_size = len(data) - train_size
train, test = data[0:train_size, :], data[train_size:len(data), :]
# reshape into X=t and Y=t+1
look_back = 3
trainX, trainY = create_data(train, look_back)
testX, testY = create_data(test, look_back)
# reshape input to be  [samples, time steps, features]
trainX = numpy.reshape(trainX, (trainX.shape[0], look_back, 3))
testX = numpy.reshape(testX, (testX.shape[0],look_back, 3))
model = Sequential()
model.add(LSTM(6, input_shape=(look_back,3)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
history= model.fit(trainX, trainY,validation_split=0.33, nb_epoch=10,
batch_size=30)
trainPredict = model.predict(trainX)
testPredict = model.predict(testX)
trainPredict_extended = numpy.zeros((len(trainPredict),3))
trainPredict_extended[:,2] = trainPredict[:,0]
trainPredict = scaler.inverse_transform(trainPredict_extended) [:,2]
print(trainPredict)
testPredict_extended = numpy.zeros((len(testPredict),3))
testPredict_extended[:,2] = testPredict[:,0]
testPredict = scaler.inverse_transform(testPredict_extended)[:,2]
trainY_extended = numpy.zeros((len(trainY),3))
trainY_extended[:,2]=trainY
trainY=scaler.inverse_transform(trainY_extended)[:,2]
testY_extended = numpy.zeros((len(testY),3))
testY_extended[:,2]=testY
testY=scaler.inverse_transform(testY_extended)[:,2]
trainScore = math.sqrt(mean_squared_error(trainY, trainPredict))
print('Train Score: %.2f RMSE' % (trainScore))
testScore = math.sqrt(mean_squared_error(testY, testPredict))
print('Test Score: %.2f RMSE' % (testScore))
# shift train predictions for plotting
trainPredictPlot = numpy.empty_like(data)
trainPredictPlot[:, :] = numpy.nan
trainPredictPlot[look_back:len(trainPredict)+look_back, 2] = trainPredict

# shift test predictions for plotting
testPredictPlot = numpy.empty_like(data)
testPredictPlot[:, :] = numpy.nan
testPredictPlot[len(trainPredict)+(look_back*2)+1:len(data)-1, 2] =
testPredict


texts: Sci-kit Pipeline and GridsearchCV returns indexError: too many indices for array

texts:
I'm trying to get to grips with sci-kit learn for some simple machine learning projects but I'm coming unstuck with Pipelines and wonder what I've done wrong...

I'm trying to work through a tutorial on Kaggle

Here's my code:

import pandas as pd

train = pd.read_csv(local path to training data) train_labels =
pd.read_csv(local path to labels)

from sklearn.decomposition import PCA
from sklearn.svm import LinearSVC
from sklearn.grid_search import GridSearchCV

pca = PCA()
clf = LinearSVC()

n_components = arange(1, 39)
loss = ['l1','l2']
penalty = ['l1','l2']
C = arange(0, 1, .1)
whiten = [True, False]

from sklearn.pipeline import Pipeline

#set up pipeline
pipe = Pipeline(steps=[('pca', pca), ('clf', clf)])

#set up GridsearchCV
estimator = GridSearchCV(pipe, dict(pca__n_components = n_components, pca__whiten = whiten, clf__loss = loss, clf__penalty = penalty, clf__C = C))

> estimator

Returns:

GridSearchCV(cv=None,
       estimator=Pipeline(steps=[('pca', PCA(copy=True, n_components=None, whiten=False)), ('clf', LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,
     intercept_scaling=1, loss='l2', multi_class='ovr', penalty='l2',
     random_state=None, tol=0.0001, verbose=0))]),
       fit_params={}, iid=True, loss_func=None, n_jobs=1,
       param_grid={'clf__penalty': ['l1', 'l2'], 'clf__loss': ['l1', 'l2'], 'clf__C': array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9]), 'pca__n_components': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38]), 'pca__whiten': [True, False]},
       pre_dispatch='2*n_jobs', refit=True, score_func=None, scoring=None,
       verbose=0)

But when I try to train data:

estimator.fit(train, train_labels)
The error is:

    428         for test_fold_idx, per_label_splits in enumerate(zip(*per_label_cvs)):
    429             for label, (_, test_split) in zip(unique_labels, per_label_splits):
--> 430                 label_test_folds = test_folds[y == label]
    431                 # the test split can be too big because we used
    432                 # KFold(max(c, self.n_folds), self.n_folds) instead of

IndexError: too many indices for array

Can anyone point me in the right direction?


texts: I have a dataset of transactional data with customer ID and I want to segment the dataset into groups using cluster analysis. I'm interested in following the evolution of each cluster over time, but since customers have very different behaviours (roughly 50% of the time a customer will change cluster the week after), I was wondering what would be a statistically sound approach. Is it a good idea to train a clustering algorithm every week and look backwards at the weekly evolution of each segment?


texts: Approaches to choosing number of bins in histogram

texts:
Choosing the number of bins in a histogram has always been something that gets me thinking a lot. Based on the number of bins chosen, the graph at time looks a lot different and also could lead to different interpretations.

Below is the square-root rule, which I use as the thumb-rule for selecting the number of bins in most occasions.

Posting this question here to hear other opinions.

data_pts = len(np.array(data))
bin_cnt = int(np.sqrt(data_pts))

plt.hist(data, bins=bin_cnt)


texts: How do we interpret the outputs of DBSCAN clustering?

texts:
I am starting to learn DBSCAN for clustering but the interpretation part of it seems to be tricky to understand.

dataset = np.vstack((quotient_times, quotient)).T

scaler = StandardScaler()
dataset = scaler.fit_transform(dataset)

db_scan = DBSCAN(eps=0.6, min_samples=1)
db_scan.fit(dataset)

colors = [int(i % 23) for i in db_scan.labels_]

plt.figure();
plt.title(fname)
plt.xlabel("time")
plt.ylabel("quotient")
plt.scatter(dataset[:, 0], dataset[:, 1], c=colors)
plt.legend()
plt.show()

This produces the following plot.

When i print the values of db_scan.labels_, I am getting

array([0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1])

I was expective my y-axis values from 0-1 and x-axis values from 1-7. Why do I have the negative scales both on the x-axis and y-axis? and how do we interpret these values and the plot?


texts: Are month and weekday rulers of surveys? How can I test the validity?

texts:
I am analyzing a survey and came across an interesting pattern.
Weekday and month seems to have a very high say in the score given by customers.
There can be many reasons for this, though it is unexpected, but how can I test whether it is significant?

From the charts Tuesday is by far the worst day to ask the customers, and February is the worst month.
Combined it goes crazy…don’t ask the customer anything on Tuesdays in February..
How can I statistically back up or reject these hypothesis?


texts:
I am referring this previously asked question in stack-overflow which remains unsolved till now.

I am facing same problem with pytorch when I am solving multiclass classification.

Here is my exception

  RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at c:\programdata\miniconda3\conda-bld\pytorch_1532501501514\work\aten\src\thnn\generic/ClassNLLCriterion.c:93

I have added whole code of mine with the dataset I am trying with but still get stuck on that

# -*- coding: utf-8 -*-
"""
Created on Thu Jul 11 22:41:07 2019

@author: Ananda Mohon Ghosh
"""

import pandas as pd
import numpy as np
import os
import torch
from torch import nn
from torch.autograd import Variable
import torch.utils.data as utils
from sklearn import preprocessing
from sklearn.metrics import mean_squared_error, mean_absolute_error
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torchvision import datasets, transforms, models
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.classificaion =nn.Sequential(
            nn.Linear(16, 12),
            nn.LogSoftmax(dim=1)
        )

    def forward(self, x):
        clsf = self.classificaion(x)
        return clsf

def oneHotCode(y):
    y = y-min(y)
    nOfClasses = len(set(y))
    labels = y.reshape(len(y), 1)
    return (labels == torch.arange(nOfClasses).reshape(1, nOfClasses)).long()

def batchData(xBatch, yBatch):
    return np.concat((xBatch, yBatch),axis=1)

def findMinMax(df):
    #find max-min
    minimum = 999999
    maximum =  -999999

    for di in  df:
        tMin = min(di)
        tMax = max(di)
        if(tMin<minimum):
            minimum = tMin
        if(tMax>maximum):
            maximum = tMax
    return maximum, minimum

dataFrame = pd.read_csv('data/features_test_encoded16_ds1_batch_sig.csv')
dataFrameValues = dataFrame.values

availabelCuda = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")
print('Availabel CUDA ', availabelCuda)

#variables
num_epochs = 10
batch_size = 128
learning_rate = 1e-3
weight_decay = 1e-5
momentum = 0.9

clsf = Net()
model = clsf.to(availabelCuda)

lossFunc= nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate,  weight_decay=weight_decay)

observations =  dataFrameValues.shape[0]
featureSetLength = dataFrameValues.shape[1]
featureSet = dataFrameValues[0:len(dataFrameValues), 0:featureSetLength-1 ]
maximum, minimum =findMinMax(featureSet)

featureSet = (featureSet-minimum)/(maximum-minimum)

labelSet = dataFrameValues[0:len(dataFrameValues), featureSetLength-1 ]

Xtrain = featureSet[0:int(observations*0.7), :]
Xtest = featureSet[int(observations*0.7):observations, :]
Ytrain = labelSet[0:int(observations*0.7)]
Ytest = labelSet[int(observations*0.7):observations]

X = torch.Tensor(Xtrain)
Y = torch.Tensor(Ytrain.reshape(-1,1))
trainDataset = utils.TensorDataset(X, Y)

dataloaderTrain = utils.DataLoader(trainDataset, batch_size=batch_size, shuffle=False)

print('Model running on ', availabelCuda)

for epoch in range(num_epochs):
    for xX, yY in dataloaderTrain:
        optimizer.zero_grad()
        dataX = Variable(xX).to(availabelCuda)
        dataY = Variable(yY.long()).to(availabelCuda)

        outclass = clsf(dataX)
        print(outclass.type(), dataY.view(-1).type())

        loss = lossFunc(outclass, dataY.view(-1))
        optimizer.zero_grad()               # clear gradients for this training step
        loss.backward()                     # backpropagation, compute gradients
        optimizer.step()
    print('Epoch: ', epoch, '| train loss: %.4f' % loss.data.numpy()) #CPU

and here is the link of my dataset. Actually, I was hoping that I have problem with data normalization but eventually I couldn't solve it and still have this issue. Any kind of help is appreciated. Thanks.


texts: What's the best way to rank aggregate imdb rating data?

texts:
I have an average rating of all votes as well as the total number of votes for all episodes of the TV show Always Sunny. Is it mathematically sound to?:

1) multiply each average rating by the total votes cast for that episode

2) take the array of all those values and normalize it so the lowest is 0 and the highest is 10

3) use that new value as the actual rating


texts: How to use machine learning to extract product info from the titles of eBay listings

texts:
Reposting here because someone correctly pointed out it is better suited for here.

So I have a bunch of titles of eBay listings. I want to extract product information from each title, so I can categorize the listings by product since these titles are coming from wide-ranging searches.

The are the following Titles:

5 PCS 3 Digits 0.56" NUMERIC DISPLAY 7 seg segment Common cathode
5 PCS LD-3361AS 3 Digits 0.36" NUMERIC DISPLAY 7 SEG COMMON CATHODE digit
6 x MAN71A Common Anode Red Led 7-Segment Display (6 pcs)
Experiment Board Breadboard Circuit Board ZYJ - 60 White NEW

The format in which I want to extract the data is as follows:

For the 1st listing:

There are 5 pieces
It has 3 digits
It is of size .56"
It is 7 segments
It is numeric
It is cathode
It is a display

For the 2nd listing:

There are 5 pieces
It is of size .36"
It is model LD-3361AS
It has 3 Digits
It is Numeric
It has 7 segments
It is cathode
It is a display

For the 3rd listing:

There are 6 pieces
It is brand MAN71A
it has 7 segments
It is Anode
It is a display

For the 4th listing:

It is New
It is a Breadboard
It is White
It is model ZYJ - 60

The reason I want to learn machine learning to do this is that it would make it very easy to just take those specifics and store it for later use and because it can adapt to any search and not just in a specific category. It would also be able to adapt to different items or formats in the title. I tried this with scikitlearn's kmeans, but it gave me a lot of overlap and just clustered them, rather than extract the details from it. What I want to do is have the program look at the title, identify specifics as what they refer to (i.e., size, color, brand, etc.) and extract the data. Ideally, this would be unsupervised learning, but as I write this, I start to realize that may be impossible to do with this. The part I'm stuck on is what to use to achieve this. Should I use NLTK? One of the classifiers from scikitlearn? Clustering?


texts: How important is the input data for a ML model?

texts:
Last 4-6 weeks, I have been learning and working for the first time on ML. Reading blogs, articles, documentations, etc. and practising. Have asked lot of questions here on Stack Overflow as well.

While I have got some amount of hands-on experience, but still got a very basic doubt (confusion) --
When I take my input data set with 1000 records, the model prediction accuracy is say 75%. When I keep 50000 records, the model accuracy is 65%.

1) Does that mean the model responds completely based on the i/p data being fed into?

2) If #1 is true, then in real-world where we don't have control on input data, how will the model work?

Ex. For suggesting products to a customer, the input data to the model would be the past customer buying experiences. As the quantity of input data increases, the prediction accuracy will increase or decrease?

Please let me know if I need to add further details to my question.

Thanks.

Edit - 1 - Below added frequency distribution of my input data:

Edit - 2 - Adding Confusion matrix and Classification report:


texts: CV (Curriculum Vitae) Recommender System using Machine Learning, Python, Apache Solr(Back-end), AngularJS (Front-end)

texts:
I am very new to Machine Learning. This is my college project. I want to develop web application of CV RECOMMENDER SYSTEM in Python. I have lots of CVs in the format of .txt.

My questions are following:

What type of learning I can apply? (Supervised/Unsupervised)
I have different file in one folder. So Is there need to make a corpus? If there is a need to make a corpus then can I create a corpus?

I am very confused with the flow of the application.

Can please anyone suggest me the steps how can I start to develop web application?

Your help will be appreciated.


texts: Why does putting a 1/2 in front of the squared error make the math easier?

texts:
Per wiki, the mean squared error (MSE) looks like:

$$
\operatorname {MSE} ={\frac {1}{m}}\sum _{i=1}^{m}(y_{i}-{\hat y_{i}})^{2}
$$

The professor added a $1\over2$ in front of the formula and explained it a little bit. I am a little bit confused. How does putting a $1\over2$ in front of the squared error make the math easier?


texts: Preparing dataframe to carry k-means clustering

texts:
Im trying to apply 3 different algorithms of clustering on my dataset.
to check which one fits the best. I'm confused how should I convert my dataframe

-k-means
-DBSCAN
-hierarchical clustering

Dataset (bank.csv)
https://archive.ics.uci.edu/ml/datasets/Bank+Marketing
(bank.csv with 10% of the examples and 17 inputs, randomly selected from
3 (older version of this dataset with less inputs))

https://archive.ics.uci.edu/ml/machine-learning-databases/00222/
(direct download)

'data.frame':   4521 obs. of  17 variables:
 $ age      : int  30 33 35 30 59 35 36 39 41 43 ...
 $ job      : chr  "unemployed" "services" "management" "management" ...
 $ marital  : chr  "married" "married" "single" "married" ...
 $ education: chr  "primary" "secondary" "tertiary" "tertiary" ...
 $ default  : chr  "no" "no" "no" "no" ...
 $ balance  : int  1787 4789 1350 1476 0 747 307 147 221 -88 ...
 $ housing  : chr  "no" "yes" "yes" "yes" ...
 $ loan     : chr  "no" "yes" "no" "yes" ...
 $ contact  : chr  "cellular" "cellular" "cellular" "unknown" ...
 $ day      : int  19 11 16 3 5 23 14 6 14 17 ...
 $ month    : chr  "oct" "may" "apr" "jun" ...
 $ duration : int  79 220 185 199 226 141 341 151 57 313 ...
 $ campaign : int  1 1 1 4 1 2 1 2 2 1 ...
 $ pdays    : int  -1 339 330 -1 -1 176 330 -1 -1 147 ...
 $ previous : int  0 4 1 0 0 3 2 0 0 2 ...
 $ poutcome : chr  "unknown" "failure" "failure" "unknown" ...
 $ y        : chr  "no" "no" "no" "no" ...

I know that my data for clustering should be converted to integers. So i converted it like this (badly):

#Import data to R

bank <- read.table(file="bank.csv", sep=";", header=TRUE,stringsAsFactors = FALSE)

# character data into numeric format
bank$job <- as.numeric(as.factor(bank$job))
bank$marital <- as.numeric(as.factor(bank$marital))
bank$education <- as.numeric(as.factor(bank$education))
bank$default<- ifelse(bank$default == "yes", 1, 0)
bank$housing <- ifelse(bank$housing== "yes", 1, 0)
bank$loan<- ifelse(bank$loan== "yes", 1, 0)
bank$month <- as.numeric(as.factor(bank$month))
bank$contact <- as.numeric(as.factor(bank$contact))
bank$poutcome <- as.numeric(as.factor(bank$poutcome))
bank$y <- ifelse(bank$y== "yes", 1, 0)

# create normalization function
normalize <- function(x) {
  return ((x - min(x)) / (max(x) - min(x)))
}
#normalize the data to get rid of outliers if present in the data set
bank <- as.data.frame(lapply(bank, normalize))


texts: Shouldn't training and validation loss be approximate before the first epoch is finished?

texts: I'm having this burning question in my head, and I couldn't find the answer anywhere. During training, at least in Keras, the training loss is computed on the current batch, so the weights can be updated. So, at least at the first epoch, every batch loss is computed before the model actually learns from that particular epoch. Given this, shouldn't the validation loss for the first epoch be somewhat near the training loss for the first epoch, since both are computed on examples not seen by the gradient descent algorithm? For every model I've built so far, the training loss is always lower than the validtion loss. I expected the training loss to be somewhat near the validation loss on the first run through the dataset, and then, as the model learns from the training dataset (but not from the validation dataset), those errors would start to create a gap between them. Am I missing something trivial here?


texts: If I get 95+ % accuracy in normal models, should I still consider Ensemble models? Why should I choose Ensemble models over normal models?


texts:
I have a set of data that I'm trying to generate a z-score with. I know I need standard deviation as part of my calculations. I am using the formula of:
$\sigma = \sqrt{p * n * (1-p)}$

My data is binary - the value can either go up or down. However, through historical data, I know that it is more likely to go up than down.

So, which value for $ p $ should I use above? Do you use the binary value of $0.50$ or should I use the historical value of my data going up?


texts:
I've been playing with Google's MLKit, and decided to detect push ups.
As a quick test, I took the position of the left shoulder, and plotted the Y Axis. Here's how a variety of trials look:

Five pushups

6 pushups, slightly farther away from the phone

No pushups at all

6 pushups, far aware and with a different orientation
Good news
If I look at these graphs, I can visually intuit pretty well how many pushups someone does.
Challenge
I am not quite sure how I could detect that algorithmically.
I thought about using something like "slope changes", but with just that I would be detecting a lot of noise too. I thought about using a fourier transform, but from some basic playing, I wasn't able to clearly differentiate the "pushup" signal from the rest.
The two big problems I think are:

a. The "pushup" waves can differ in both width and height, based on how close the user is to the phone
b. I can't rely on just slopes to figure this out, and need some sort of "wave" detector

Current Plan
Currently, I'm thinking about doing some special heuristics:

Maybe I can use other info, like:

coordinates of the hips (to make sure they are within the same latitude as the shoulder)
X coordinate info

I could use the above two pieces of information, to "remove" the bad areas

(any place that has a big movement in X coordinates, or hip is not aligned, can't be a pushup), than maybe I could rely on something like a "significant" slope change.

Question
Right now I am in the position where there's a lot of stuff I don't know I don't know. What do you think of my "current plan?". Would you approach it differently? Is there some method you would look into?
Current exploration so far: https://github.com/stopachka/pushups-data-exploration/blob/main/exploration.ipynb


texts: Nyström approximation of the non-linear mapping $\phi$ for a RBF kernel - what is the impact of weak duality?

texts:
For SVM, it is better to solve the problem in the primal for very large data-sets. However, the non-linear mapping $\phi$ for a RBF kernel is not explicit. Approximation methods for $\phi$ like the Nyström methods are then used.

I am currently working on a classification problem where there is no strong duality (duality gap between the primal and the dual). I have already tested a linear kernel (there $\phi$ is completely explicit: $\phi$(x) = x). I would however be very interested to also test a RBF kernel and then solve the problem in the primal.

=> Can a method like the Nyström method still be applied in my case?


texts: Sample size equation for multi-class distribution

texts:
I have a large (k>15) number of potential classes involved in a text classification problem, and don't know the true distribution of these classes in the prediction set. Moreover, I currently do not know any of the true labels.

The plan is to manually tag a random subset of the N data points (N>200,000) to approximate the true distribution, but I am unsure how to determine the number of points (n<N) that need to be tagged.

Cochran’s Sample Size Formula allows calculation of a recommended sample size, given a desired level of precision, desired confidence level, and the estimated proportion of the attribute present in the population. However, I am unsure whether this is only meant for a binary classification (as are the examples I have read), and/or how to extend it to fit my situation.

Can anyone refer me to an example of calculating sample size to determine a multi-class distribution? The method need not be Cochran's.


texts: Recommended data cleaning techniques for multivariate time series prediction?

texts:
I have to predict the next step(s) in a multivariate time series with about 30 features and 50.000 samples. I am thinking of using LSTM. Which techniques are usually recommended for cleaning the data when using LSTM?

Does it make sense to transform the data into a stationary time series when using LSTM? Should the data be normally distributed when you are using PCA?

There is also a very large amount of missing timestamps. Does it make sense to impute/fill (by forward filling or something else) big gaps or is it just better in that case to ignore the missing data completely?


texts: How to understand the definition of Greedy Target-based Statistics in the CatBoost paper

texts:
There is a method named Target statistics to deal with categorical features in the catboost paper. I still some confusion about the mathematical form. Could you some guys to expain how to compute it!
$$ \hat{x}^i_k = \frac{\sum^{p-1}_{j=1}[x_{\sigma_{j},k}=x_{\sigma_p,k}]Y_{\sigma_j}+a\cdot P}{\sum^{p-1}_{j=1}[x_{\sigma_{j},k}=x_{\sigma_p,k}]+a}$$


texts: Why don't we transpose $\delta^{l+1}$ in back propagation?

texts:
Using this neural network as an example:

The weight matrices are then
$$ W_0=[2\times4], W_1=[4\times4], W_2=[4\times2]$$
To find the error for the last layer, we use
$$ \delta^{[2]} = \nabla C \odot \sigma'(z^{[2]})$$
which makes sense. This will produce a $[1\times 2]$ vector. But to find the error in the next layer, we use
$$ \delta^{[1]} = (W_2^T\delta^{[2]})\odot \sigma'(z^{[1]}) $$
This appears to try to multiply a $[4\times2]$ matrix and a $[1\times 2]$ matrix together, which is illegal. Am I just wrong about how the layers are represented? Should $z^{[n]}$ really be a $[l\times 1]$ vector? That doesn't really make sense to me, because it would be multiplied by an $[l\times m]$ matrix as the feed-forward continues. Do we just always represent $\delta^{[n]}$ as a $[l\times 1]$ vector, and the formula doesn't mention this as it's common knowledge?
What am I missing here?
(Note: these formulae are based on this book)


texts: Changing the shape of the input layer in tensorflow

texts:
I have a convolutional network which consists of multiple CNN layers (which could work with any size) and a fully connected layer at the end which can only work with fixed input size.

I have trained the network for certain image size and the fully connected layer can only work for those input sizes.

I don't want to retrain the entire network when operating on different input sizes. Is there any way to achieve training of only the newly added fully connected layer.

My code for inference looks like this

def inference(images, reuse=False, trainable=True):
    coarse1_conv = conv2d('coarse1', images, [11, 11, 3, 96], [96], [1, 4, 4, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse1 = tf.nn.max_pool(coarse1_conv, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='VALID', name='pool1')
    coarse2_conv = conv2d('coarse2', coarse1, [5, 5, 96, 256], [256], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse2 = tf.nn.max_pool(coarse2_conv, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool1')
    coarse3 = conv2d('coarse3', coarse2, [3, 3, 256, 384], [384], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse4 = conv2d('coarse4', coarse3, [3, 3, 384, 384], [384], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse5 = conv2d('coarse5', coarse4, [3, 3, 384, 256], [256], [1, 1, 1, 1], padding='VALID', reuse=reuse, trainable=trainable)
    coarse6 = fc('coarse6', coarse5, [6*10*256, 4096], [4096], reuse=reuse, trainable=trainable)
    coarse7 = fc('coarse7', coarse6, [4096, 4070], [4070], reuse=reuse, trainable=trainable)
    coarse7_output = tf.reshape(coarse7, [-1, 55, 74, 1])
    return coarse7_output


texts: Suppose you apply PCA on the data $x_1,...,x_6$ and find that data can be fully described using M principal components $u_1,...,u_M$. How the original data can be written in the space defined by these M principal components?


texts: How do I deal with changing values in a categorical variable when I am aggregating customer records

texts:
My requirement is to build a model to predict if a new customer will return to their website. I need to analyze what drives customer repeat for both new and returning customers. The only information given is the dates in which the customer has performed the transaction, the marketing channel through which they came, the deal which they used, their age and income, a flag which says whether they are a new customer or not. The data is for the last 6 years. A customer is regarded as a new customer if they don’t return within 24 months.

My approach is to aggregate the customer transaction records and derive the variables such as the Average number of visits, the average amount of transaction, create a flag variable which records did they return since their last visit.  My problem here is there are a lot of categorical variables as you can see (marketing channel they came through, the interface they used, their income, the deal program they used, etc) which keeps on changing for the same user in different transactions. How do I deal with those variables?

Any small hint at least is appreciated as I am stuck here completely.
Thanks in advance


texts:
I am reading the paper Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings here is the pdf.

On page 6, we read:

Step 1: Identify gender subspace. Inputs: word sets W , defining sets D_1 , ..., D_m.

However, they paper before and after this statement does not mention what these defining sets are? Can anyone give me a definition or description of these sets?

Thank you.


texts: Sanity check: low PPV but high AUC scores?

texts:
I have two algorithms running on a piece of data, both of which perform differently.

One of them (call it A) consistently gets a positive predictive value of about 0.75-0.78. Looking at the AUC of the Receiver operating characteristic it has a score of about 0.82

The second algorithm (let's call it B) consistently get's a lower predictive value of about 0.72-0.75. However this gets a higher AUC of the Receiver operating characteristic value of about 0.85.

Does this definitively indicate an error of some kind as AUC is so tightly associated with the positive predictive value? Or is this entirely reasonable, subject to other factors?


texts: Ideal difference in the training accuracy and testing accuracy

texts: In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference of 5% between the accuracy of training and testing set okay? Or does it signify overfitting?


texts: Stackoverflow API Structure data storage

texts:
I'm downloading Stackoverflow questions & answers for a specific tag using Stackoverflow API and Python.

The goal is to perform document clustering to find relevant terms across the documents and find similarity among them.

Example:

curl --compressed -H "Accept-Encoding: gzip" -X GET 'http://api.stackexchange.com/2.2/questions?page_number=1&pagesize=25&order=desc&sort=activity&tagged=amp-html&site=stackoverflow&run=true'

This is the procedure I'm using with Python 2.7 requests library:

Request Question  via API
Create Question object and assign API response to body property after using BeautifulSoap.
Find associated answers in Question.
Request Answers via API
Create Answer array
Assign Answer array to parent Question object.
For each Answer object assign API response to body property after using BeautifulSoap

API question request:

curl --compressed -H "Accept-Encoding: gzip" -X GET 'https://api.stackexchange.com/2.1/questions/37745529?site=stackoverflow&filter=withbody'

API response example:

{"items":[{"tags":["html","input","amp-html"],"owner":{"reputation":314,"user_id":5426326,"user_type":"registered","accept_rate":39,"profile_image":"https://www.gravatar.com/avatar/cbff9d2f96733be04cb022f8a724ad49?s=128&d=identicon&r=PG&f=1","display_name":"Pranav Bilurkar","link":"http://stackoverflow.com/users/5426326/pranav-bilurkar"},"is_answered":true,"view_count":47,"accepted_answer_id":37787460,"answer_count":1,"score":1,"last_activity_date":1465850038,"creation_date":1465553340,"last_edit_date":1465797429,"question_id":37745529,"link":"http://stackoverflow.com/questions/37745529/input-tags-elements-in-amp-html","title":"Input Tags/Elements in AMP-html","body":"
we are implementing our current site in AMP version and i am new to this. I have below queries regarding AMP -
\n\n
\n
How to get user input from user in AMP-HTML.
\n
In the above mentioned site [desktop version] we have comment section at the bottom of the page. We need to implement the same functionality in AMP.
\n
Are there any websites which are build and developed in AMP? If yes, i need links to check.
\n
\n\n
Any suggestion and ideas related to above queries would be appreciated.\nThanks.
\n"}],"has_more":false,"quota_max":300,"quota_remaining":298}

Since questions and answers contain HTML, URLs, special characters, quotes, single quotes, commas, etc in body property. I'm looking to convert this text into structured data which can be represented as a single text chunk and be analyzed as tokens and then I use TF-IDF.

Details:

2 Objects: Question and Answers
Question object has an array of answers objects as properties.
Each Question and Answer objects contain a body property which is a single string containing each text.

What's the recommended way to store this information?


texts: Choice of objective function for log transformed data for feedforward NN

texts:
I am processing some data using a feedforward neural network in Keras. I have noticed that if I log transform the data, the model trains better, however the error metric on the transformed data doesn't necessarily reflect very well what's going on with the original dataset.

At the moment I am using Mean Average Percentage Error metric (mape), and a CV error of 3.8% on log transformed data set corresponds to about 25% when I reverse-transform the predictions and check against the original outputs.

Is there a better choice of cost/objective function for situations like this?


texts: Understanding Reinforcement Learning with Neural Net (Q-learning)

texts:
I am trying to understand reinforcement learning and markov decision processes (MDP) in the case where a neural net is being used as the function approximator.

I'm having difficulty with the relationship between the MDP where the environment is explored in a probabilistic manner, how this maps back to learning parameters and how the final solution/policies are found.

Am I correct to assume that in the case of Q-learning, the neural-network essentially acts as a function approximator for q-value itself so many steps in the future? How does this map to updating parameters via backpropagation or other methods?

Also, once the network has learned how to predict the future reward, how does this fit in with the system in terms of actually making decisions? I am assuming that the final system would not probabilistically make state transitions.

Thanks


texts: Do I need to encode samples during inference?

texts:
I recently started saving (pickling) my fitted encoders. The thinking was that I would need them to encode previously unseen samples during inference.

Encode training features and labels.
Train model on encoded data.
Export fitted encoders alongside model.
New/ unseen data arrives.
Encode features of new data.
Run model prediction.
Decode prediction output.

Is this mandatory?


texts:
I am trying to create an ANN with as many as 50 independent variables. Based on my intuition and business logic, I have tried to create many combination of input variables to feed the ANN, retrain it and try to predict. But the results are not as expected and although the model make very good test set predictions it fails when new data is fed. Can someone plz guide where I am doing wrong? My immediate thought is I need to select the input variables based in a better way. Statistically is there a way to decide that?

What information can I provide about the model or results which would help to solve this?


texts: Is it acceptable not to transform() test data after train data is being fit_transform()-ed

texts:
We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, we apply transform() testing data so that the learned parameters from scaling the train data are applied on testing data. Similar to this:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

The question is: Does it also make sense to perform fit_transform() on the training data but NOT transform() testing data at all so that we get to test the model performance on actual real-world data that are not transformed at all?

Thank you


texts:
I have two sets of topics obtained from two different sets of news paper articles.

In other words, Cluster_1 = ${x_1, x_2, ..., x_n}$ includes the main topics of 'X' news paper set and Cluster_2 = ${y_1, y_2, ..., y_n}$ includes the main topics of 'Y' news paper set.

Now I want to find clusters in the two sets that are similar/related by considering the cluster attributes as given in the example below.

Example 1,
**X1 in Cluster_1** is mostly similar/related to **Y2 in Cluster_2**
**X2 in Cluster_1** is mostly similar/related to **Yn in cluster_2**
and so on.

Example 2:
News about Yet in Cluster_1 is mostly similar/related to News about Science in Cluster_2
News about Floods in Cluster_1 is mostly similar/related to News about Rains in Cluster_2

Since, I am dealing with two separate sets of clusters, what would be a suitable measurement/method I can use to connect the clusters in the two different sets?


texts: How to extract audio features for each video frame using pyAudioAnalysis

texts:
I'm trying something like extracting audio features for each video frame. I know there are 30 video frames and 16000 audio frames per second in the video file. I'm using pyAudioAnalysis python lib to achieve the goal but no success.
Here is my code.
from __future__ import print_function
from pyAudioAnalysis import audioBasicIO
from pyAudioAnalysis import ShortTermFeatures,MidTermFeatures
import matplotlib.pyplot as plt
import os,shlex, subprocess
import pandas as pd
import numpy as np

# -acodec pcm_s16le  -vn -ar 16000
command_line = "ffmpeg -i test.mp4 -ac 1 -ar 16000  -vn test_mono.wav"
#command_line = "ffmpeg -i test.mp4 -ab 160k -ac 2 -ar 44100 -vn test_stereo.wav"
args = shlex.split(command_line)
print(args)
processResult = subprocess.call(args) # Success!
print(processResult)
[SamplingRate, signals] = audioBasicIO.read_audio_file("test_mono.wav")
print(SamplingRate)

Output

['ffmpeg', '-i', 'test.mp4', '-ac', '1', '-ar', '16000', '-vn', 'test_mono.wav']
1
16000

#mid_feature_extraction(signal, sampling_rate, mid_window, mid_step,short_window, short_step):
MidFeatures,ShortFeatures,MidFeatureLabels=MidTermFeatures.mid_feature_extraction(signals, SamplingRate, 0.043*SamplingRate,
                                                                                0.043*SamplingRate,0.00016*SamplingRate,
                                                                                0.00016*SamplingRate)
print('Mid Features Extr Success')
MidFeatures_Dataframe = pd.DataFrame(data=MidFeatures.transpose(), columns=MidFeatureLabels)
print(type(MidFeatures))
#print(MidFeatures)
print(type(ShortFeatures))
#print(ShortFeatures)
print(type(MidFeatureLabels))
#print(MidFeatureLabels)
MidFeatures_Dataframe.to_csv('Audio_MidFeatures.csv')
print('Mid Features File Success')

Output

Mid Features Extr Success
<class 'numpy.ndarray'>
<class 'numpy.ndarray'>
<class 'list'>
Mid Features File Success

As per my calculations I should get 338 rows of audio features, and after a long time of struggle I'm getting 326 with the above parameters but still don't know how. If anyone can help me how, window and steps are working here. I know the basic concepts of window and step as work in CNN but not getting in this context.


texts: Error in using sklearn's GridSearchCV on Word2Vec

texts:
I am using the sklearn_api of gensim to create an estimator for a Word2vec model to pass it to sklearn's gridsearch . My code is as follows :
from gensim.sklearn_api import W2VTransformer
from sklearn.model_selection import GridSearchCV

s_obj = W2VTransformer(size=100,min_count=1,window=5)

parameters = {'size':(100,150,200),'min_count':(1,2,4),'alpha':(0.025,0.015)}

s_model = GridSearchCV(s_obj,parameters,cv=2)
s_model.fit(sentences)

print(s_model.best_params_)

Running the above code, I get the following error:
"If no scoring is specified, the estimator passed should have a 'score' method. The estimator W2VTransformer(alpha=0.025, batch_words=10000, cbow_mean=1,
               hashfxn=<built-in function hash>, hs=0, iter=5,
               max_vocab_size=None, min_alpha=0.0001, min_count=1, negative=5,
               null_word=0, sample=0.001, seed=1, sg=0, size=100,
               sorted_vocab=1, trim_rule=None, window=5, workers=3) does not."

I do not know how to resolve this. I tried using scoring='accuracy' or scoring='hamming' but they don't seem to work either.
Can someone please help me get rid of this error?


texts: Logistic Regression performs better on longer texts

texts:
I trained the LogisticRegression model with TF-IDF (both birgams and unigrams) and while predicting class it revealed that in longer texts (up to 3000 symbols)it works better that if I use short (+-100 symbols) texts. I assume that the reason is that bigram weights "work better" on longer texts, but it doesn't help me to understand why.
So any advice what to read to clarify this situation is welcome


texts: How should I tackle this real-life hypermarket problem?

texts:
I registered myself in the payback program of the hypermarket I am going to.

For every 2$ I get 1 point.
I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...).
I visit only in weekdays.

I would like to maximize the amount of points I gather, while I also maximize the times I visit that hypermarket (so buying all the stuff at once is an awful solution).

How should I go about modeling this real-life situation in terms of a Data Science problem in the first place? If you feel like it, suggest an approach of tackling it as well...


texts:
I have time series data and more or less 200 features for each sample, I used a recurrent neural network for the binary classification task.
After the classification I would like to know which features contribute most to one of the target(let's say target=1). Any suggested method? Thank you


texts:
I am new to NNs and I have a question about the convolutional layers in CNN.
A convolution layer is said to perform feature extraction or work as a feature extractor in CNN. The first convolutional layer learns to extract low-level features which means that the first layer will convert the original image into multiple copies of the original image (depending on the number of filters used) that contain only the low-level features, neglecting any other features in the original image, right ?
How does the next layers extract other features high level features i.e. faces if they were already neglected by the first layer ?
Why don't we have several separate layers, each of which does a specific extraction of some features and once all the needed has been extracted, the output is merged before the fully connected layer ?
I don't get the point of applying different filters (after the first layer)on the entry already filtered.


texts: how to determine percentage below which a feature can be removed from a model

texts:
Let feature $feat$ contain one value $A$ that occurs 5% of the time, while 95% of the time it is empty.

Instead of arbitrary saying features that have less than 5% should not be included into the model, is there a way to quantify mathematically this percentage?


texts: Meaning of "TRUE" column in R RandomForest output for Importance()?

texts:
I want to assess the importance of variables in my model using the Importance() function of R RandomForest package. I have a binary response variable / class and binary feature values.

mytree.rf <- randomForest(class ~ ., data=mydata, ntree=1500,keep.forest=FALSE,importance=TRUE)
importance(mytree.rf)

The output matrix contains the MeanDecreaseAccuracy and MeanDecreaseGini. I understand those two.

My problem is with two other columns in the output. One simply says "TRUE" the other one "FALSE". Neither in the documentation nor online I was able to find an answer what those values are and how they are calculated...

Can anyone help me out?

Cheers

Edit 1: Thanks to Davids answer I realized TRUE and FALSE are my class "names". I still don't understand how the value given in the matrix is calculated though... Can anyone help with that?

Edit 2: Thanks to David again, it turns out the answer is in the documentation. But it cannot be found in the chapter about the importance() function, but rather in the description of objects of class randomForest. Importance is one of these objects.

[...] a matrix with nclass + 2 (for classification) or two (for regression) columns.
For classification, the first nclass columns are the class-specific measures computed
as mean descrease in accuracy. The nclass + 1st column is the mean
descrease in accuracy over all classes. The last column is the mean decrease
in Gini index.


texts:
I've fully trained the VGG16 model on my dataset, resulting in a 97% validation accuracy. I'm using the code from this github: https://github.com/PowerOfCreation/keras-grad-cam but for 2 of my 16 classes, I'm getting all negative gradients, resulting in a heatmap with all null values. The other classes generate normal consistent maps which clear similar patterns.

The problem is obviously occurring here because the 0 is always the max value.

    cam = np.maximum(cam, 0)
    heatmap = cam / np.max(cam)

The only solution I've found so far is rescaling cam to (-1,1). It gives me a map for those 2 classes, without significantly altering the maps for the other 14 classes.

Here is my model:

vgg16 = VGG16(include_top = False, weights = None,input_shape =(INPUT_SIZE,INPUT_SIZE,3))
last_layer = vgg16.output
last_layer = Flatten(name = 'flatten')(last_layer)
last_layer = Dense(NUM_CLASSES, activation = 'softmax')(last_layer)
model = Model(input = vgg16.input,output = last_layer)

I had additional dense layers but removed them trying to solve this, but since the last conv layer is the last layer of vgg16, it doesn't make a difference. I've tried running the gradcam from the original training script and by loading the saved model from '.h5'

Is this right? Are all the gradients for those classes negative? Is rescaling an acceptable solution? I don't know where the error is coming from, I suspect it's from this area, but I'm not sure how to troubleshoot.

    # conv_output =  [l for l in model.layers if l.name is layer_name][0].output
    conv_output = model.get_layer(layer_name).output
    grads = normalize(_compute_gradients(loss, [conv_output])[0])
    gradient_function = K.function([model.input], [conv_output, grads])

    output, grads_val = gradient_function([image])
    output, grads_val = output[0, :], grads_val[0, :, :, :]


texts: Always getting value one for a binary classifier

texts:
I'm using keras. I have one classification problem. The output should be either 0 or 1. I trained my model and I'm getting 86.59 accuracy. But when i check the predicted output what I'm seeing is all ones. I tried creating a categorical classifier with two nodes and tried the same. The test accuracy is 86.59% but when I check the output the prediction contains only one node with value one for the entire dataset.

This is the code

from keras.models import Sequential

model = Sequential()
from keras.utils import to_categorical
y_traine = to_categorical(y_train)
y_teste = to_categorical(y_test)

from keras.layers import Dense
x_train = x_train.reshape(1300,64)
model.add(Dense(units=64, activation='relu', input_dim=64))
model.add(Dense(units=2, activation='softmax'))
model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
# x_train and y_train are Numpy arrays --just like in the Scikit-Learn API.
model.fit(x_train, y_traine, epochs=50, batch_size=32)
x_test = x_test.reshape(len(x_test),64)
loss_and_metrics = model.evaluate(x_test, y_teste, batch_size=128)
classes = model.predict(x_test, batch_size=128)
print (loss_and_metrics)
print (classes)

output

[0.32096896952051834, 0.875968992248062]

[[0.84422934 0.1557707 ]
 [0.8332991  0.16670085]
 [0.86778754 0.13221247]
 [0.9261704  0.07382962]
 [0.85143256 0.14856751]
 .
 .

What I'm doing wrong here? Why I'm getting training accuracy as 86% if my predictions are wrong?


texts: Can I use cosine similarity as a distance metric in a KNN algorithm

texts: Most discussions of KNN mention Euclidean,Manhattan and Hamming distances, but they dont mention cosine similarity metric. Is there a reason for this?


texts: Is there any way to read Xlsx file in pyspark?Also want to read strings of column from each columnName

texts:
pd is a panda module is one way of reading excel but its not available in my cluster. I want to read excel without pd module. Code1 and Code2 are two implementations i want in pyspark.

Code 1: Reading Excel

pdf = pd.read_excel(Name.xlsx)
sparkDF = sqlContext.createDataFrame(pdf)
df = sparkDF.rdd.map(list)
type(df)

Want to implement without pandas module

Code 2: gets list of strings from column colname in dataframe df

stringsList = []
columnList = list(df[colname])
for i in range(len(columnList)):
    if type(columnList[i]) != float:
        text = columnList[i]
        stringsList.append(text.lower())
    else:
        stringsList.append(u'')
return stringsList

I want to implement this in pyspark.


texts:
I have a large data set of RGB satellite data that classifies 64x64 pixel images with a spatial resolution of 10m per pixel into 10 classes (e.g. highway, industrial, river, forest). Now I want to train a model (e.g. using resnet34 with fast.ai) on it. So far my model has a error rate of 4% on the test data.

Now I want to classify chunks of an unseen larger scene. My approach is to iterate over the new image and create n x n pixel subimages that get classified and create a stitched 10-color image out of the classified n x n images.

From a visual standpoint I get good results down to about n=32. Below that the predictions are just wrong. I assume that below that mark the model cannot recognize shapes in the chunks.

My question as a novice in data science are:

Is it even legitimate to classify chunks of n<64 considering that the model was trained on n=64 images
Is there a way to classify my unseen picture with a model built on the dataset on a per-pixel basis (i.e. n=1) or do I have to find a different dataset / create my own dataset?


texts: Validation fluctuating on keras training

texts:
My model is showing a very large variation in validation.
In generating the model used, 25% of the data as a test and 75% as a validation. The data was also divided.

Is there a way to reduce / eliminate this variation?


texts: Predicted and true values distributions comparison

texts:
Is this alarming when a distribution of predicted values differs from a distribution of true values? I use xgbregressor and get the following plots

Usage of boxcox doesn't improve the case.
My data is spatial-temporal. I make a cash-flow forecasting for some city and time is treated like 12 features corresponding to months that I feed to xgb.
 The figure shows data for one year.


texts: Is there a corpus of toy datasets specifically designed for finding bugs in data science software?

texts:
I'm looking for a corpus of toy tabular datasets that can be used to test data profiling, machine learning, data manipulation, etc. software. Some example attributes:

Strange column names (empty string, long names, duplicate names, names with spaces, periods, syntax, escaped delimiters and tokens)
Non-rectangular
Mixed scientific notation in floats, inf literals
Row-empty or column-empty
Mixed file encodings
Numeric and string values designed to overflow memory buffers/cause truncation/rounding to int
Ambiguous and invalid dates
Diacritics, emojis

I was going to build a corpus myself, but surely there is some prior work here?


texts:
Dictonary:

{(8758148.0, 'CI Alpine Growth Equity Fund'): D    0.000016
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI American Growth Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI American Small Companies Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI American Value Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Equity Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Investment Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Small Cap Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Small/Mid Cap Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Global Bond RSP Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Harbour Fund'): D    0.0
 Name: 8758148.0, dtype: float64 }

I need the dataframe to be:

id          Name                                Value
8758148.0   CI Alpine Growth Equity Fund        0.000016

I am stuck in issue where I need to convert list into such a data frame with certain name of the columns


texts: Is random forest a kind of spatial feature encoding?

texts:
From the book "Deep Learning and Convolutional Neural Networks for Medical Image Computing"

  On the other hand, CNN models have been proved to have much higher
  modeling capacity, compared to the previous image recognition
  mainstream pipelines, e.g., HAAR, SIFT, HOG image features followed by
  spatial feature encoding, then random forest or support vector
  classifiers. Given millions of parameters to fit during model training
  (much more than previous pipelines), CNN representation empowers and
  enables computerized image recognition models, with a good possibility
  to be able to handle more challenging imaging problems. The primary
  risk is overfitting since model capacity is generally high in deep
  learning but often very limited datasets are available (that are with
  good quality of labels to facilitate supervised training). The core
  topics of this book are represented by examples on how to address this
  task-critical overfitting issue with deep learning model selection,
  dataset resampling and balancing, and the proper quantitative
  evaluation protocols or setups.

It seems that then indicates random forest is a kind of spatial feature encoding. Is this the case?


texts: Adaboost - Show that adjusting weights brings error of current iteration to 0.5

texts:
I'm trying to solve the following problem but I've gotten sort of stuck.

So for adaboost, $err_t = \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i}$

and $\alpha_t = \frac{1}{2}ln(\frac{1-err_t}{err_t})$

Weights for the next iteration are $w_i' = w_i exp(-\alpha_t t^{(i)} h_t(x^{(i)}))$
and this assumes $t$ and $h_t$ takes on a value of either $-1$ or $+1$.

I have to show that the error with respect to the new weights $w_i'$ is $\frac{1}{2}$.
i.e., $err_t' = \frac{\sum_{i=1}^{N}w_i' \Pi (h_t(x^{(i)}) \neq t^{(i)})}{\sum_{i=1}^{N}w_i'} = \frac{1}{2}$

i.e., we use the weak learner of iteration t and evaluate it according to the new weights, which will be used to learn the $t+1$-st weak learner.

I simplified it so that $w_i'=w_i \sqrt{\frac{err_t}{1-err_t}}$ if $w_i$ was correctly classified and $w_i'=w_i \sqrt{\frac{1-err_t}{err_t}}$ if $w_i$ was incorrectly classified. I then tried plugging this into the equation for $err_t'=\frac{1}{2}$ and got $\frac{err_t}{1-err_t} \frac{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) = t^{(i)})}{\sum_{i=1}^{N}w_i \Pi (h_t(x^{(i)}) \neq t^{(i)})} = 1$ but at this point I sort of ran into a dead end and so I'm wondering how one might show the original question.

Thanks for any help!


texts: Why is my Celcius to Kelvin Model performing so bad compared to my Celcius to Fahrenheit Model

texts:
I am new to tensorflow and I saw an example model which was made as following -
l0 = tf.keras.layers.Dense(units=1, input_shape=[1])
model = tf.keras.Sequential([l0])
model.compile(loss = 'mean_squared_error', optimizer = tf.keras.optimizers.Adam(0.1))
history = model.fit(sampl, sampl2, epochs=500, verbose=False)

Now when trained on significantly less data -
celsius_q    = np.array([-40, -10,  0,  8, 15, 22,  38],  dtype=float)
fahrenheit_a = np.array([-40,  14, 32, 46, 59, 72, 100],  dtype=float)

It gave a very close prediction for this data but when I used the following data which is significantly more for celsius to kelvin I get predictions which are very off
sampl = np.random.uniform(low=-50, high=50, size=(200,))
sampl2 = sampl+273

I get the prediction 267 Kelvin for 22 Celsius.
I increased my epochs to 1000 and now the prediction is bang on
The codes are the same for both so why do I see such disparity?


texts: how can I generate a Bernoulli block mixture model in matlab?

texts:
I am trying to write the code of a Bernoulli block mixture model in matlab, but am facing an error every time I run the function. In particular, I'm having a problem with how to relate the distribution parameter $\alpha$ to the latent variables $Z$ and $W$.

In other words, \mathbf{Z} follows a multinomial distribution of parameter $\pi$, such that $\dim(\pi)=(1, g)$, where $g$ is the number of clusters of rows and $\sum \pi=1$.

$W$ follows a multinomial distribution of parameter $\rho$ such that $\dim(\rho)=(1,m)$, where $m$ is the number of column clusters and $\sum \rho=1$.

$\mathbf{Z}$ is a matrix of $\dim(N,g)$, and $\mathbf{W}$ is a matrix of $\dim(d,m)$, where $N$ is the number of observations, $p$ is the number of variables, $g$ is the number of row clusters, and $m$ is the number of column clusters.

$X$ follows a bernoulli distribution of parameter $\alpha_{ZW}$, where the notation $\alpha_{ZW}$ denotes the values of $\alpha$ depends on $\mathbf{Z}$ and $\mathbf{W}$, and $\alpha$ is a $\dim(g,m)$ matrix. Please find attached a graphical representation of the model.

How do I write the code of $\alpha_{ZW}$ in order to get generate the model?


texts:
from XGBoost tutorial, it described:

In this equation $w_j$ are independent with respect to each other, the form $G_j w_j + \frac{1}{2}(H_j+λ)w_j^2$ is quadratic and the best $w_j$ for a given structure $q(x)$
and the best objective reduction we can get is:
$w^∗_j = \frac{−G_j}{H_j+λ}$
$obj^∗=\frac{−1}{2}\sum_{j=1}^T\frac{G^2_j}{H_j+λ} + γT$

So my confusion is:
if we would like to minimize the $obj$ function, clearly, the best way is to set:
$G_j w_j + \frac{1}{2}(H_j + λ) w_j^2 = 0$
thus we would have the best $w_j^*$ is:
$w_j^* = \frac{-2G_j}{H_j + λ}$
which is different with the official explanation.
anyone please help to point out where am I wrong?


texts: Time series - is it necessary to retrain the model when new time series data is present

texts:
Say you're building a sales prediction model to predict tomorrow's sales value, as well as the next 2 weeks of daily sales. The model is being trained using daily data for the previous 1.5 years, and it follows a strong weekly seasonality pattern.

Obviously once you are happy with model performance, would it be necessary to re-train the model everyday to capture data up to and including yesterday in order to get the most accurate prediction for tomorrow's sales value? Essentially, the model would be trained on a rolling 1.5 year data set to capture yesterday's sales value.

Or does it depend on the type of model being used? Whether you need to to completely re-train when new data is present

I can see why it would make sense to re-train your standard time series forecasting models (ARIMA etc.), but I can also understand more sophisticated models (Neural Networks etc.) may generalize well enough to not have to re-train everyday.

I am looking for an explanation of models where you would and wouldn't re-train when new time series data is present.


texts: Non-linear transformations input dataset for support vector machines

texts:
I have two classes (A,B) that I would like to classify using a SVM. Say that I have a class C and a function f. Can I do this:
A' =  f(A,C) = |A-C|
B' =  f(B,C) = |B-C|

and then perform the classification on A' and B' instead? In the context of my problem A and B are classes where elements are vectors. The f function measures the Mahalanobis distance of each vector with respect to the distribution imposed by C.


texts: Different Results from Pre-Trained Model Between Python vs R

texts:
I am using a pre-trained Model. Firstly, I trained the Model in python which came with 94% accuracy. Now, I am trying to reproduce the same results with same configuration of the model in R, but it is giving terrible results and not crossing the accuracy of 83%.
There is a difference in the data type of both languages with which I train the model: R uses data type Double while Python uses data type float32.
The second key difference is, I was running Python Notebook on GPU and TPU but the Notebook in R running on CPU because of Kaggle GPU memory issue.
Do the above-mentioned differences cause performance gaps? If not, then what could be the problem?


texts: Running into a shape error when implementing GradCAM with Keras

texts:
I've been trying to implement GradCAM from keras tutorial (https://keras.io/examples/vision/grad_cam/).
I have a custom model with a binary classification with 2 units in the final layer.
def get_model():
  model = keras.Sequential()
  model.add(Conv2D(filters=32, kernel_size=(3,3),input_shape=(64,64,3), activation='relu',))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu',))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu',))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(Flatten())
  model.add(Dense(128))
  model.add(Activation('relu'))
  model.add(Dropout(0.5))
  model.add(Dense(2, activation='softmax'))

return model
model = get_model()

I am able to fit and achieve a high accuracy however implementing the def make_gradcam_heatmap(img_array, model, last_conv_layer_name, classifier_layer_names): gives me a shape error as seen below.
ValueError: Input 0 of layer dense_1 is incompatible with the layer: expected axis -1 of input shape to have value 128 but received input with shape (1, 12, 12, 64)
More error details below:
<ipython-input-24-60aff2d42dcb> in make_gradcam_heatmap(img_array, model, last_conv_layer_name, classifier_layer_names)
 22         tape.watch(last_conv_layer_output)
 23         # Compute class predictions
---> 24         preds = classifier_model(last_conv_layer_output)
     25         top_pred_index = tf.argmax(preds[0])
     26         top_class_channel = preds[:, top_pred_index]

My parameters have been set as below:
last_conv_layer_name = 'conv2d_2'
classifier_layer_names = [
'dense_1',]

While image is taken from a file path and converted to an array as given from the tutorial.
img_array = get_img_array(img_path, size=img_size)
#predict on the image
preds = teacher.predict(img_array)
#print("Predicted:", preds, top=1)[0])  #decode_predictions not used since not a pretrained model.
print(preds)
print(img_array.shape)

Also the img_array is all 0s. Surely this is not right?
 [[[[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 ...
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]

Could you guys please guide me on where I am going wrong?
I am running this on Google colab with tensorflow version at 2.4.0
Second Edit
I have figured out that the array of all 0s just represents the dark patches in my image. Secondly, the variable used classifier_layer_names should contain all the layers from the last convolution layer up to the last dense layer.


texts: Recommender Model for Human Action in Income Protection

texts:
Problem Domain
I'm working on a project that involves building a model to provide recommendations on the next best step for Human supervisors to take on income protection claims.
Income protection is an area of insurance that involves providing the insured party with a proportion of their income if ever they fall ill or cannot work. It's a safety net designed to help you if you ever get sick or fall on hard times. Many of our customers get Cancer, Depression, Broken bones, etc. and avail of the insurance service, which is usually provided by their company.
On our side, we have a number of human supervisors that manage the ongoing claims. This means scheduling doctor's appointments, getting up to date medical records and assessments/diagnoses, arranging rehab, talking with the customer, etc. (the cost of all of this is also covered under their insurance). These management items are finite, categorical and described as actions.
The supervisor's job is to help the customer Return To Work (RTW) as soon as possible, both to help the customer heal/get back on their feet, and to reduce costs incurred in handling the claim for the business.
I have datasets that describe what actions each supervisor has taken for every claim for the past 20 years (time series/ sequential data), along with text data that describes the progress of the claim, and finally categorical data describing the claim and customer (type of disease, age, occupation, etc.)
What I'm trying to build is a recommender system which harnesses the data to recommend which action maximises the probability of a Return To Work for a given customer in an on going claim.
What I've Tried
Using the data described above, I have built a neural network to predict the outcome of a closed claim. The NN is a combination of LSTM and Dense layers - LSTM for the sequential action data, Dense for the categorical data and Dense for the TfIdf transformation of the text data. There are 39 possible outcomes, one of which is Return to Work, and the network performs relatively well, achieving ~70% accuracy.
I had planned to use the finished model like so: Iterate through all possible actions, and for each action append it to the claim in question to produce a new "hypothetical" action sequence. For each hypothetical sequence do a feed forward through the trained model and see which action maximises the "score" at the index of the output vector which corresponds to return to work. The action that maximises this score is then the recommendation. A simple toy example for an ongoing claim with 10 actions is shown below:
scores = []
# ongoing_claim.shape = (10, 30)
for action in potential_actions:
    hypothesis = ongoing_claim.append(action)
    # ongoing_claim.shape = (11, 30)
    hypothesis = pad_sequence(hypothesis)
    output = model.predict(hypothesis)
    scores[action] = output[index_for_return_to_work]

The recommendation would be the key corresponding to the highest value in scores.
Unfortunately this doesn't work. No matter which action, when I feed forward the new action sequences in the model the scores stay constant for Return to Work - no variation whatsoever. I think this is because simply adding a tiny new data-point considering the amount of data used to train the model does not add enough variance to perturb the output. I have tried using both sigmoid and softmax activation functions on the final layer but this didn't work either.
Question
What kind of model should I use which harnesses all of the data to then examine an ongoing claim and recommend the action which maximises the probability of a particular outcome (in this case Return to Work)?


texts: Fine tuning Conditional GANs for low data scenarios

texts:
I was wondering what the process was for fine tuning a conditional GAN. For example, say I wanted to generate pictures of an object X given a certain condition such as a sentence describing it, which is proposed and implemented in this paper. The condition would be embedded into a vector, call it c. However, say that annotated images of object X are rare, but un-annotated images of object X (images that are of object X but do not have a sentence describing it) are plentiful.

How would I go about training a conditional GAN on this data? Would I initially train the GAN with the un-annotated images of object X while setting the condition vector c to a zero vector so that the model learns to generate images of object X without conditions, then fine tune the model on the annotated images of object X, where c is the condition, so that the model learns the relationship between the given condition and the images it should generate at the end? Or is there some other way to go about this?


texts: Classification/Prediction based on Multivariate Time Series

texts:
So, I have a time series with many independent variables (X's) and an outcome variable Y (that I want to predict, think a 2 class logistic regression where output would either be 1 or a 0). Kindly see a sample below:

Timestamp       X1      X2      X3          X4          Y
1:00            1       0.5     23.5        0           0
1:01            1       0.8     18.7        0           0
1:02            0       0.9     4.5         1           0
….
1:30            1       1.9     5.5         1           1
1:31            0       1.7     4.3         0           1
…
…

Now I want to predict or rather classify Y as 0 (stable) or 1 (unstable) (Note that when Y becomes 1 it remains 1 for certain interval of time, same when it is 0)

So Y will be dependent on sequence variables (Please note that it is a time series, and not a standard regression problem where every row can be fed to an Algorithm for classification, the output here is dependent on a sequence of inputs/rows), for instance Y may become 1 when X2 starts increasing and X3 starts decreasing and so on (there are many independent variables X1…XN).

The way I was thinking in order to solve this problem was to extract say m hours of data before Y becomes 1 and do some descriptive statistics on X in order to derive new features (like mean of X1, std of X2, last change point of X4 and so on for the set of this extracted data) to convert the X’s to a single row feature vector. The outcome ‘Y’ of this single row feature vector is 1 as we have just extracted the data before Y became 1. So this way I am able to convert a time series into a standard classification/prediction problem. Similarly I can take the other class i.e. Y=0 and follow the same process.

The other approach that I thought about was to incorporate a sequence model, something like Hidden Markov Model where the hidden states might be stable (say for Y=0) and unstable (for Y=1) and then I go about emission and transition probabilities. But this HMM will be multivariate considering there are many X’s on which Y is dependent. This seems a bit complex?

Any ideas on modeling the above problem will be appreciated.


texts: Which one of these is the most efficient way to model training data for a neural network that will play a snake-like game?

texts:
I am building an AI using a neural network that will play Tron against a human player. The game consists of a board with fixed width and height where each player can move at any direction (except for the oposite direction he is facing). Each player's body increases per play and the first one to hit the border or any body loses.

For that I am building a neural network that will be trained with data that is built with games that will be played randomly (i.e. both the "human" player and the AI will do random moves for the sake of creating data for the NN). Whenever the AI wins a game, all the moves it used during this game will be appended to the training data. However, I still don't know how should I model it. I came down to two options:

Simply inputing my board as a matrix to the NN. it is 10x10 big so I would my input layer would have 100 + 2 neurons for the position of each player. Or:
Calculate the angle and distance between the two "heads" (the players position) and input it along each player's position and body size.

Which one will make my AI play better?


texts: How does an autoencoder 'fill in the blanks' in the context of a recommender system?

texts:
My understanding is that an autoencoder takes an input, produces a lower dimensional representation of the input, which should explain the original features in the dataset, and then reconstructs the dataset from this lower dimensional representation.
I can see how this is useful in the context of dimensionality reduction.
However, in the context of a recommender system, given that the 'missing ratings' are going to be present on both sides of the autoencoder - how does the autoencoder learn to predict these missing ratings?
Let us consider for example a recommender system that uses binary scores (1 if the user has interacted with an item, and 0 if they have not). Given that we have the same 1s and 0s in both the input and the output (maybe slightly different if we use some kind of dropout), how is it that the 0s are replaced by 'predicted scores'?


texts: Curve fitting through cloud of daily values using machine learning

texts: I want to plot laboratory values (SCr_v) over time and find the best fitting regression curves (see plot 1 and 2). I don't want to restrict myself to a specific model if possible. Is there a function in python which I can use to solve my problem? Or any good article that would help? I'd like to solve my problem in Python.


texts: Explanation of why Neural Networks are non convex

texts:
Why having a symmetry of values for the hidden state imply that the neural network is non convex? I could not find an intuitive answer for this yet.
Also, if we consider a Fully Connected network wtih ReLU non linearity(which is convex) then it is a composition of convex functions and ReLU is also non decreasing. Still the overall optimization problem becomes non convex.


texts: Qualitative prediction based off data containing mixed qualitative and quantitative values?

texts:
I'm trying to find a way to predict an integer value based off of an item's prior sale history and am inquiring for a starting point on how to approach this.

The factors are as follows (Data Type in Parenthesis):

A) ReportDate (DateTime)

B) Part# (String)

C) Part Type  (String)

D) Lost Sales in past 150 days (Int)

E) Sales in past 150 days  (Int)

F) Quantity currently in stock (Int)

G) Quantity to be Purchased  (Int)

My goal is to minimize Factor D so that it closes in at 0 as Factor G has better inputs. It should also account for what is already in stock aka Factor F

Factor G is manually inputted data while the other data is automatically generated. Factor G's value is based on the item's sale history.

There is a monthly reporting of each item spanning over the past 5 years.

For now, I am just concerned with being able to predict one item correctly.

So if I had 5 years of data from 2013-2018, that would be 60 rows of data.

It is now January 2019, so I should expect a row of data with Factor G missing, this is where I would need to predict a value.

Also, is it possible to account for report dates since Lost 150 and Sales in past 150 are dependent on the past 5 rows?


texts: Agglomerative Hierarchial Clustering in python using DTW distance

texts:
I am new to both data science and python.
I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW) is a useful method to find alignments between two time series which may vary in time or speed.

I have found dtw_std in mlpy library and scipy.cluster.hierarchy in SciPy in order to cluster my data.

From the scipy docs, I find that I could use my custom distance function:

  metric : str or function, optional The distance metric to use in the
  case that y is a collection of observation vectors; ignored otherwise.
  See the pdist function for a list of valid distance metrics. A custom
  distance function can also be used.

But I am stuck matching this information to implement clustering.

My dataset is in the format of dataframe which each row corresponds to a sample.

Here is my questions:

1- How can I provide distance matrics for the linkage function?

2- How to set my custom distance function?

import pandas as pd
import scipy.cluster.hierarchy as hac
import mlpy

dataset = pd.read_csv ( "dataset.csv",encoding='utf-8' )
X # distance matrics
Z = hac.linkage(X, metrics=mlpy.dtw_std, method='average')
cluster = hac.fcluster(Z, t=100, criterion='maxclust')
leader = hac.leaders(Z, t=100, criterion='maxclust')

fig = plt.figure(figsize=(25, 10))
dn = dendrogram(Z)
plt.show()

edit:

Here is how I compute distance matrix, then I pass it to linkage:

# computing distance matrix
dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))
z = hac.linkage(dm, method='average')
cluster = hac.fcluster(z, t=100, criterion='maxclust')
leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='maxclust')


texts: What's the strategy for deciding which feature level is excluded from one hot encoding of a categorical variable?

texts:
I'm working on a regression problem with a continuous dependent variable (sale price of a home). Amongst my features are several categorical features, which I'm transforming to "one hot encoded" dummy variables via Pandas get_dummies function.

My question is, in order to avoid the trap of multi-colinearity, one level from each categorical variable must be excluded from the model.Pandas provides this with the drop_first argument within the get_dummies function in which I can optionally exclude the first level from being included in the one hot encoding. However, this strikes me as somewhat arbitrary. Is there a strategy to ensure that the excluded level doesn't affect the accuracy of my regression model? How does one decide which level to exclude?


texts:
I do at the moment some data experiments with the Graphlab toolkit. I have at the first next SFrame, with the three columns:

Users Items Rating

The pair in the same row from every Users and Items values build the unique key and the Rating is the corresponded float value. These values are not normalised. First of all, I do someself next normalisation:

Division of every rating value of specific user by the rating maximum from this user (scale between 0 and 1)
Take the logarithm by every rating value

Afterward I create a recommender model and evaluate the basic metrics for it.

In this topic I invite everybody to discuss another interesting normalisation methods. If anybody could tell some good method for data preparation, it would be great. The results could be evaluated because of the metrics and I can publish it here.

PS

My dataset is comming from some music site, the users rated some tracks. I have approximately 100 000 users and 300 000 tracks. Total number of ratings is over 3 millions (actually the matrix is sparse). This is the most simple data set, which I analyze now. In the future I can (and will) use some additional information about the users and tracks (f.e. duration, year, genre, band etc). At the moment I just interest to collect some methods for rating normalisation without to use additional information (users & items features). My problem is, the data set doesn't have any Rating at the first. I create someself the column Rating, based on the number of events for unique User-Item pair (I have this information). You can of course understand that some users can hear some tracks many times, and another users only one time. Consequently the dispersion is very high and I want to reduce it (normalise the ratings value).


texts: What is the difference between fasttext and DANs in document classification?

texts:
I came across two interesting papers that describe promising approaches for document classification using word embedding.

1. The fasttext algorithm

Described in the paper Bag of Tricks for Efficient Text Classification here.

(With further explanation here).

2. DANs

Described in the paper Deep Unordered Composition Rivals Syntactic Methods for Text Classification here.

Question:

What is the difference between both approaches?

Are they essentially the same as they both seem to average word embedding and pass it through an MLP or am I missing something crucial?


texts: Is there any formal explanation for the sensitivity of AdaBoost to outliers?

texts: AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.


texts: Masters thesis topics in Applied probability and Probabilistic models in Machine Learning

texts:
I'm looking for a topic for my masters thesis. Machine learning is my primary domain and I want to work on probabilistic models and applied probability in Machine Learning. Please suggest some exciting new topics that would make for a good masters thesis subject.
Anything related to Markov chains Monte Carlo, Bayesian methods, Probabilistic graphical models, Markov models and so on in context of machine learning would be great!