texts:
I learn machine learning from sciki and read its documents.

Clustering cluster groups based on the euclidean distance and filter them by different ways ex: guassian distribution, or mean-shift...etc.

But none of clustering algo cluster samples based on the variation ratio

EX: below every items has there sold numbers of everyday.

Item,D1,D2
A,1,5
B,10,50
C,4,70

The variation ratio below:
A:500%
B:500%
C:1750%

So A and B are the same group, C isn't.

Any cluster algo can cluster time series datasets based on variation ratio(or quantity)?


texts: What are the best ideas or approaches to trade off between bias and variance in Machine Learning models.


texts: Help me understand the pytorch translation of this mathemetical equation

texts:
I am looking at the second problem of the homework solution from the course unsupervised learning here.
Can someone explain how does the author convert this equation:

$p_\theta(x) = \sum_{i=1}^4 \pi_i[\sigma((x+0.5 - \mu_i)/s_i) - \sigma((x-0.5-\mu_i)/s_i)]$

Into the following piece of code in Pytorch ( I am having trouble understanding the forward function).
Specific questions that. I have are :

Why is log_space which represents $s_i$ from the equations, a log value who's exp we have to calculate instead of a simple vector of 4
Why do we need to calculate log_cdf_delta and why do we need to use torch.clamp
Why do we need to calculate x_log_probs and why are we selecting different values based on x < 0.001

class MixtureOfLogistics(nn.Module):
  def __init__(self, d, n_mix=4):
    super().__init__()
    self.d = d
    self.n_mix = n_mix

    self.logits = nn.Parameter(torch.zeros(n_mix), requires_grad=True)
    self.means = nn.Parameter(torch.arange(n_mix).float() / (n_mix - 1) * d,
                              requires_grad=True)
    self.log_scales = nn.Parameter(torch.randn(n_mix), requires_grad=True)

  def forward(self, x):
    d = self.d
    x = x.float()
    x = x.unsqueeze(1).repeat(1, self.n_mix) # b x n_mix
    means, log_scales = self.means.unsqueeze(0), self.log_scales.unsqueeze(0) # 1 x n_mix
    inv_scales = torch.exp(-log_scales)

    plus_in = inv_scales * (x + 0.5 - means)
    min_in = inv_scales * (x - 0.5 - means)

    cdf_plus = torch.sigmoid(plus_in) # CDF of logistics at x + 0.5
    cdf_min = torch.sigmoid(min_in) # CDF of logistics at x - 0.5

    cdf_delta = cdf_plus - cdf_min # probability of x in bin [x - 0.5, x + 0.5]
    log_cdf_delta = torch.log(torch.clamp(cdf_delta, min=1e-12))
    log_cdf_plus = torch.log(torch.clamp(torch.sigmoid(inv_scales * (0.5 - means)),
                                                       min=1e-12))
    log_cdf_min = torch.log(torch.clamp(1 - torch.sigmoid(inv_scales * (d - 1.5 - means)),
                                        min=1e-12))

    x_log_probs = torch.where(x < 0.001, log_cdf_plus,
                              torch.where(x > d - 1 - 1e-3,
                                          log_cdf_min, log_cdf_delta))
    pi_log_probs = F.log_softmax(self.logits, dim=0).unsqueeze(0)
    log_probs = x_log_probs + pi_log_probs
    return torch.logsumexp(log_probs, dim=1)

  def loss(self, x):
    return -torch.mean(self(x))

  def get_distribution(self):
    with torch.no_grad():
      x = torch.FloatTensor(np.arange(self.d)).cuda()
      distribution = self(x).exp()
    return distribution.detach().cpu().numpy()


texts:
Has documentation of the GPT-3 API been made public?
I would be interested in keeping myself up to speed on the API's capability.


texts: How to handle features with very broad range

texts:
I have a long list of continuous values like in the image below:

The plot looks like this:

How to handle such features? If I train the model with this, the model will not have the best precision, because there are a lot of outliners.


texts: How to handle a zero factor in Naive Bayes Classifier calculation?

texts:
If I have a training data set and I train a Naive Bayes Classifier on it and I have an attribute value which has probability zero. How do I handle this if I later want to predict the classification on new data? The problem is, if there is a zero in the calculation the whole product becomes zero, no matter how many other values I got which maybe would find another solution.

Example:

$P(x|spam=yes) = P(TimeZone = US | spam=yes) \cdot P(GeoLocation  = EU | spam = yes)  \cdot  ~ ... ~  = 0.004 $

$P(x|spam=no) = P(TimeZone = US | spam=no) \cdot P(GeoLocation  = EU | spam = no)  \cdot  ~ ... ~  = 0 $

The whole product becomes $0$ because in the training data the attribute TimeZone US is always Yes in our small training data set. How can I handle this? Should I use a bigger set of training data or is there another possibility to overcome this problem?


texts: Adjust class weights due to class imbalance and class importance Multi class classification XGBoost

texts:
With respect to this question and the answer given by @Esmailian, Would anyone be able to let me know if Class B has a higher importance or the positive class ( i.e. it needs to have a higher predicted accuracy not necessarily at the expense of other classes). How should we readjust the weights of the classes. I tried to increase the weight of class in importance (Class B) but the accuracy of it reduced.

Thank you

Michael


texts: Converting Json file to Dataframe Python

texts:
I have a json file which has multiple events, each event starts with EventVersion Key. The data looks similar to the following synthesized data.

{"Records":[{"eventVersion":"1.04","userIdentity":{"type":"R","principalId":"P:i","arn":"arn:aws:sts::5","accountId":"50","accessKeyId":"AW","sessionContext":{"attributes":{"mfaAuthenticated":"f","creationDate":"2013-09"},"sessionIssuer":{"type":"R","principalId":"WA","arn":"arn:aws:iam::6","accountId":"70","userName":"user1"}}},"eventTime":"2027-6","eventSource":"a.com","eventName":"DS","awsRegion":"UZ","sourceIPAddress":"2.1.3","userAgent":"li","requestParameters":null,"responseElements":null,"requestID":"OO","eventID":"09","eventType":"ABC","apiVersion":"2010-4","recipientAccountId":"78"},{"eventVersion":"1.04","userIdentity":{"type":"R","principalId":"P:i","arn":"arn:aws:sts::5","accountId":"50","accessKeyId":"AW","sessionContext":{"attributes":{"mfaAuthenticated":"f","creationDate":"2013-09"},"sessionIssuer":{"type":"R","principalId":"WA","arn":"arn:aws:iam::6","accountId":"70","userName":"user1"}}},"eventTime":"2027-6","eventSource":"a.com","eventName":"DS","awsRegion":"UZ","sourceIPAddress":"2.1.3","userAgent":"li","requestParameters":null,"responseElements":null,"requestID":"OO","eventID":"09","eventType":"ABC","apiVersion":"2010-4","recipientAccountId":"78"}]}

I'm using the following code in Python to convert this to Pandas Dataframe such that Keys are columns and values of each event is a row.

with open('/Users/snehahonnappa/Documents/NLP_AWSlogs/Model/Data/505728423372_CloudTrail_ap-northeast-1_20160913T1700Z_yKA3wB5Nx6juR6Kg.json') as json_data:
    sample_object = json.load(json_data)

df = pd.io.json.json_normalize(sample_object)
df.columns = df.columns.map(lambda x: x.split(".")[-1])

print df.shape

When I print shape of the dataframe its 1X1. I'm expecting (Number of unique keys X Number of records)

Snippet of how I'm expecting the dataframe to be

eventVersion               userIdentity                          eventTime
             type   principalId P   arn     accountID userName
1.04         R      P           i   arn:aws 50        user1      2027-6
1.06         Q      O           i   arn:aws 67        u2         2027-7

Appreciate any help.

Update :

I'm writing the json file into a csv and then trying to convert this to dataframe on which my models can be applied on. Following is my code.

import json
import csv
import sys

data_parsed = json.loads(open('/tmp/A.json').read())
log_data = data_parsed['Records']

# open a CSV file for writing
data = open('/tmp/log.csv', 'w')

# create the csv writer object
csvwriter = csv.writer(data)
count = 0

for i in log_data:
      if count == 0:
             header = i.keys()
             csvwriter.writerow(header)
             count += 1
      csvwriter.writerow(i.values())

data.close()

This is writing the keys as headers and values of each record as a separate row which is as expected. However the nested json objects are being written as one value.

Following is a snippet of my csv file which was obtained by executing the above code.

eventVersion eventID eventTime requestParameters                eventType
1.04         0       2016-20                                    AwsApiCall
1.04         8       2016-20    {u'tagKeys': [u'User Name']}    AwsApiCall
1.05         4       2016-30    {u'filterSet': {u'items': [{u'name': u'resource-type', u'valueSet': {u'items': [{u'value': u'*'}]}}, {u'name': u'tag:User Name', u'valueSet': {u'items': [{u'value': u'*'}]}}]}}
                                                                AwsApiCall

Any suggestions to tackle this?


texts: image recognition: fully connected network vs CNN

texts:
To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training and got excellent results. So I wonder if CNNs aren't necessary for this purpose if a double layer fully connect network can do the job well?!
For example, I'd like to recognize digit '1'. I use only 4 or 5 images that resemble '1', and other 4 or 5 images that look like anything else. Every time I take~2000 pixels from each image. It turns out this code with this tiny amount of training data can recognize correct and incorrect digits really well.


texts: What is the selection criteria to choose between XGBoost and Random Forest

texts:
I am trying to understand - when would someone choose Random Forest over XGBoost and vice versa. All the articles out there highlights on the differences between both. I understand them. But when actually given a real world data set, how should we approach the problem to choose between these?

For eg: Is there a set of statistical tests for variance check, and then choose? Or is it simply like you have a number of features, and cannot really choose to do parameter tuning, so you apply Random Forest to get results?


texts:
Should kernel size always be a prime number? E.g. (3,3) (5,5) (7,7).
While tinkering with sklearn.preprocessing.KernelCenterer(), I noticed that I could only get it to work on a 3,3 array not 2,3 not 6,1.
Then I realized that in a square that is prime-by-prime, there is always a center - hence Centerer.
[ ][ ][ ]
[ ][X][ ]
[ ][ ][ ]

[ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ]
[ ][ ][X][ ][ ]
[ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ]


texts:
I am going through the deep learning book by Goodfellow. In the RNN section I am stuck with the following:

RNN is defined like following:

And the equations are :

Now the $O^{(t)}$ above is considered as unnormalized log probability. But if this is true, then the value of $O^{(t)}$ must be negative because,

Probability is always defined as a number between 0 and 1, i.e, $P\in[0,1]$, where brackets denote closed interval. And $log(P) \le 0$ on this interval. But in the equations above, nowhere this condition that $O^{(t)} \le 0$ is explicitly enforced.

What am I missing!


texts: How to properly compare these two confusion matrix?

texts:
I have used Vader, a sentiment analysis tool for social media, on a database of movie reviews. These two confusion matrices differ in the vader.py algorithm, as the first one is from nltk:

The second one is deriving from Vader's original code on github and includes fixes to negation words, etc.

I was wondering how could I properly compare the two, as I'm not really able to read them. It seems there is not a big difference between them and I don't understand what could be the sources of the errors here.


texts: How can I measure data anonymity when present summary statistics


texts: Does the test set has to be in [0,1] range?

texts:
I have standardized training set using
mean = XTrain.mean()
XTrain-=mean

std = XTrain.std()
XTrain/=std

And then used mean and std to standardize validation and test sets. The training and validation sets have values that are greater than 1 and less than zero is that okay?


texts: Dataset where svm performance is significantly different from random forest

texts:
Is there a specific dataset where svm performs significantly better or worse than random forest?

I know that the performance could depend on the dataset but is there a specific dataset?


texts:
I have two signals "Signal A" and Signal B" and I plot the autocorrelation for each signal with itself as shown below. How I can determine which signal has a stronger correlation with itself.
To calarify more, I want to answer this question : "Which signal is more correlated with itself "Signal A" or Signal B" ?


texts: Boosted tree regression loss function when data has occasionally very large values to predict?

texts:
I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious outliers that should be removed, but are values I'd like the prediction algorithm to try to capture. However, I do not want the error on these variables to dominate the training of the tree. Conceptually, the percent error is more akin to what I'm interested in (although it doesn't have to be it exactly). Specifically, when the target is 30 and I predict 15, I consider that just to be similar as when the target is 5000 but my prediction is 2500. I don't want a 2500**2 squared error to overwhelm the 15***2 squared error.

For this type of problem, what is the best way for me to tackle this issue? Data transform? Custom loss function? Etc?


texts:
I have following attributes in my data-set.
Name, Date of Birth, Annual Income, Tax Paid, Gender.
What is the type of each variable i.e. Categorical or Continuous? and why ?

PS. I believe categorical attributes are the ones having fixed set of Values. Please let me know if I am wrong with this assumption. Also the "Name" attribute is just a label. but what category does it fall in ?


texts: Encoding features for multi-class classification

texts:
I have a question regarding how to setup a dataset for modeling.
Let’s say I have a dataset representing which car a person will buy depending on some characteristics:
The dependent variables are individual cars (Car 1, Car 2, … Car 100).
The independent variables are:
Budget (of the buyer)
Favorite Color (of buyer)
…..
…..
Color (of Car 1)
Color (of Car 2)
….
Color (of Car 100)
MPG (of Car 1)
MPG (of Car 2)
…..
MPG (of Car 100)
Let’s assume this is a multi-class classification problem. So, only one of the cars can be chosen in each situation.
My question is: is it appropriate to have independent variables like that - that are specific to each of the dependent variables? (Color of Car X, MPG of Car X, …). Is it appropriate to just fit a row like that into a model? How does the model know to understand that each of the Colors are discussing the same feature? Color
Lastly, is there a name for this type of data/problem? I'm not sure how to look for it on Google.


texts: I've always relied on the Keras embedding layer for my NLP work. But for my latest project I want to use a custom embedding layer. I have gone through the steps to create a word2vec file but now what? Can someone provide an example of how I can replace the Keras embedding layer with my own layer when constructing a model?


texts: How to visualize multi-instance multiclass classification?

texts:
Let's say I have 3 classes and 1 score for each data point

score z
class 1
class 2
class 3

E.g. the input looks like:

0.529 5 7 4
0.310 3 4 2
0.774 10 7 6
0.774 10 8 5
0.172 3 0 2

In code:

>>> import pandas as pd
>>> df = pd.DataFrame([[0.529, 5.0, 7.0, 4.0], [0.31, 3.0, 4.0, 2.0], [0.774, 10.0, 7.0, 6.0], [0.774, 10.0, 8.0, 5.0], [0.172, 3.0, 0.0, 2.0]])
>>> df
       0     1    2    3
0  0.529   5.0  7.0  4.0
1  0.310   3.0  4.0  2.0
2  0.774  10.0  7.0  6.0
3  0.774  10.0  8.0  5.0
4  0.172   3.0  0.0  2.0

Each row represents a data point and column 0 is the score and column 1,2,3 are some sort of coefficients for the classes.

The score is some sort of computed entity independent of the coefficients and the goal is to visualize the interaction between the coefficients and the score.

How to visualize the interaction between the 3 classes (column 1,2,3) and the score (column 0)?


texts: Reproducing randomForest Proximity Matrix from R package in Python

texts:
I am trying to port this little piece of R code to python:
rf <- randomForest(features, proximity = T, oob.prox = T, ntree = 2000)
dists <- as.dist(1 - rf$proximity)

with parameters
oob.prox: Should proximity be calculated only on “out-of-bag” data?
proximity: if proximity=TRUE when randomForest is called, a matrix of proximity measures among the input (based on the frequency that pairs of data points are in the same terminal nodes).
I am currently trying using sklearn.ensemble.RandomTreesEmbedding for this task, however there is no functionality for the proximity matrix. I found the following developer comment though:

We don't implement proximity matrix in Scikit-Learn (yet).
However, this could be done by relying on the apply function provided
in our implementation of decision trees. That is, for all pairs of
samples in your dataset, iterate over the decision trees in the forest
(through forest.estimators_) and count the number of times they fall
in the same leaf, i.e., the number of times apply give the same node
id for both samples in the pair.

And so I tried, utilizing numpy's pdist() function along with my custom distance (or in this case, proximity) measure. I still have several problems:

The proximity function is outstandingly slow
How to handle the out-of-bag behaviour
How to recreate the exact behaviour of as.dist(1- rf$proximity): I think I need to normalize my count matrix, then subtract it from 1 and then afterwards compute the euclidean distances between its rows!?

My code as of now looks like this:
# grow a random forest from points
rf = ensemble.RandomTreesEmbedding(n_estimators=200,
    random_state=0,
    max_depth=5
)
rfdata = rf.fit_transform(xdata);

# define an affinity measure function to use with numpy's pdist
def treeprox(u, v):
    leafcount = 0
    # needs reshaping for single samples
    u = u.reshape(1,-1)
    v = v.reshape(1,-1)
    a = rf.apply(u)
    b = rf.apply(v)
    # count number of times they fall in the same leaf
    # (use of np forces element-wise)
    c = np.sum(np.array(a)==np.array(b))
    return c

distm = pdist(xdata, proxfun)
distm = squareform(distm)

There must be a better way I guess, since this functionality is readily provided by the R package randomForest.
Any suggestions?
tia


texts: Why my perceptron doesn't train well and produces bad results on test data?

texts:
I am a newbie in Machine learning and I am writing a small code for Perceptron. This is the first time I am writing code in Python.
I have four training data points (X). As they are used for supervised learning so, each data point has its corresponding correct output pair (D). I have implemented SGD and used generalized Delta rule (wij ← wij + α δixj). I have trained my perceptron 10,000 times (epochs= 10,000).
Although everything looks fine to me, I don't get the right results when I test it with test values. I need some suggestions so that I can improve my results on test data. P.S. How can I improve this code?

Code

import numpy as np

def sigmoid(x):
return 1 / (1 + np.exp(-x))

def Delta_SGD(W, X, D):
 N = 4
 for x in range(N):

    v1 = np.dot(X[x][0], W[0])
    v2 = np.dot(X[x][1], W[1])
    v3 = np.dot(X[x][2], W[2])
    #weighted sum
    V = v1+v2+v3

    #output of neuron
    y = sigmoid(V)

    #error
    e = D[x] - y

    #derivative of sigmoid(y)
    delta = (y)*(1-y)*e

    #Delta rule
    DW = alpha*delta*X[x]

    #updated weights
    W[0] = W[0] + DW[0]
    W[1] = W[1] + DW[1]
    W[2] = W[2] + DW[2]

return W

#input data points
X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])

#Correct output pairs
D = np.array([[0,0,1,1]]).T

#learning rate
alpha = 0.9

#random weights
W =  2*np.random.random((3,1)) - 1

#10000 epochs
for epoch in range(10000):
 W = Delta_SGD(W, X, D)
 print(epoch)

#Final weights after all epochs
print("Final weights are \n", W)

#testing network
N = 4
for x in range(N):

 v1 = np.dot(X[x][0], W[0])
 v2 = np.dot(X[x][1], W[1])
 v3 = np.dot(X[x][2], W[2])

 V = v1+v2+v3
 y = sigmoid(V)
 print("output of neuron is \n ", y)


texts: How do I use TF*IDF scores for my machine learning model?

texts:
I have applied TF*IDF on the 'Ad-topic line' column of my dataset.
For every ad-topic line, I get the same output:

Firstly, I am unable to make sense of the output. The TF*IDF values are mentioned to the right, but what exactly are the numbers in brackets?

I plan to use these for my logistic regression model for classification. How exactly do I feed these values to the algorithm?


texts:
I have three different histograms (Impact parameter distributions) corresponding to three groups of the same particle with different properties. However, the three distribution have more or less the same shape. Now I want to predict a fourth distribution (with another property), which should have more or less the same shape as the three other ones. I have only a part of this latter distribution in a certain range.

A first attempt was to predict the rest of the distribution using a Transfer Factor (TF), which is the ratio of integrals in two different ranges, and by multiplying this TF to the integral of the known range in the fourth distribution, I could estimate the number of events in the unknown range. I did this using the three known distribution, but I obtained three different results, which were not even close to each other. I think that one of the three must be the good one, but I don't know which one.

My question: Is there a statistical method to determine the best one? or is there another way/method to approximate my last distribution?


texts: Should I repeat the testing/training split while repeating K-fold cross validation?

texts:
I'm fitting random forest regressions on my data, and using 10 K-fold cross-validation to evaluate model performance. While re-runing the cross-validation, I noticed that the results differed between each run, sometimes by a lot. So, I decided to repeat the cross-validation calculation 20 times, creating a for-loop, and then summarising the results afterwards. Just to illustrate, I'm doing something like this:
for (i in 1:20) {

trainIndex <- createDataPartition(data$response, p = .8, list = FALSE, times = 1)
data_train <- data[ trainIndex,]
data_test  <- data[-trainIndex,]

train.control <- trainControl(method = "cv",
                              number = 10)

model <- train(formula, data = data_train, method = "rf",
               trControl = train.control)

pred <- predict(model, data_test)
t<- postResample(pred = pred, obs = data_test$estimate)
t <- t[[2]]

result[[i]] <- t }

So, essentially, each re-run I am splitting my dataset into training and testing sets again. This results in a lot of variance, when ploting the resuls of each run together they vary from ~0.2 to 0.6 R.squared. If I don't do this within the loop (i.e. if I split into training/testing before the loop), then the results of the 20 runs are very similar.
Which way is the right way to go about this?


texts: Hypothesis Testing on Prediction from Random Forest Classifier

texts:
I used a random forest classifier to predict 0,1, 0 indicating non-prevalence of a disease, and 1 indicating the disease is prevalent. I want to see if having a feature would increase the chance of zero prevalence.
H0: The prevalence of having a facility = The prevalence of not having the facility
H1: The prevalence of having a facility > The prevalence of not having a facility
If we reject H0, is it sufficient to say that having a particular facility significantly lower the risk of the disease?
stat = pd.DataFrame({'factor':[],'pval':[],'status':[]})
for column in dummies_test.columns[0:-2]:
    zero = dummies_test[dummies_test[column] == 0].result
    one = dummies_test[dummies_test[column] == 1].result
    test = stats.ttest_ind(zero, one)
    if test.pvalue<0.01:
        stat = stat.append({'factor':column, 'pval':test.pvalue,'status': 'Reject H0'},ignore_index =True)
    else:
        stat = stat.append({'factor':column, 'pval':test.pvalue,'status': 'Insufficient to Reject H0'},ignore_index =True)


texts: Different accuracy of random forest and neural net

texts:
I ran a set of time series data in  a neural net model and random forest model. For neural net I normalised the data split it into test 20%, validation and training 80%. Keras library neural net was used here.The accuracy is 85%(I get different accuracy every time).
For random forest I didn’t normalise the data and the test train split is 20% and 80%.I used scikit learn here. I am getting 93% accuracy .Here time series data is used as well. Am I not supposed to get similar accuracy as neural net? What am I doing wrong?
(I was reading some papers there they had similar kind of accuracy. That’s why I am asking. Also can random forest overfit? Is mine one overfitting?!


texts: Effect of a few wrongly scaled feature values on logistic regression model

texts:
I was given a situation to predict the validity of the logistic regression model when it was found that certain values of a heavily weighted feature were found to be erroneously multiplied by 1000. For example, the feature has values 20.000, 78.000, 56.000, 10.000 and so on. However, few values were found to be missing the decimal point and were therefore scaled up by 1000.
Will the results of the previously fit model be valid in such a situation and why ? Also, how will the model be affected by the wrongly scaled up values.


texts: Is it a acceptable way to write a loss function in this form?

texts:
I found a loss function of a perceptron on a book is in this form

$$ L(w,b) = - \sum\limits_{x_i \in M}y_i(wx_i+b) $$

In the context of machine learning loss functions, is it acceptable to put $x_i$ on the bottom of a summation notation?


texts: Multi-class classification with discrete output: Which loss function and activation to choose?

texts:
I'm working with a multi-class classification problem, using Keras Sequential models. In my dataset, the output class has one of the following values: (1, 2, 3, 4, 5).

Yes, I have already tried one-hot-encoding the output and using categorical_crossentropy as a loss function and softmax in the output layer.

However, I don't think that softmax and categorical cross entropy is the right choice in my case. In my dataset, the output classes have a certain "discreteness" (or scale). Class 1 is the "worst" and class 5 is the "best".

Let's say if on a specific input the model predicts 2 and the true class is 1, it's a much better prediction as when the model predicts class 5 and true class is 1.

I would like, that the loss function would take these "minor errors" into consideration and not treat all errors the same way.

To sum up, I'm wondering what my options are in terms of loss function and activation of the output layer for my given problem.

UPDATE: I have now tried to use mse as a loss function and replace my output layer with this:

model.add(Dense(1, activation='linear'))

The new method produces much better results, but I'm still wondering if there are any other options for my problem. Maybe a custom loss function?


texts: Should I relabel this data or remove the potentially leaky feature?

texts:
Putting together a Keras MLP to predict whether a value will exceed a static percent threshold in the next 15 minutes. The incoming data is a rolling percentage which moves smoothly for the most part because the data comes in every few milliseconds and is windowed over some fixed number of minutes. So, when the data is already above the threshold, it tends to stay there for a while.

When putting the data through the NN, it gets high accuracy, but this seems to be due to it correctly predicting that when it is currently over the threshold, it will also be over the threshold at some point (the next point) in the next x timesteps. The usefulness in the model would be if it could accurately predict before it crosses the threshold.

Features:

1) current point is over threshold - 1/0
2) current point is AM - 1/0
3) current day is weekday - 1/0
4) current percentage - 0.0-1.0
5-9) average of percentages in past 1/5/10/20/30 minute - 0.0-1.0

Label:

1 if a point is over threshold at any point in time after now and before now+15 minutes

Features 5-9 are intended to capture the inertia of the current percentage.

From feature importance, it looks like the current value is heavily used, followed by whether or not its over threshold, followed by the rolling means in order of time. I am currently changing the NN architecture and number of epochs in order to increase f1 score. Should I remove features 1 and 4, or rework the label in order to increase the accuracy of the predictive ability before its actually over the threshold?


texts: Good introductory reference for Bayesian Non-parametric (Dirichlet Process / Chinese Restaurant Process)

texts:
I am looking for a recommendation for basic introductory material on Bayesian Non-parametric methods, specifically Dirichlet Process / Chinese Restaurant Process. I am looking for material which covers the modeling part as well as the inference part from ground-up.

Most of the material I found on the internet has slightly advanced material and they skip the inference part, which is usually harder to grasp.


texts:
I have data set that has data for patients:

Arrival_Date : is when the patient has arrived

Seen_By_Nurse : is number of minutes patient take to be seen by nurse since arrival
when value is NaN it means patient is still waiting and has not been seen by Nurse yet

Seen_By_Dr: is number of minutes patient take to be seen by dr since arrival
when value is NaN it means patient is still waiting and has not been seen by Dr yet

Upset_Patient : Is this patient upset or not 1 or 0

The data look like this:

Arrival_Date   Seen_By_Nurse   Seen_By_Dr   ... many other fields ...   Upset_Patient
14/8/19 14:15  10              25                                       1
14/8/19 15:32  13              NaN                                      0
14/8/19 15:35  16              NaN                                      1
14/8/19 15:39  NaN             NaN                                      0
14/8/19 15:44  NaN             NaN                                      1

My question is: I know that working with NaN failing many predictive models. Therefore, I have to either fill it with other values, such as mean. Which is going to be wrong in my case. Or delete it which also going to be wrong in my case as the majority has not been seen by Dr or Nurse yet.

Should I fill it with big number such as 9999? Are there any other approaches?

What is the best way to deal with NaN in this case?


texts: How to extrapolate given dataset in machine learning

texts: i have given a submission dataSet with fewer rows how to extrapolate it so that model could perform better


texts: Can regression algorithms treat multiple input features as a single feature for prediction?

texts:
I have a regression problem where I have multiple RSSI values from 3 beacons and I need to predict the x & y coordinates of the mobile object sending the signals based on the RSSI values captured by the 3 beacons.
The dataset is as follows:
RSSI values from 3 beacons have been recorded 301 times. However, the timestamp has not been provided so I don't know whether RSSI values coming from 3 beacons at a single time has been recorded or not.
X & y coordinates mobile object sending the signals is given.
My problem does not arise from the scenario.
My question is can regression algorithms like multivariate regression, support vector regression can treat all 301 input features of RSSI values from the three beacons as one whole stack of feature?


texts: Using of 100s of Binary features in regression model

texts:
I have 100s of columns with binary values [0, 1] plus some extra columns without binary values. I am trying to do regression model but the model performance is very low. For non-binary features, I have used PCA to decrease the dimension of it. I don't think its appropriate to use PCA for binary values. I am guessing, its because of the large number of binary columns, the model isnt doing great. What can be done on this kind of situation ? I have tested with almost all of the regression model available in sklearn.

What approach I could take to improve the model performance ? Any suggestions.


texts: Use convolutional variational autoencoders for time series prediction

texts:
I want to use convolutional variational autoencoders for time series prediction. For example, here is the dimension of my data.
x_train = np.random.rand(100, 5, 2)
y_train    = np.random.rand(100, 2)
y_test     = np.random.rand(100,  2)
x_test    = np.random.rand(100, 5, 2)

However, all of the examples and applications of this method are for IMAGE which the dimension of the data is 4d. Could you help me with how to use convolutional variational autoencoders for the provided data set? I have seen a lot of examples, like https://github.com/ivanlen/autoencoders_safari


texts:
I have several user names and their salaries.
Now I need to cluster user based on their salaries.
I am using KMeans clustering and following is my code

from sklearn.cluster import KMeans
from sklearn.preprocessing import LabelEncoder
import pandas as pd

le = LabelEncoder()
data = pd.read_csv('kmeans.data',header=None, names =['user', 'salary'])

# Numerical conversion
data['user'] = le.fit_transform(data['user'])
km = KMeans(n_clusters=4, random_state= 10, n_init=10, max_iter=500)
km.fit(data)

data['labels'] = le.inverse_transform(data['user'])
data['cluster'] = km.labels_

print data

But my results are bad and there are lot of overlapping salaries.

Is there anything wrong in the code ? How to improve the results ?

Or whether clustering is not a right approach here ?
Then how can I cluster users only based on salary ?

 km.fit(data['salary'])

EDIT:

I figured out a way to solve my problem using numpy.reshape

km.fit(data['salary'].reshape(-1,1))


texts:
ERROR SUMMARY:

I'm getting the following error:

TypeError: Expected binary or unicode string, got [

BACKGROUND:

I have several features that are histories of user activity. I am trying to predict whether a given user will take an action (represented by a 0 or 1 in my y_train list below) based off of their histories of different types of actions. For example, one feature might be button_A_click_per_day and the history for a given user would be a vector of button A clicks per day over the past 365 days.

The relevant snippets of my code is as follow:

import tensorflow as tf

# Build feature columns for classifier
feature_columns = []
for key in X_train:
    col = tf.feature_column.numeric_column(
        key=key,
        shape=max_width,
    )
    feature_columns.append(col)

# Build classifier
classifier = tf.estimator.DNNClassifier(
    feature_columns=my_feature_columns,
    hidden_units=[10, 10],
    n_classes=2,
)

And X_train is structured as follows:

>>> X_train['<feature_name>']
129     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
...
1294    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
860     [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
Name: <feature_name>, Length: 1377, dtype: object

My train input function is as follows:

def train_input_fn(features, labels, batch_size):
    dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
    dataset = dataset.shuffle(buffer_size=1000).repeat(count=None).batch(batch_size)
    return dataset.make_one_shot_iterator().get_next()

Finally, when issuing the following command...

classifier.train(
    # input_fn is a fn that takes not arguments and returns an iterator.
    input_fn=lambda: train_input_fn(X_train, y_train, batch_size=100),
    steps=1000,
)

I get the error:

~/.pyenv/versions/anaconda3-5.0.1/lib/python3.6/site-packages/tensorflow/python/util/compat.py in as_bytes(bytes_or_text, encoding)
     63   else:
     64     raise TypeError('Expected binary or unicode string, got %r' %
---> 65                     (bytes_or_text,))
     66
     67

TypeError: Expected binary or unicode string, got [0.0, 0.0, 0...

MY QUESTIONS:

Do you know what I'm doing wrong to get this error?
Is there a better way to model features that are histories?


texts:
This is code for inexact line search:

function alpha = LineSearch(f, g, x, p)
a = 1-2/(1+sqrt(5)) ;
b = 2/(1+sqrt(5)) ;
alpha = 1.0 ;
while f(x+alpha*p) > f(x) + a*alpha*g(x)'*p
    alpha = b*alpha ;
end
end

How to implement exact line search?


texts: No graph is displaying while plotting value with time interval using python

texts:
I have a large dataset with values and date and time. So I want to plot graph of value with time and date. So I wrote a code for time difference. Afterthat I tried to store value into that time difference. Afterthat I tried to plot it. It run with no graph. Can anyone helps me to solve this error?
here is my code:

x= df1,iloc[:,2]
time_interval = 14400 #interval in seconds (14400s = 360 minutes)
date_array = []
date_array.append(pd.to_datetime(df1['date'][0]).date())
start_time = []
end_time   = []
temp_date  = pd.to_datetime(df1['date'][0]).date()
start_time.append(pd.to_datetime(df1['time'][0], format='%H:%M:%S').time())
for i in range(len(df1['date'])):
cur_date = pd.to_datetime(df1['date'][i]).date()
if( cur_date > temp_date):
    end_time.append(pd.to_datetime(df1['time'][i-1], format='%H:%M:%S').time())
    start_time.append(pd.to_datetime(df1['time'][i], format='%H:%M:%S').time())
    date_array.append(cur_date)
    temp_date = cur_date
    end_time.append(pd.to_datetime(df1['time'][len(df1['date'])-1], format='%H:%M:%S').time())
  datetime_array = []
  for i in range(len(date_array)):
s_time = datetime.datetime.combine(date_array[i],start_time[i])
e_time = datetime.datetime.combine(date_array[i],  end_time[i])

timediff = (e_time - s_time)
#num_periods = int(timediff.total_seconds()/t3ime_interval) +1
num_periods = abs(int(timediff.total_seconds()/time_interval)) + 1
time_list = pd.date_range(start=s_time, end = e_time, periods=num_periods ).to_pydatetime()
datetime_array.extend(time_list)
time_stamps = [datetime.datetime.strftime(t,'%H:%m:%S') for t in datetime_array]
x = np.zeros([num_periods], dtype='timedelta64[s]')
plt.xticks(np.arange(num_periods), time_stamps)

my csv file:

after running the code output came like this:

image:


texts: Train and validation sets splits using load_data

texts:
I'm using the package "datasets". The code I have:
squad_v2 = True
model_checkpoint = "roberta-base"
batch_size = 10

from datasets import load_dataset, load_metric
datasets = load_dataset("squad_v2")
datasets

and output:

It's working fine, but as the dataset really huge is a lack of memory on the gpu during the training model. Is there any way how to using "load_dataset" make train and validations splits smaller? But still to have them both on "datasets"?


texts: Training set, validation set, and test set with Orange

texts:
Is it possible with Orange (only using its widgets, without writing Python code) to implement the following typical machine learning processes?

Train a training set,
Validating a validation set (e.g. using k-fold cross validation), and
Testing the model with a test set.

Using the 'Test & Score" widget, a single dataset is used. Does this mean that only validation is performed, and there is no actual test? Can in this case classification results (accuracy, AUC, etc.) considered reliable?


texts: feature Extraction with mobileNet visualization

texts:
I am trying to create a logical visualization regarding how feature extraction with mobile net works in ml5.js. There's a good explanation here: https://youtu.be/kRpZ5OqUY6Y
With ml5, you use a part of the pre-trained model: the features. Those features allow you to 'retrain' or 'reuse' the model for a new custom task (transfer learning).
ml5's functions allow you to extract features of an image via a pre-trained model and re-train that model with new data. Then, we can map the features/logits to our own set of labels. I tried to visualise this according to the video:

However, this figure looks like we're not doing any training at all and just mapping the whole thing to our custom labels. What could be a good way to visualise the feature extraction process with mobile net?
Edit:


texts: How large of a value should a weight have in a neural network?

texts: If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can weights take on any value? Could a network potentially have weights of 0.1, 0.0009, and 100000?


texts: How to deal with training set that overfits very easily

texts:
I have a dataset consisting of 72 one-hot encoded (thus binary) features and 2.5K training examples. With this I am trying to solve a 10-class classification problem.

My main problem is that no matter which classification algorithm I use, the model overfits instantly to the train set while in the test all metrics seem pretty much bad, e.g. at best average f1-score around 0.18 and avg accuracy less than 20% average accuracy.

To deal with this I tried to reduce the number of features to 30 and finally to 10, simplified the models I use (mostly tree based). Also note that the split is done using stratification. Would it be a good idea to oversample ?
Note that I cannot reduce the number of target classes.

Example reports:

Random Forest

          precision  recall   f1-score   support

    a       0.05      0.06      0.05        50
    b       0.15      0.17      0.16        42
    c       0.14      0.13      0.14        60
    d       0.14      0.18      0.16        77
    e       0.12      0.14      0.13        50
    f       0.26      0.21      0.23        33
    g       0.08      0.05      0.06        41
    h       0.14      0.10      0.11        51
    i       0.00      0.00      0.00        18
    j       0.20      0.21      0.20        78
accuracy                           0.14       500
macro avg      0.13      0.12      0.13       500
weighted avg   0.14      0.14      0.14       500

Gradient Boosted Trees with one-over-rest

      precision  recall  f1-score   support

a       0.00      0.00      0.00        50
b       0.08      0.02      0.04        42
c       0.23      0.10      0.14        60
d       0.13      0.42      0.20        77
e       0.25      0.02      0.04        50
f       0.00      0.00      0.00        33
g       0.20      0.02      0.04        41
h       0.00      0.00      0.00        51
i       0.00      0.00      0.00        18
j       0.17      0.44      0.24        78

accuracy                           0.15       500
 macro avg      0.11      0.10      0.07       500
 weighted avg   0.12      0.15      0.10       500

Similar results I obtain for KNN and for Histogram Gradient boosting.

The same statistics for the training set are all almost 1.0.


texts: Reversing Naive Bayes to find extreme points of data sets

texts:
I'd like to know if this is a sensible idea and if there exist any already formed methods to do this (I'm new to the data science area).

Essentially, I have used Naive Bayes to accurately classify three types of food, based on their nutritious value (fat, salt, sugar, protein, and carbohydrates as my features).

Now that I can accurately classify these foods, Is there a method which uses the Naive Bayes to reverse this approach, and find the extreme values these features can be to be still classified as a type of food?

E.g: The max fat food1 can be, to still be considered food1.

I realize that these values will change, as other nutrient variables are changed, but I wondered if an optimized set of equations could be obtained in 5 dimensions?


texts:
Has anyone attempted Principal component analysis on time series and acceleration data(ie. data from accelerometer and sensors) and tried compressing it as well as regenerating back the data with minimum error?

I wanted to know if this is possible using PCA. Kindly suggest alternate methods available to compress and regenrate my data.


texts: When hierarchical clustering is preferred over k means clustering?


texts: How to set input for proper fit with lstm?

texts:
My input training and test dataset is the following size:

print(trainX.shape):(53394, 3)
print(testX.shape):(17799, 3)
print(trainY.shape):(53394,)
print(testY.shape):(17799,)

I reshaped it as follows:

trainX.shape:(1, 53394, 3)
testX.shape: (1, 17799, 3)
trainY.shape: (1, 53394)
testY.shape: (1, 17799)

Now, I pass it as the input layer of a LSTM:

model = Sequential()
model.add(LSTM(66, input_shape=(len(trainX),3)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(trainX, trainY, epochs=100,batch_size=1, verbose=2)

I am getting the error:

Error Message:
ValueError: Error when checking input: expected lstm_6_input to have shape (1, 3) but got array with shape (53394, 3)

Please help me to fit my data properly into a LSTM.


texts:
Given a string that describes a situation or method, are there algorithms that create fingerprints out of it, compare it with a corpus to then point to pages where a similiar concept is being discussed?
The simple form is word search.
You search for word X and X appears in a text whose excerpt is shown to you.
This is the next step.
You describe X without writing "X" and you get text excerpts as a result where X is also described using different phrases (or even words) with or without an explicit mention of "X".
How do you search for content, not words?
Hint: I am looking for technical terms for this problem to find research papers.


texts: Is time series multi-step ahead forecasting a sequence to sequence problem?

texts:
I'm using the keras package in order to train an LSTM for a univariate time series of type numeric (float). Performing a 1-step ahead forecast is trivial, but I'm not sure how to perform a, let's say, 10-step ahead forecast. Two questions:

1) I read about sequence to sequence NNs, but can barely find anything of it in the context of time series forecasting. Am I right with the assumption that the forecasting of more than 1 time step in advance is a seq2seq problem? That makes sense to me because each forecast depends on its predecessor.

2) An intuitive solution without seq2seq would be: Perform 1-step ahead forecast, then append this forecast to the series and use it to obtain the next forecast, and so on. How would this differ from a seq2seq approach?


texts: How to dual encode two sentences to show similarity score

texts:
I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool.

With universal sentence encoder I can first pre-encode all sentences and put them in the database. When user wants to perform query, input will too be converted in 512-dimensional vector, and we will perform sequential search on whole database by comparing cosine similarity (highest similarity vector is picked). But this is extremely slow...

Fortunately, on their semantic experiences page, they wrote the following:

  The Universal Sentence Encoder model is very similar to what we're
  using in Talk to Books and Semantris, although those applications are
  using a dual-encoder approach that maximizes for response relevance,
  while the Universal Sentence Encoder is a single encoder that returns
  an embedding for the input, instead of a score on an input pair.

One of the simpler methods they use for transforming a sentence into embedding vector is DAN (deep averaging neural network).

From what I understand, sentence is split in words, which are converted into vectors (word2vec), then we get average of all vectors that we have obtained. Average value is finally passed to one or more hidden feedforward layers, and eventually output layer that has softmax activation functions and has 512 neurons.

How can I create a dual encoder though? Do I use two different neural networks? Or does output just contain one neuron that outputs similarity?

Thank you!

P.S

My apologies if anything ignorant has been mentioned here, I'm quite confused with the concept of projecting lower dimensional space to 512 dimensional vector space.


texts: How to scale prediction back after preprocessing

texts:
So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself.

The issue I'm having is that I'm using preprocessing to scale the data to train my classifier. However when I make a prediction, the answer is also scaled. When I comment out all the preprocessing, I get accurate results. Is there a way to scale the prediction back?

The outputs are rounded to 0, 1 or 2 with each number representing one of three species.

You can see my code below:

import pandas as pd
import numpy as np
from sklearn import preprocessing, model_selection
from sklearn.linear_model import LinearRegression

df = pd.read_csv("iris.csv")

# setosa - 0
# versicolor - 1
# virginica - 2
df = df.replace("setosa", 0)
df = df.replace("versicolor", 1)
df = df.replace("virginica", 2)

X = np.array(df.drop(['species'], 1))

y = np.array(df['species'])

# Scale features
# X = preprocessing.scale(X)

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)

clf = LinearRegression(n_jobs=1)  # Linear regression clf

clf.fit(X_train, y_train)

confidence = clf.score(X_test, y_test)

print("Confidence: " + confidence)

# Inputs
sepal_length = float(input("Enter sepal length: "))
sepal_width = float(input("Enter sepal width: "))
petal_length = float(input("Enter petal length: "))
petal_width = float(input("Enter petal width: "))

# Create panda data frame with inputted data
index = [0]
d = {'sepal_length': sepal_length, 'sepal_width': sepal_width, 'petal_length': petal_length, 'petal_width': petal_width}
predict_df = pd.DataFrame(data=d, index=index)

# Create np array of features
predict_X = np.array(predict_df)

# Need to scale new X feature values
# predict_X = preprocessing.scale(predict_X, axis=1)

# Make a prediction against prediction features
prediction = clf.predict(predict_X)

print(predict_X, prediction)

rounded_prediction = int(round(prediction[0]))

if rounded_prediction == 0:
    print("== Predicted as Setosa ==")
elif rounded_prediction == 1:
    print("== Predicted as Versicolor ==")
elif rounded_prediction == 2:
    print("== Predicted as Virginica ==")
else:
    print("== Unable to make a prediction ==")

Here is an example of my output with preprocessing enabled. I'll be using one of the lines from the CSV as an example (6.4 sepal length, 3.2 sepal width, 4.5 petal length and 1.5 petal width) which should equal the versicolor species (1):

Confidence: 0.9449475378336242
Enter sepal length: 6.4
Enter sepal width: 3.2
Enter petal length: 4.5
Enter petal width: 1.5
[[ 1.39427847 -0.39039797  0.33462683 -1.33850733]] [0.41069281]
== Predicted as Setosa ==

Now with preprocessing commented out:

Confidence: 0.9132522144785978
Enter sepal length: 6.4
Enter sepal width: 3.2
Enter petal length: 4.5
Enter petal width: 1.5
[[6.4 3.2 4.5 1.5]] [1.29119283]
== Predicted as Versicolor ==

It seems I'm either doing the preprocessing wrong, or there's an extra step that I've missed out. I'm sorry if I get some of the terminology wrong and thanks in advance for answering.


texts: Can I use LSI (Latent Semantic indexing) to get similar docs for several documents at the same time?

texts:
I'm working on a Recommander system in which I'm using LSI to get similarities between videos.
I wonder if I can provide to LSI matrix more than one document and get similar docs for all those. In the case of one input doc, it runs perfectly.
I thought to concatenate all input documents into one and get similar docs for this one, but I'm not sure if this would be a good idea.
thanks in advance


texts:
I'm trying to optimize an expensive function for which I can choose sample points. The difficulty is that many function evaluations may be computed in parallel, taking varying amounts of time. I don't know which keywords to search for to find existing literature(/implementations).

So at a time, I might have already computed function values at 18 points, with 15 still being computed, and I want to start evaluating the function another point. Without the running jobs, I could make a model and find which next point might provide the most information. But now, I need to somehow tell the model that there are also 15 points for which I don't have values, but near which I don't want to evaluate the function.

Specifically, I'm looking for something with minimal assumptions and no gradient information. But I'll be happy if I just know the keyword(s) to search for. (I can come up with several hacks to that kind of work, but I wonder if there is any 'real' solution).

For illustration, say every column is a function evaluation point and every row is a timestep in the below ascii-figure. An X means the evaluation is still being computed and a * in the bottom row means it's done. A new evaluation starts as soon as an old one terminates, with others still busy.

XXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXX
 XXXXXXXXXXX XXXXX
 XXXXXXXXXXX XXXXX
 XXXXXXXXXXX XXXXX
 XXXXXXXXXXX X  XXX X
 XXXX XXXXXX X  XXX XX
 XXXX XXXXXX X  XXX  XX
 XXXX XXXXXX X  XXX  XX
 XXXX XXXXXX X  XXX  XX
 XXXX XXXXXX X  XXX  XX
 XXXX X XXXX X  XXX  XXX
 XXXX X XXXX X  X X  XXXX
 XXXX X XXXX X  X X  XXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XXXX X  X X  XXXXX
  XXX X XX X X  X X  XXXXXX
  XXX   XX X X  X X  XXXXXXX
  XXX   XX X X  X X  XXXXXXX
  XXX   XX X X  X X  XXXXXXX
  XXX   XX   X  X    X XXXXXXXX
  XXX   XX   X  X    X XXXXXXXX
  XXX   XX   X  X    X XXXXXX XX
  XXX   XX   X  X    X XXXXXX XX
  XXX   XX   X       X XXXXXX XXX
  XX    XX   X       X XXXXXX XXXX
**  ****  *** ******* *      *

Each time an evaluation completes, how to choose the next point to evaluate, given several complete and several incomplete evaluations?


texts: NLP SBert (Bert) for answer comparison STS

texts:
I've been researching a good way to automate short answer evaluation. Essentially a teacher gives a test with some questions like:
Question: why did columbus sail westward to find asia?
Answer: so he could find a new trade route to Asia through the ocean. Three goals of the Spanish in the Americas were the desire to attain great amounts of riches, to establish claims on as much land as possible, and to colonize as much as possible.
With that we have the correct answer and would like to compare that with the students answer and produce a score based on similarity. I know this isn't a reliable replacement for human grading, but for the sake of the example.
I've come across this paper and codebase:
https://arxiv.org/pdf/1908.10084.pdf
https://github.com/UKPLab/sentence-transformers
It seems like the ideal method for solving this problem, but most examples are based on scoring/ranking of semantic search. I question whether I'm on the right path, given that I'm just comparing two answers and not a cluster. Anyone with more experience, possibly can provide some guidance?


texts:
So, I'm having problems learning for a data analysis lecture. The topic is the construction of the "central interval" as our lecturer calls it. The concept shall path the way to understand confidence & credible intervals. But I don't even get his concept and therefore wanted to ask for help.
I will give an excerpt of the passage I don't get (it is in the context of a bionmial distribution where r is the possible outcome of a bernoulli experiment):

"To this end, we define the Central Probability Interval and Smallest
Probability Interval containing a probability at least 1-$\alpha$.
This  interval  is  constructed  by  defining  a  smallest  member  of  the  set  of possible outcomes, $r_{1}$, and a largest member, $r_{2}$, subject to the criterion that
$$P(r<r_{1}) <= \alpha/2$$and $$P(r>r_{2}) <= \alpha/2$$ while maximizing these probablities."

First of all, $\alpha$ seems to be an arbitrary factor because it isn't mentioned anytime before. After choosing such an $\alpha$, we look for two numbers $r_{1,2}$. These numbers are limits which, when r surpasses these number, will yield a probability smaller than $\alpha$/2. But what about the maximizing of the probability? What probability?

"Stating this precisely, we write:
$$
r_{1} = 0 \quad\text{if}\quad P(r=0|N,p) > \alpha/2\quad\text{ else}\quad r_{1} =  \sup_{r\in0,...,N}[\sum_{i=0}^{r}P(i|N,p) <= \alpha/2]+1
$$
and
$$
r_{2} = N\quad\text{if}\quad P(r=N|N,p) > \alpha/2\quad\text{ else}\quad r_{2} =  \sup_{r\in0,...,N}[\sum_{i=r}^{N}P(i|N,p) <= \alpha/2]-1
$$
The set of our results in our Central Interval is
$$
O^{C}_{1-\alpha}=\{ r_{1}, r_{1}+1,...r_{2}\}.
$$"

Now, I cannot make anything of the equations simply because I don't know how to read and process them properly. Maybe someone has an idea or can give me a good example?
Your help is much appreciated.

Edit:
I found a hint online and I think, now I get it.
We construct an interval which contains a prbability 1-$\alpha$. Therefore, the probability of all r values outside the interval [$r_{1}$, $r_{2}$] has to be $\alpha$. If you split this symmetric you get the criterion stated above.

Now as for the equations. They add up all the probabilities which contribute to an outcome smaller than $\alpha$/2. Taking the $\sup_{r \in 0,...,N}$ now gives the r value for which this sum is maximized. Then you have to add the +1 in order to take the next r which then is $r_{1}$.


texts: Hindi English Machine translation using OpenNMT

texts:
I am trying to perform machine translation using this.
I am using this dataset for training.
Result looks like this -
root@krm-datascience:/home/krm.khan/OpenNMT-py# python translate.py -model demo-model_acc_15.97_ppl_486.59_e1.pt -src data/src-test.txt -output pred.txt -replace_unk -verbose
Loading model

SENT 1: डेली हिन्दी न्यूज - बुंदेलखंड का प्रथम अन्तरजालीय स्थल
PRED 1: This .
PRED SCORE: -8.3647

SENT 2: मैं भारत से प्यार करता हूँ
PRED 2: This .
PRED SCORE: -8.1020

SENT 3: पाकिस्तानी सरकार के अनुसार इसी समय पाकिस्तान की नींव डाली गई थी ।
PRED 3: This .
PRED SCORE: -7.8469

SENT 4: ज़बूर जो कि दाउद को प्रदान की गयी
PRED 4: This .
PRED SCORE: -8.3363

SENT 5: प्रशासनिक विभाजन
PRED 5: India .
PRED SCORE: -6.6838

SENT 6: वे इस देश के प्रथम <unk> -LRB- अफ्रीकी <unk> -RRB-
PRED 6: This .
PRED SCORE: -8.2260

SENT 7: नेपाली विदेश <unk>
PRED 7: Hindi of India .
PRED SCORE: -11.5715
PRED AVG SCORE: -3.6957, PRED PPL: 40.2738

I could not understand the result. Is it correctly trained? Why result is so bad.


texts:
I am currently working with a dataset that seems very easily separable and I have an accuracy of 99% for SVM (NN-98%, RF-98%, DT-96-97% and I have checked for leakage & overfitting). As part of my project, I am also learning how to implement a hybrid model but its accuracy is also 99% (1 record misclassified). It seems to be misclassifying the same record as SVM and no other algorithm seems to get that record correct. Probability wise- RF and Logistic Regression give a probability of 0.5-0.7 but all other models give 0.9-1 probability for the wrong prediction. I was wondering what I should do now. Are there certain techniques I can use to reduce the probability to below 0.5 in the stacked model so that it isn't classified the way it currently is? I am using the StackingCVClassifier for implementation and this is a binary classification project.
I am currently using SVM, LR, RF as base models and NB as the meta-model.
My other question is if there is even value in making a hybrid model if the accuracy is the same as SVM's?
Thank you!


texts: What does it mean when your NN model has "multiple" inputs or outputs in Keras?

texts:
I'm reading the guide on Keras website: https://keras.io/guides/sequential_model/
And it says don't use a sequential NN if you have multiple inputs or outputs.

What is the difference between multiple inputs and just having many input variables?
Does it mean having two np.arrays of features?
Why wouldn't someone just combine the two arrays into a single array?
Also, what does "multiple" outputs mean?
Would this just mean that you want to jointly produce two predictions?
Could someone give an example (non - code) of a real life application so I understand what "multiple" input and output means?

Thank you!


texts: An error with respect to filter weights in CNN during the backpropagation

texts:
Let's say a convolutional layer takes an input $X$ with dimensions of 5x100x100 and applies 10 filters $F$ 5x5x5, thus produces an output $O$ 10 feature maps 96x96.

During the backpropagation the layer receives $\frac{dE}{dO}$ of shape 10x96x96.

My question is how to compute $\frac{dE}{dF}$ ?

According to that article $\frac{dE}{dF}$ can be calculated as convolution between $X$ and $\frac{dE}{dO}$. Unfortunately, the article does not cover a case with multiple filters and multiple input channels.

Since $X$ has shape 5x100x100 and $\frac{dE}{dO}$ has shape 10X96x96 the depth of $X$ equals to 5 and the depth of $\frac{dE}{dO}$ equals to 10. So the depth dimension does not match. How to compute convolution in that case ?


texts: Do I need to learn Hadoop to be a Data Scientist?

texts: An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn Hadoop to be a Data Scientist?


texts: Principle Component Analysis on multiple functions

texts:
I am working on a research project that uses data of geometrical structure complexities of different land coverage types (primarily cities, pastures and natural structures). My supervisor has instructed me to use PCA for the reduction of dimensions, but I have trouble understanding how it would work on my data.

That data is a collection of a hundred 2D plots of which the x-axis runs from 0 to 255 (in steps of 1) and the y-axis from 1 to 2 (in non-integer steps). The individual plots are not linear, but to some extent have the same shapes.

My problem is that for as far as I know, PCA won't work here because for every x-value there are multiple y-values, if I plot all of the individual datasets into one big graph. Also, isn't it a problem that the individual plots are nonlinear?

So my question is: would PCA work on this 'multivalued and nonlinear' dataset? If so, where could I look for an explanation on how exactly it would work?


texts: Why PyTorch is faster than sklearn models?

texts:
Recently, I get to know about the hummingbird library for Python. I trained a RandomForest on a 10M-sized dataset with 2 labels. With sklearn it was taking 450 ms for inference. But after converting the same model to PyTorch, now it takes 128ms on CPU inference.
If both are running on the CPU, then why hummingbird's Pytorch model is faster than sklean model?
I am not getting what hummingbird does to my sklearn model to increases speed.


texts: Still have the same error after reshaping

texts:
I set my column 1 to 13 as X and column 14 as my Y.
My data set has 14 columns and 401 rows including the parameters
Bp,Sg,Al,Su,Rbc,Bu,Sc,Sod,Pot,Hemo,Wbcc,Rbcc,Htn,Class
80,1.02,1,0,1,36,1.2,137.53,4.63,15.4,7800,5.2,1,1

50,1.02,4,0,1,18,0.8,137.53,4.63,11.3,6000,4.71,0,1

80,1.01,2,3,1,53,1.8,137.53,4.63,9.6,7500,4.71,0,1

70,1.005,4,0,1,56,3.8,111,2.5,11.2,6700,3.9,1,1

80,1.01,2,0,1,26,1.4,137.53,4.63,11.6,7300,4.6,0,1

This is my code
import pandas as pd

from sklearn.metrics import accuracy_score

from sklearn.model_selection import train_test_split

from sklearn.ensemble import RandomForestClassifier

from PIL import Image

import streamlit as st

skipinitialspace = True

data = pd.read_csv("C:/Users/ADMIN/PycharmProjects/pythonProject/venv/KidneyPrediction/Kidney.csv")

st.subheader('Data Information: ')

st.dataframe(data)

st.write(data.describe())

chart = st.bar_chart(data)

X = data.iloc[:,0:13].values

X.reshape(-1, 13)

Y = data.loc[:,'Class'].values

Y.reshape(-1,1)

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.25, random_state=0)

The error is
ValueError: Expected 2D array, got scalar array instead: array=nan. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Traceback:
File "C:\Users\ADMIN\PycharmProjects\pythonProject\venv\KidneyPrediction\Kidney_Prediction.py", line 72, in <module>
    prediction = RandomForestClassifier.predict(user_input)


texts:
I have a dataset that pertains to calls received to a hospital emergency helpline. My task is to identify patterns in the data.
How do I approach examining the dataset to identify patterns?
Extract from dataset.


texts:
I have previously used WHL (wheel) files to install various Python packages. But, it seems there's no such file for NLTK. Any workaround for this please?

https://pypi.org/project/nltk/

The problem is I don't have access to install or *.exe files. Neither PIP3 works because of some reason (firewall i believe!)

Error:

  NewConnectionError('pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection
  object: Failed to establish a new connection: [WinError 10061] No
  connection could be made because the target machine actively refused
  it',)': /simple/nltk/

Update-1

Just discovered, there was a Zip installation file. And tried it. Had a dependency on "singledispatch-3.4.0.3-py2.py3-none-any.whl" but was able to install that being a WHL file.
Got a success message -

Installed c:\users\rb\appdata\local\programs\python\python36\lib\site-packages\nltk-3.4-py3.6.egg
Processing dependencies for nltk==3.4
Searching for singledispatch==3.4.0.3
Best match: singledispatch 3.4.0.3
Adding singledispatch 3.4.0.3 to easy-install.pth file

Using c:\users\rb\appdata\local\programs\python\python36\lib\site-packages
Searching for six==1.11.0
Best match: six 1.11.0
Adding six 1.11.0 to easy-install.pth file

Using c:\users\rb\appdata\local\programs\python\python36\lib\site-packages
Finished processing dependencies for nltk==3.4

Update-2

Still i wasn't able to do Import NLTK. But i noticed an Egg file in the Python folder.

C:\Users\rb>python -m easy_install "C:\Users\rb\AppData\Local\Programs\Python\Python36\Lib\site-packages\nltk-3.4-py3.6.egg"
Processing nltk-3.4-py3.6.egg
nltk 3.4 is already the active version in easy-install.pth

Installed c:\users\rb\appdata\local\programs\python\python36\lib\site-packages\nltk-3.4-py3.6.egg
Processing dependencies for nltk==3.4
Finished processing dependencies for nltk==3.4

Still i'm unable to proceed. But the hunt is still on ...


texts: Multivariate Time-Series with Differing Selling Windows

texts:
I have a data-set in the format below. I am attempting to forecast the target_value by Product_Id for an upcoming season, which is always 26 weeks in length. The main challenge is that in an upcoming season, a product may sell in a different group of stores than last season, and may be sold in a different set of weeks than a previous season (some products sell all 26 weeks, others sell in windows i.e. wk 6-13). I wanted to rely on product attributes to help determine how a product would sell in a different set of weeks/stores based on how similar products had sold in those weeks/stores.

The data is time-series and has many different products. It consists of categorical variables, numerical variables, as well as a store_id which indicates what store the sales are coming from.

Summary for data:

has 3 years of history, target variable is sales for a 6 month forecast horizon
multi-variate time series
consists of categorical and numerical variables
has products that are sold in differing time-frames (those time-frames may change in the future for the same product. i.e. last year it sold in weeks 3-10, but in the forecast horizon next year it's expected to sell in weeks 6-14)
has products that are sold in different stores in different time-frames (can also sell in different stores in the future forecast horizon than it has sold in in the past)
is sparse for subsets of products

My questions are:

What is the appropriate algorithm that could handle the sparse history as well as multiple variables
What is the proper way to structure the data as input into the algorithm?
--For this I had planned on converting categorical variables to dummy variables, but was unsure on whether to keep time as is or convert the week to a dummy variable also since I could be forecasting different sets of weeks for different products. I had the same question for store_id.

I'm new to structuring these problems, so please forgive any errors or missing information, and let me know what additional data is needed. Any and all feedback is appreciated!


texts: How to mathematically explain the translational and rotational invariance of PCA

texts:
There is a homework question for a course I am self studying (not a student) that is:

let our $n \times d$-dimensional data vectors be denoted by $x_1,\ldots,x_n$  and let $R$ be a $d \times d$ rotation matrix. For simplicity, you may assume that the $x_t$'s have been centered at $0$. Let $$x_t' = Rx_t + v$$
where $v$ is some fixed translation. Forming a second dataset.
Now, for any $K$ we pick, let us use PCA on each of the two data sets to obtain $K$-dimensional projections $y_1,\ldots ,y_n$ and $y_1',\ldots,y_n'$, respectively.

Write down a relationship between the two PCA projection matrices $W$ and $W'$ in
terms of the rotation matrix $R$ and the translation vector $v$. Explain mathematically how you arrived at this answer.

My answer is basically that while for the untransformed dataset we have that $W$ is the top $k$ eigenvectors of $S[\mu] (S[\mu]$ is the covariance matrix of the untransformed dataset centered at $\mu (\mu$ is $0$ for untransformed dataset)) that $W'$ will be the top $k$ eigenvectors of the matrix $(R^T) \times S[R^T(\mu-v)] R$. ($R^T$ is $R$ transposed). I did this by applying the transformation in the definition of the covariance matrix $S'$ to find a relation between $S'$ and $S$.

The goal of the problem is to show the rotational and translational invariance of PCA. Can anyone give an explanation for this?


texts: Using embedding layer output as input to .fit() call in Keras

texts:
I want to build a classifier in Keras that predicts the next item bought by a customer (i.e. multiclass classification). One of the features I intend to input to the model will be the last item bought by a particular customer. My problem is that the list of possible items is extremely large, several tens of thousands. With that in mind, I'm going to feed the ItemID feature into an Embedding Layer in Keras, and concatenate that with other features before running it through the model.

My question is, can I use the Keras shared layer functionality to also embed the labels for the training data (since they are all from the same vocabulary as that embedded ItemID input), such that instead of having a softmax output of n_classes I have an output of n_dimensions, where n_dimensions is however many dims I decided to set that Embedding layer too?


texts:
I have a dataset which have two features,"feature_1",and "feature_2" and time in timestamp of 10 second intervals.
While plotting,there is  "trend" of both features with respect to time,but don't have seasonality .And both features are also highly correlated.
The task is to predict "feature_2" values.
And importantly,in test set "feature_1" values are also given.
Which model and approach should I go with


texts: What is coadaptation of neurons in neural networks?

texts: Looking for a bare minimum example (3 hidden units only maybe?) for what weights of a neural network with heavily coadapted weights would look like and showcase why they are bad. Also, how is coadaptation a sign of overfitting?


texts: Which ML algorithm to use if we have categorical data, numeric data, derived data (derived from) other variable in our data set?

texts: I am a beginner in Data Science. I have a data set which contains numerical data, categorical data and derived data (derived from other columns). The target column (dependent) is binary. Which Machine Learning algorithm will predict my target column most accurate?


texts: When, if ever, is nearest neighbor classification the best choice?

texts:
I've been researching the history and use of k-nearest neighbor classification and regression, and various tweaks including k-d trees and LAESA.

I understand that it is useful because it is simple and flexible, but can be computationally expensive and requires a lot of data storage.

But here's what I don't know:

  Is there any class of problems for which nearest neighbor classification is the best or one of the best algorithms to use?

By 'class of problems' I mean either a class based on data structure (for instance, maybe KNN is great for low-dimensional data with a mix of nominal and numerical data), or a class of real-life problems (maybe KNN is useful in predicting diseases for insurance holders).


texts: Pardon my question if it seems too general... I'm new to networks in general, but recently I came across a type of network called citations network. However, I couldn't figure out why we do citation analysis or what kind of special information we can get out of it.


texts: I'm new to computer vision, and I'm looking for a good place to start from, what's better between open cv in python or open cv in c++


texts: Some of the p-values are NaN - logistic regression

texts:
I am trying to do logisitc regression, but have this issue - some of the p values are NaN

model = sm.Logit(y2,X2.astype(float))
result = model.fit()
result.summary()

Any ideas what to do?


texts: How to choose the support vectors after minimizing the objective function?

texts:
I'm training a SVM that uses the following objective function:

$$
\frac{1}{2}\sum_i{\sum_j{\alpha_i\alpha_j t_i t_j \mathcal{K}(\vec{x_i}, \vec{x_j})}} - \sum_i{\alpha_i}
$$

The objective function is minimized by scipy.optimize.minimize in Python. The resulting $\alpha$s can be split between the "huge" ones (several millions) and the others (from $1$ to $10^{-5}$).

Can I consider that only the datapoints corresponding to "huge" alphas will be support vectors? In my previous SVMs, I used a threshold of $10^{-5}$ or $10^{-4}$ to distinguish them, but it doesn't seem to make sense to have such a difference in support vectors weights.


texts: We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of sentences with and without grammatical errors. The retrained model is then used to identify sentences with or without errors. Are there any other approaches we could make use of using BERT, other than building a classifier?


texts: How curvature information in second order optimization methods helps

texts: It is said that second order optimization methods in neural networks work better than first order because they contain information about rate of change of gradient or the curvature. This information helps to choose a better step size for moving forward on the error surface. It is not clear as to how the rate of change of gradient is controlling the step size and leading to better optimization. For simplicity, consider only one weight update iteration.


texts: Why is my CNN activation maximization split vertically?

texts:
I am using this code in Python:
import matplotlib.pyplot as plt
import tensorflow as tf
from tf_keras_vis.activation_maximization import ActivationMaximization

def get_conv(in_model):
    target_layer = in_model.get_layer(name="block5_conv3")
    new_model = tf.keras.Model(inputs=in_model.inputs, outputs=target_layer.output)
    new_model.layers[-1].activation = tf.keras.activations.linear
    return new_model

def get_score(output):
    outputs = [o[0] for o in output]
    return outputs

model = tf.keras.applications.vgg16.VGG16()
vis = ActivationMaximization(model, get_conv)
seed = tf.random.uniform((1, *model.input.shape[1:]), 0, 255)
maps = vis(get_score, seed)
plt.imshow(maps[0])
plt.show()

And this is what I get:

I was wondering why there is a huge flat image region in the bottom half of the image.


texts: AUC higher than accuracy in multi-class problem

texts:
I stumbled upon a 3-class classification problem where all compared classifiers yield a higher AUC than accuracy (usually around 10% higher). This happens both when the dataset is balanced or slightly imbalanced.
Now, after looking at this answer: Why is AUC higher for a classifier that is less accurate than for one that is more accurate? I understand that, for binary classification, this might happen because the accuracy is typically computed at a threshold of 0.5 whereas AUC is based on all threshold values.
But what happens with multi-class classification? Specifically, those scenarios where accuracy is defined as the frequency with which the predicted labels match the true labels (tf.keras.metrics.CategoricalAccuracy) and AUC is defined as the weighted average of the AUC for each class vs the rest (One-vs-rest) (sklearn.roc_auc_score). Why might AUC be higher there?
In other words, I'm trying to understand what this result means. Does it mean that my classifiers can discern well when each class is measured against the others (AUC), but not as well when the prediction probabilities are an output of the softmax function, and therefore spread out for the three classes?


texts: how to use (read) google pre-trained word2vec model file?

texts:
I am trying to apply open() function in keras to use Google news-vectors-negative300.bin which is a pre-trained file via word2vec such as GloVe, but after downloading GloVe it contains 4 files with txt prefix vs the Google news-vectors-negative300.bin folder contains a file with binary prefix namely 'data' which is 3.4 GB.
I write the commands on ubuntu 17.10 via keras with tensorflow backend on spyder with python 3.5, and after implementing the command it gave me this error:

File "/home/mary/anaconda3/envs/virenv/lib/python3.5/codecs.py", line 321, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 19: invalid start byte.

the written code is as follow: f = open('data').
I have already implemented the same code successfully when I applied
f = open('glove.6B.100d.txt').
What is the main problem?


texts: Should I resize my object localization dataset in order to train a fully convolutional network?

texts: I want to create a fully-convolutional neural net that trains on wider face datasets in order to draw bounding box around faces. The dataset is highly diverse in the image sizes. So my question is, are there no better ways other than to resize the image in order to train a fully convolutional network? Even if there is no other way than to resize the input image what should I do about the annotated bounding boxes? I do not wish to manually annotate all bounding boxes after resizing the images.


texts: Is there a way to cluster encoded data (encoded with variational autoencoder)?

texts:
I have managed to encode data through a variational autoencoder. The difference between an autoencoder and a variational autoencoder is that instead of encoding an input as a single point, we encode it as a distribution over the latent space. I want to use the encoded data (so the distributions) to cluster on (with for example KmL3D in r). The encoded data looks like this:

It is a tuple (type). I was wondering how I can cluster on these parameters (parameters of the distributions) now? The parameters include 3 values as you can see, decoded data, mu and sigma.


texts: How to use Autoencoders for outlier detection on images

texts:
I have a bunch of images taken from a camera showing a pipe and would like to detect if the pipe is leaking or not. There are very few examples of leaking pipe in the data set. So considering this problem as a supervised learning problem, I think that it may not give us good result due to imbalanced data. I'm thinking of using autoencoder and considering it as an outlier detection problem.

I'm new to deep learning so I'd like to know how the architecture of my neural network should look like. Should I have some convolutional layers first and then an autoencoder or I should only have autoencoder? What would be the best DL library for such a use case? I also thinking of using only the photos which do not have any leak for the training phase, is that OK?


texts: Computationally Inexpensive Imputation Techniques in R

texts:
I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers.

I'd like to use a method that is much faster than it is accurate, because of the size of the data-set and the fact that I don't have much to lose in a speed-accuracy tradeoff.

Running missForest() takes several hours while Hmisc's impute() function gives unsatisfactory results.

What functions in R might be useful in such (or similar) case?


texts: Make use of relationships on recommendation systems

texts:
I have a data set of user rating for movie as

user_name, product_name, user_rating

and I am using this data to recommend new movie to user (collaborative filtering).

I also have another data set which show relationships between user like

user_1, CO_WORKER, user_2
user_1, FATHER_OF, user_3
user_3, FRIEND_OF, user_4 etc

Now what are the ways to include 'relationship data' along with 'ratings data' to produce better recommendations ?

example : user_1 might like the movies which user_2 likes since they are co_workers


texts: Gradient Boosting RandomSearchCV or GridSearchCV

texts:
In your algorithms, when you use Gradient Boosting, do you prefer RandomSearchCV or GridSearchCV in order to optimize your hyperparameters ?

Thanks for sharing your experience.


texts: Which algorithm to use for efficient resource assignment?

texts:
I am a starter in ML. So pardon me if the question is naive.

We have a Project Management tool where users can create a ticket and assign it to others. This is just one part of the tool but we are trying to automate assignment of ticket to people. Based on the history, i am trying to suggest who should the creator assign the ticket to so that the ticket is closed without any delay, minimal risk etc.. While i have the features/variables ready, I am having a hard time trying to find which algorithm or method to use.

Appreciate the help. Thx.


texts:
Are social network analysis and graph analytics the same thing? If not, what are the differences? Is social network analysis perhaps a subset of graph analytics?

Are they just modern extensions of graph theory which have become relevant due to modern types of data and data analysis?

What role does software like NetworkX and neo4j play?


texts: How to determine sentence similarity labels for sentence transformer fine-tuning?

texts:
I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the similarity of the provided strings.
However, without knowing the existing similarity score of two arbitrary strings (e.g. "apple" vs "orange"), what is the best way to determine the best label values for training?  A solution I thought of was to generate cosine similarity scores from other models and then use it as training labels, but then this would not work for domain specific settings, which I'm curious about what one could do.


texts: what ETL technique should i use for text documents using Hadoop?

texts:
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document where a "keyword" is more relevant, he also wants us to divide the project in 3 parts:

Data acquisition, preprocing (cleaning, transform, join, etc), loading. ETL PROCESS.
Data processing.
User friendly application.

I need to define what technologies or methods i'm gonna user for each of the parts of the project, but i have no idea what to do on the ETL part since the documents are gonna be written in legible English (they are books), i will appreciate any information you can give me on this but also about the other parts of the project.

Thanks a million for reading.


texts: How can I use a class variable with many possible values in logistic regression?

texts:
I am attempting to build a logistic regression model that determines the probability of an outcome based on a set of independent variables.

For context, the data is based on a project in which sales representatives and branch managers for a builders merchant were given price recommendations for their customers' deals, and were given the option of saying 'Yes' or 'No' to these price recommendations.  The Yes or No answer is my dependent variable; I need to determine which variables can predict with the highest probability whether the respondent will say 'no' to a price recommendation.

Most of the independent variables work fine in this model, except for one which I am currently unable to test; individual differences between the respondents.

My hypothesis is that the propensity to say 'no' will be stronger in some individual respondents than others, be it for psychological or geographical reasons, and that these individual differences will be a stronger determiner for the dependent variable than any other class variable.

There are about 800 respondents, so simply shoving them in as an independent variable does not produce the desired results.

Is there a method of doing this in logistic regression?  Should I use another analysis technique for this?

I am using Statistical Analysis Software (SAS) to carry out the logistic regression.


texts: How do I show plots when using Graphlab?

texts:
Sorry, if this topic is not connected directly to Data Science.

I want to understand how the Graphlab tool works. Firstly I want to execute the toy examples from the Gallery site. When I try to execute the example code, everything is OK except one command: I can not see the graphlab plot after show(). The command show() returns to me some kind of object in IPython and nothing in the IPython Notebook.

If the example code has the plot, which depends directly on the matplotlib module, I can produce the real plots and save it on my machine. Consequently, I suppose the main error depends on the graphlab (or object from its class).
If somebody already used this tool and rendered the plot, can he/she tell me, how I can execute the plots command?

In [8]: import graphlab

In [9]: from IPython.display import display

        from IPython.display import Image

        graphlab.canvas.set_target('ipynb')

In [10]:import urllib

        url = 'https://s3.amazonaws.com/GraphLab-Datasets/americanMovies   /freebase_performances.csv'

        urllib.urlretrieve(url, filename='freebase_performances.csv')  # downloads an 8MB file to the working directory

Out[10]: ('freebase_performances.csv', <httplib.HTTPMessage instance at 0x7f44e153cf38>)

In [11]: data = graphlab.SFrame.read_csv('remote://freebase_performances.csv', column_type_hints={'year': int})

...
...
...

In [15]:data.show()

No plot after this line
...
...

In [19]:print data.show()

<IPython.core.display.Javascript object at 0x7f44e14c0850>

The object of graphlab (?) after print command


texts: Analyzing country data. Does duplicating observations by population for regression make sense?

texts:
Question: While analyzing country happiness data via OLS regression, should I duplicate observations based on country population?

Example: If duplicating per million, the U.S. would have 327 observations and Denmark would have 6. Currently every country has 1.

If it matters...

y: Happiness score

X: GDPPerCapita, InfantMortalityRatePer1000, UnemploymentPercent, GiniIndex, etc.

Function to duplicate observations based on population:

def create_population_weighted_df(df):
    df_pop = pd.DataFrame(columns=df.columns)
    for index, row in df.iterrows():
        for i in range(int(row['PopulationInMillions'])):
            df_pop = df_pop.append(row)
    return df_pop

Thanks.


texts: Sliding window time series data with Python Pandas data frame


texts: Am I doing a log transformation of data correctly?

texts:
I'm doing some exploratory data analysis on some data and I get these histograms:

That looks like a candidate for a log transformation on the data, so I run the following Python code to transform the data:

df["abv"].apply(np.log).hist()
df["ibu"].apply(np.log).hist()
plt.show()

And I get this new plot of the transformed histograms:

Am I correct that a log transform was ok to do in this case, and if so, what's the best way to interpret the results?


texts: Association rules in R / Subset rules based on feature

texts:
I tried using the apriori algorithm in the arules library for association rules mining in R. There's two thing I'd like to ask:

Why are the lhs and rhs of the rules in intervals? For example U get something like this
lhs : Variable1[0,4] - rhs : Variable2[0,3]

How can I subset the rules extracted for a certain variable? Say I want rules for Variable 1. How can I  subset the result to get rules containing that variable only?


texts: Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?


texts: What should I do in case the resolution of the image varies in the dataset?

texts:
Suppose that I train my image dataset on CNN, but the resolution of the image varies significantly on the dataset.

In this case, should I scale the images up to the image that has the maximum resolution, or scale down to the lowest resolution? In that case should I scale up/down the whole images even if the highest likelihood of the whole samples are somewhere in the middle of the distribution of the resolution?

Or should I use another technique to deal with the varying resolution problem?


texts: customer segmentation with unbalanced data

texts:
I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some clusters but I am more interested in the best approach to do it.

My data has basically 3 different products. Each product has a dozens of features. The issue is that one product is purchased by 95% of the customer while the other 2 are only purchased by 10 to 20% of the customers ( each customer can purchase any of the 3 products multiple time). It means that my dataset has a lot of zeros for product 2 and 3 (which is also informative).

I have done segmentation on each product individually and now I am trying to do a clustering based on all the data . I basically merged 3 datasets by customer to get one file.

Note that the data file include 1 to 2 millions of rows.

I have looked at the distribution of every variables and while some have a nicely normal distribution, many other have peak at 0 and  sometimes 1 ( I have several percentage variables).  Some examples below:
enter image description here

After some data cleaning I used K-mean in pyspark to get some clusters. I can see some interesting segment based on the distribution of some of the features in the cluster. However I am not sure if using euclidian distance is the best option in my case. I have been thinking of trying other algorithm such as k- medoid but it also come with challenges as I need to transform to panda data frame and then run it in python which might  fail with so much data.

What would be the best way to handle this scenario when you have features for 3 different products but 2 of them are not purchased as much as the first one?All the features are numerical in my file but  I also thought about adding binary variables to tell if customer purchased the product or not and then use them as features.


texts:
My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age.

Statistically, it's a conditional probability. Example: When 0 years old, I'm expected to live until 70 years old (70 years left). However when I'm 70 years old, I'm expected to live until 83 (13 years left).

In my dataset, I have the true age of when someone died, and it follows a distribution. I have thus augmented my data, if someone died at the age of N, then there will be N datapoints of current age(feature) from 0 -> N and years left to live (target) from N -> 0 accordingly.

Through this augumentation, I'm hoping to model the distribution of current age vs years left to live through the data. I would never want my network to output (after denormalization) that if the current age was 70, the person would have 50 years left.

Now, the best practice of an MLP states that 1-2 hidden layers is enough, and amount of nodes in the hidden layers should be between the amount of input and output layers. However if I just normalized age through feature scaling, I would have one input node, one hidden node and one output node. This is realistically not enough, I'd expect that I need a lot of hidden nodes.

If I used one hot encoding that represents current age from 0 to 100. I'd be able to use 50 hidden nodes through good practice. If I was 70 years today, I would probably never activate the node that will tell me I'd have another 50 years to live.

My question then, is that should I skip the one hot encoding, and just scale the age and use for example 50 hidden nodes?

Theoretically, because age is continuous as not as distinct nor discrete as say "cat" and "robot" then one-hot is not necessary, but is there any drawbacks if I choose to one-hot encode?


texts: Difference between output of probabilistic and ordinary least squares regressions

texts:
If I execute the commands

my_reg = LinearRegression()
lin.reg.fit(X,Y)

I train my model. To my understanding training a model is calculating coefficient estimators.

I do not really understand the difference between this and e.g.

scipy.stats.linregress(X,Y)

calculating a 'normal' regression that also gives me the coefficient estimators and all the other statistics connected with it.

Could anyone tell me what is the difference here?


texts: I am getting .1 after column name while read csv  file from pandas


texts: How to deal with item belonging to more than one category

texts:
So I'm training a classification task which takes as input some description of the state of a 2x2x2 rubix cube and outputs the optimal move to take. And a potential problem I noticed is that in many states more than one move is optimal. In particular, only about 46% of states have only one optimal move, and 24% have 2, 12% have three, etc. So I have a few choices.

The options I thought of were

have each data point choose an optimal move at random
Do cross-entropy minimization with the data point containing the same probability for each optimal move. i.e (0.5,0.5,0,0,0,0,0,0,0) if the first two are optimal
Discard states with more than one optimal move (this seems really bad)

What is the standard practice? Also, is there a difference between 1 and 2?

If necessary you may assume that I'm using a neural network model


texts: Handling outliers and Null values in Decision tree

texts:
Outliers : As I understand, decision trees are robust to outliers. Can anybody please confirm if my hypothesis is right with an example?
(What if I have a features ranging from 0 to 9 but there is an outlier of which value is 10000?)
Whether it creates a separate leaf for that outlier sample or would it be merged with some other tree's leaves?

NULL Values : Do we need to replace the null values before building the model using Decision tree or it would be taken care automatically by the decision tree technique?

Thank you.


texts: How to calculate Accuracy, Precision, Recall and F1 score based on predict_proba matrix?


texts:
I have an e-commerce website where customers can purchase items directly from the site. I have training data which includes order id, user id, order number, days since prior order, product id, add to cart order, reordered...

I am trying to predict, for each user, what items he will purchase on his next order. I tried to use Naive Bayes, average purchase items per user and the following equation: posterior ~ Bayes Factor x prior but the prediction outcome is not good and has many false positives and/or negatives.

Maybe I can try to first train on the number of items a user will purchase then train on the specific items he will get but not sure will it get better results.
I think this can go in the multi label classification but has not used multi labels in classification before.

I am using python with sklearn, pandas...

Any better models I can use and how to train and predict variable multi labels and whether I can do it in sklearn? Keep in mind that the data is large and predicting using some of the classification algorithms in sklearn unfortunately takes huge amounts of memory so, any ideas on how to reduce memory consumption would also be useful.


texts: Determining correlated product categories using store purchase history

texts:
I have a large dataset that contains product purchase history, like so:

userID  productID   category        subcategory
123     ABC         Kitchen         Knives
123     BEA         Kitchen         Organization
233     ZZS         Electronics     Phones

For a first project, I'm looking to answer the question: "What discrete groups of categories/subcategories do shoppers tend to shop in?". For example, we may find that shoppers who buy Monitors are highly likely to buy Keyboards and Mice as well.

Any direction on getting started on a problem like this is appreciated!