texts: What is the relationship between "square loss" and "Mean squared error"?

texts:
Loss functions are important part of machine learning.

Square loss is one of the most popular loss functions.

Mean squared error (MSE) measures  the average squared difference between the estimated values and what
 is estimated

The estimated value is denoted as $\hat y$, what is estimated is denoted as $y$.

$${\displaystyle \operatorname {MSE} ={\frac {1}{m}}\sum _{i=1}^{m}(y_{i}-{\hat y_{i}})^{2}.}$$

in the context of distance/difference, it seems that MSE and square loss do the same job.

what is the relationship between "square loss" and "Mean squared error"?


texts: Convergence in Hartigan-Wong k-means method and other algorithms

texts:
I have been trying to understand the different k-means clustering algorithms mainly that are implemented in the stats package of the R language.

I understand the Lloyd's algorithm and MacQueen's online algorithm. The way I understand them is as follows:

Lloyd's Algorithm:

Initially ‘k’ random observations are chosen that will serve as the centroids of the ‘k’ clusters. Then the following steps occur in iteration till the centroids converge.

The Euclidean distance between each observation and the chosen centroids is calculated.
The observations that are closest to each centroids are tagged within ‘k’ buckets.
The mean of all the observations in each bucket serves as new centroids.
The new centroids replace the old centroids and the iteration goes back to step 1 if the old and new centroids have not converged.

The conditions to converge are the following: the old and the new centroids are exactly identical, the difference between the centroids is small (of the order of 10^-3) or the maximum number of iterations (10 or 100) are reached.

MacQueen's Algorithm:

This is an online version where the first 'k' instances are chosen as centroids.

Then each instance is placed in buckets depending on which centroid is closest to that instance. The respective centroid is recalculated.

Repeat this step till each instance is placed in the appropriate bucket.

This algorithm only has one iteration and the loop goes on for 'x' instances

Hartigan-Wong Algorithm:

Assign all the points/instances to random buckets and calculate the respective centroid.
Starting from the first instance find the nearest centroid and assing that bucket. If the bucket changed then recalculate the new centroids i.e. the centroid of the newly assigned bucket and the centroid of the old bucket assignment as those are two centroids that are affected by the change
Loop through all the points and get new centroids.
Do a second iteration of points 2 and 3 which performs sort of a clean-up operation and reassigns stray points to correct buckets.

So this algorithm performs 2 iterations before we see the convergence result.

Now, I am unsure if what I think in point 4 in the Hartigan-Wong algorithm is the correct method of the algorithm. My question is, if the following method for Hartigan-Wong is the correct method to implement k-means? Are there only two iterations for this method? if not, what is the condition for convergence (when to stop)?

Another possible implementation explanation what I understand is.

Assign all the points/instances to random buckets and calculate the respective centroid.
Starting from the first instance find the nearest centroid and assign that bucket. If the bucket changed then recalculate the new centroids i.e. the centroid of the newly assigned bucket and the centroid of the old bucket assignment as those are two centroids that are affected by the change.
Once there is a change in the bucket for any point, go back to the first instance and repeat the steps again.
The iteration ends when all the instances are iterated and none of the points change buckets.

This way there are a lot of iterations that start from the beginning of the dataset again and again every time when an instance changes buckets.

Any explanations would be helpful and please let me know if I my understanding for any of these methods is wrong.


texts: Which algorithm to apply for choosing the right point

texts:
Picture below shows 7 points around the origin. One of them has been selected by a human based on rules and experience and is colored red (the one in the lower left quadrant).

Now we have over 1000 of these sets of points and for each set a human has selected a single point. These conditions apply to all sets:

Each set has about 3 - 10 points
There are no outliers
Points can have positive and negative values
No mistakes were made when selecting a point

My question is: Does there exists a machine learning algorithm to learn from these sets and human made selections so it can automatically decide which point to select when a new set of points is given? This new set satisfies the first 3 conditions from above of course.

2 final remarks:

The example I gave is just a randomly constructed example by me to support the idea about points in a plane around the origin together with a selected one. In real life there might be more structure but for now I am curious and would like to know what is possible for this case.
Would variations be possible? Say it is about 2 selected points or you have circles with a given radius instead of points.


texts: Slight confusion on the learning process

texts:
Hi guys I have a slight confusion on the learning process of neural networks.
When the input layer receives inputs, goes through the hidden layers and then into the output layer. How does the neural network know that the outputs at the output layer are incorrect?
When the error is calculated at the output layer, it needs the predicted output which is fine but how is the actual output found?
Thanks


texts: Find the number of homes with 3000 sq ft , 3 number of bedrooms and 40 years of age?

texts:
there is a dataset like this
area   bedroom  age  price

2600        3    20  550000
3000        4    15  565000
...

Now the questions are

find the price of the house with 2500 sft and 4 bedrooms and 5 years old?

find the home with 3000sft and 3 bedrooms and 40 years old?

I know 1st question can be done with Linear Regression. How should I do second question ?


texts:
In my dataset i have 3 classes-> 0,1,2.
0(72k),1(13k)and 2(13K) in brackets are there count.
So whenever i try to predict them with any algorithm ,i observed that almost all the "2"'s are predicted as "0".
On little exploration i found that there are some rows where attributes of 0 and 2 are exactly same.
Any technique to tackle this issue?


texts: ML methods for prediction, using categorical variables and time

texts:
Most of the time series analysis tutorials/textbooks I found time series data, usually deal with continuous numerical variables. I am currently trying to solve a problem that deals with multivariate time series data, where the fields are all categorical variables.

Specifically, my data is a stream of alert data, where at each time stamp, information such as the alert monitoring system, the type of alert, the location of the problem, etc. are stored in the alert. These fields are all categorical variables.

My data is very similar to the one described in another question, so I used a similar description. However my question is a bit more general.

Given a chain of alerts, what is a good approach for using ML to predict the location and type of the next alert (at t(1) ), having knowledge of common alert patterns and the values for t(0), t(-1), t(-2), ...t(-n) ?


texts: How to handle a feature vector that could be variable length?

texts:
I would like to train a machine learning model with several features as input as X[] and with one output as Y. For example Every sample has a Data frame like this: X[0], X[1], X[2], X[3], X[4], Y
Let's say One sample the followings Data is only one value: X[0], X[1], X[2], X[4], Y This is normal machine training problem.
But now, if I would like to set X[3] multiple values for example sample 1 Data is:
X[0] | X[1] | X[2] |         X[3]         | X[4] | Y
 10  |  5   |   6  | [10, 20, 30, 40, 50] |  7   | 90

Data in sample 2 is:
X[0] | X[1] | X[2] |         X[3]         | X[4] | Y
 11  |  7   |   5  | [20, 30, 40, 50, 60] |  3   | 80

Is this possible to follow the normal machine training process and got a model which could calculate a sample with other example with Data like:
X[0]   | X[1] |  X[2]  |         X[3]         | X[4] | Y
 10.5  |  6   |   5.5  | [15, 25, 35, 45, 55] |  5   | ???

If the length for each X[3] is not long, it is possible to divide the X[3] into multiple new features, but if the length of X[3] is very long (len > 1000) with different distribution, making binary is also lead to too many new features. Is there any way to treat the X[3] directly without adding new features?
Really appreciate for your help.


texts: Training the parameters of a Restricted Boltzman machine

texts:
Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?

Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$.  The probability of $x$ is:
\begin{equation}
P(x) = \sum_h P(x,h) = \sum_h \frac{e^{-E(x,h)}}{Z}.
\end{equation}
The goal is to maximize the probability of $x$ conditional on the parameters of the model $\theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element  $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood:
\begin{equation}
\left\lbrace \hat{\theta} \in \hat{\Theta} :  N^{-1} \sum_{x_i} \frac{\partial  \log p(x_i)} {\partial \theta} = 0 \right\rbrace
 \end{equation}
and chose the one $\theta^\star \in \hat{\Theta} $ maximizing the empirical likelihood.  There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $\theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:
\begin{equation}
 \theta_{i+1} = \theta_{i} - \eta_i \frac{\partial  \log p(x_i)}{\partial \theta_i}
 \end{equation}
  The practice I learned in Hinton et al. (2006) and in Tieleman (2008)   is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!


texts: How to interpret feature importance (XGBoost) in this case?

texts:
I found two dominant features from plot_importance.  My dependent variable Y is customer retention (whether or not the customer will retain, 1=yes, 0=no). My problem is I know that feature A and B are significant, but I don't know how to interpret and report them in words because I can't tell if they have a position or negative effect on the customer retention. Is there a way to find that out or anything that helps make it clear?

Thanks.


texts:
Why my performance of new testing data is poor in terms of accuracy?
I have created a CNN model for the classification. Before submitting into to model, I have normalized the data and split it into training, validation, and testing set. All of them have high accuracy (more than 90%). But what if I put a new testing data into the model, the accuracy immediately drop down to around 70%. Can anyone answer this issue?
The story is that I want to predict what action the user is doing when the user is holding a sports device, and then I collect sensors data (time series) based on 10 users let's say (A,B,C,...,J). If I use 8 users (A to H) to training, and test it using (I and J), the accuracy shows around 70%. I have tried many combinations (train:8 users, test:2 users), the result perform 60% - 80% accuracy


texts: Looking for CART/ML model that works with relative data

texts:
I am a beginner at AI and ML. I have been given a dataset, where I have noticed the columns are relative to one another. So is there any CART or ML model that can work with relative data ?
For example in Decision Tree, the tree looks like :
  if X[0]<192:
     if X[1]>24:
       if X[2]<12:
       ...

I'm looking for a Decision Tree, that works like this :
  if X[0]>X[1]:
   if X[1]<X[2]:
      ...

Is there any such Machine Learning Model ?


texts: IS there any way we can add range in hyperparameter tuning of Decision Tree?

texts:
For example,
           "min_samples_leaf":[1,2,3,4,5,6,7,8,9,10],
           'criterion':['gini','entropy'],
            "max_features":["auto","log2","sqrt",None],
           "max_leaf_nodes":[None,10,20,30,40,50,60,70,80,90] }

but i want to add range of 1 to 100 in max deapth. is there any way i can do it?
Like shown below....
parameters={"max_depth" : range(1,100), ??????????
           "min_samples_leaf":[1,2,3,4,5,6,7,8,9,10],
           'criterion':['gini','entropy'],
            "max_features":["auto","log2","sqrt",None],
           "max_leaf_nodes":[None,10,20,30,40,50,60,70,80,90] }


texts: 50% accuracy on multiclass classification

texts:
I am trying to do a multiclass classification on a significant amount of output labels (1000).

I built a model using KNN. The accuracy given by accuracy = knn.score(X_test, y_test) is 0.5.

Does this mean that given an input, the model is able to predict 50% of the time which label the data belongs to? If yes, I would intuitively say that this is good, since randomly choosing a label would have a probability of 0.1%.


texts: how to do incremental deep learning on data stream that can adapt to constantly generated data points?

texts: I am currently trying to learn a deep learning model on a data stream, which constantly generate new data points over time.  The goal is to generate a real-time DL model that can well adapt to newly generated data while does not affect the prediction for old data too much.  I am wondering if anyone know what might be the best way to do this "incremental deep learning on data stream"?  Any information would be much appreciated, thank you!


texts: Fastest way to relearn machine/deep learning

texts:
I hope I came to the right place to ask this question.

Back when I was at collage I studied machine and deep learning in-depth. My whole programme was based on those areas. I knew all underlying maths, even today I know how to derive backpropagation for any feed-forward network. Well, maybe I would need to take a peek. But I still understand the math and I can follow without problems. Back then (2017) I was even doing something with and researching GANs which were novelty back then.

Basically, I was at the hotspot at that time. I was working with all sorts of algorithms, from logistic regression, SVMs, to MLPs, CNNs, RNNs (mostly LSTM) and was trying already mentioned GANs. Oh, heuristics too: GAs, tabu search, simulated annealing, etc. There was also some NLP involved too.

And then after college I went to gaming industry, heh. I was/am still, during that time, working with ML/DL (and some OpenCV) but mostly easy, toy projects (although one project was real life project, but it was easy, I had to extract written digits from paper and classify them).

So, my question is, what is the best (and fastest) way to get back on the track considering my, I would say, pretty strong background? I saw that Kaggle has some courses on their site, for example course on DL is estimated 4 hours, course on feature engineering is also 4 hours. That is not a lot of time, but I am afraid it would be too easy for me and consequently a waste of time.

What are some good resources to refresh/relearn ML/DL considering my background?


texts: How to see which transformation is the best

texts:
In R I have data and I want to make a regression analysis, finding a function that can fit the data.
So head(data) gives

promotion   new_users
39.5              100
36.1               79
 0.0               18

To find the optimal regression function that fitted data I plot the residuals of the regression model to see if the residuals are systematic close the zero line. But I do not know which transformation is the best one to use. Here I tried to do linear transformation, sqrttransformation and finally log-transformation.

lm.linear = lm(formula= data$new_users ~ data$promotion )
plot(resid(lm.linear), col="blue")

lm.sqrt = lm(formula= data$new_users ~ sqrt(data$promotion) )
plot(resid(lm.sqrt), col="blue")

lm.log = lm(formula= data$new_users ~ log(0.1+data$promotion ))
plot(resid(lm.log), col="blue")

If I simply just plot the data and the fitted regression function I can't see which regression function fittest the data best because they are very similar.
Which transformation is the best one and is there another way to find out what is the best transformation ?

Thanks.

To see if I can use poission regression model I type

model=glm(data$new_users ~data$promotion, familiy="poisson", data=data)

I use goodnes of fit to see if the model fits.

with(m1, cbind(res.deviance = deviance, df = df.residual,
  p = pchisq(deviance, df.residual, lower.tail=FALSE)))

I get a low p-value meaning that the model is a good fit?

Furthermore say we want to compare two poison regression models (one poission regr. model from data and another one from another dataset say data_new) and see if there is a significantly difference between the two, how would one do that? I assume one could use anova test to solve this.


texts:
I am trying to multiply two array in python 3.7 using numpy by using the following syntax:

array1 = np.array([{1,2,3,4},{5,6,7,8}])
print (array1)
array2=array1*array1
print(array2)

but this error arises

TypeError: unsupported operand type(s) for *: 'set' and 'set'


texts: Given a target vector and a feature vector, how to computer the weight

texts:
In page 13 of the slide, given $t$ and $X$ as following. I don't understand how we get $w$.

$$t=[t^{(1)},t^{(2)}, \ldots, t^{(N)} ]^T$$
$$X=\begin{bmatrix}1, x^{(1)} \\ 1, x^{(2)} \\ \vdots\\1, x^{(N)} \end{bmatrix}$$

Then:

$$w=(X^TX)^{-1}X^Tt$$


texts: Is this the correct way to apply a recommender system based on KNN and cosine similarity to predict continuous values?

texts:
My data is:
userID, gameID, rating (1.0 through 10.0)
First, I normalize the values the ratings of each row.
I use cosine similarity to create a similarity matrix where each cell represents similarity between a pair of userIDs (value 0.0 through 1.0).
For each unrated gameID, I find the 10 most similar users that have rated the gameID. The predicted rating is equal to the sum of each neighbors rating times similarity, then divided by 10 (number of neighbors).

This seems fine for finding the best predictions, just take the top N rating values, but for actually predicting the values it doesn't perform so well. Intuitively, the average of a group of similar userID's ratings would be an accurate prediction. When each rating is made smaller by multiplying by a value that is always less than 1.0, the predicted value is consistently smaller than expected.

This seemed to follow formulas that I found, but I feel like I missed something. I am using Python, Pandas, and Numpy.

Edit
I am seeing that this final value is sometimes referred to as a "weighted predicted rating". I am wondering how to "de-weight".


texts: Explanation for MLP classification probability

texts:
I showed some results of one implemented NN MLP model.

In the result, for classification of two categories, if I sum up the probabilities of both cats, them some sum would be greater than 1.

When I was asked why the sum is greater than 1, I gave the guess that the probability stands for confidence concept in mathematics.

Could anyone tell me am I right?


texts: Multiclass Classification with large number of categories

texts:
I am making a recommendation system (kind of) and I have to recommend the item a user is most likely to buy in his next purchase. Doesn't matter if he already bought this item.

Given this, I'm treating this problem as a multiclass-classification problem with 4000 categories (number of different items users can buy).

Searching in Wikipedia I found this link and decided to use the One vs -rest method. So I decided to train one random forest for each item using as covariates flags if the user bought each item before (so I have around 4000 covariates). Then I will decide a rule to decide the recommended item (something like the one which has the largest probability to be bought or the largest lift.)

My problem is that it's taking too long to train (5 to 10 min per item):

> 5*4000
[1] 20000
> 20000/60
[1] 333.3333
> 333.3333/24
[1] 13.88889

So in the best case it would take 2 weeks to train.

I would like to know if the method i'm using is right, and if there's another faster method to achieve this.


texts: What are the best practices to anonymize user names in data?

texts:
I'm working on a project which asks fellow students to share their original text data for further analysis using data mining techniques, and, I think it would be appropriate to anonymize student names with their submissions.

Setting aside the better solutions of a url where students submit their work and a backend script inserts the anonymized ID, What sort of solutions could I direct students to implement on their own to anonymized their own names?

I'm still a noob in this area. I don't know what are the norms. I was thinking the solution could be a hashing algorithm. That sounds like a better solution than making up a fake name as two people could pick the same fake name.possible people could pick the same fake name. What are some of the concerns I should be aware of?


texts:
I tried to create a manual RNN and followed the official PyTorch example, which tries to classify a name to a language. I should note that it does indeed work. I'm not using the final logsoftmax, since I use nn.CrossEntropyLoss() and that should apply that automatically (it gives exactly the same results).

class ManualRNN(nn.Module):
    def __init__(self, batchSize, inputSize, outputSize):
        super(ManualRNN, self).__init__()
        self.batchSize = batchSize
        self.inputSize = inputSize
        self.hiddenSize = 64

        self.hidden = nn.Linear(self.hiddenSize + self.inputSize, self.hiddenSize)

        #I tried using both, the first one is from the example suite
        #self.linear = nn.Linear(self.hiddenSize + self.inputSize, outputSize)
        self.linear = nn.Linear(self.hiddenSize, outputSize)
        self.tanh = nn.Tanh()

    def forward(self, x, hidden):
        x = x.view(1, -1)
        hidden = hidden.view(1, -1)
        combined = torch.cat( (x, hidden), dim=1)

        newHidden = self.tanh(self.hidden(combined))
        #output = self.linear(combined)
        output = self.linear(newHidden)
        return output, newHidden

    def InitHidden(self):
        return torch.zeros(1, self.hiddenSize)

My goal is to tie this out vs the internal RNN module, so I used tanh() for the hidden layer, since that's what the documentation shows for nn.RNN. I also tried making the input for the final linear layer only dependent on the hidden layer which seems to be what nn.RNN is doing, but I might be wrong - in either case I tried both.

Now I attempted to tie this out vs the nn.RNN module (or should I say cell?):

class TorchRNN(nn.Module):
    def __init__(self, inputSize, hiddenSize, outputSize):
        super(TorchRNN, self).__init__()
        self.inputSize = inputSize      #number of possible letters, i.e. 31
        self.hiddenSize = hiddenSize    #64
        self.outputSize = outputSize    #18 (number of classes)

        #this should be applying tanh() at each time step
        self.rnn = nn.RNN(input_size=self.inputSize, hidden_size=self.hiddenSize, batch_first=True)

        #note that here I am only using hiddenSize, because that's the output from nn.RNN ...?
        #that's why in the ManualRNN example above I tried the same
        self.linear = nn.Linear(self.hiddenSize, outputSize)

    def forward(self, x, hidden):
        output, hidden = self.rnn(x, hidden)

        output = self.linear(hidden)
        return output

    def InitHidden(self, batchSize):
        return torch.zeros(1, batchSize, self.hiddenSize)

I made sure to use only names with 5 letters, and using the final hidden layer as input, since I didn't want to further confuse the situation with pad_packings.

With enough iterations both, the ManualRNN and TorchRNN models perform similarly well. However, they don't tie out. There shouldn't be any randomness and I checked that the inputs are being fed in the same order. I am using the following to establish seeds (even though I am not using cuda on this machine):

np.random.seed(666)
torch.manual_seed(666)
torch.cuda.manual_seed_all(666)
torch.backends.cudnn.deterministic = True

What am I doing wrong? My understanding is that the example from the website isn't actually replicating the nn.RNN class properly. nn.RNN outputs seem to be hidden_for_each_k and last_hidden. The last element of hidden_for_each_k is the same as last_hidden. Consequently it seems that the only inputs available for the linear layer are last_hidden - which is not the same as in the tutorial, hence I changed the input for the linear layer in the ManualRNN to accept only the hidden layer. However, even that doesn't tie out. The values are completely different for basically any point in time.

So in short: Is there a way to go from "manual RNN" to "nn.RNN" so that the results tie out exactly? Any way you could show this to me would be appreciated (it doesn't have to be the same example).


texts:
I'm categorizing 30 types of clothes from the image using R-CNN Object Detection Library from tensorflow : https://github.com/tensorflow/models/tree/master/research/object_detection

Does color matter when we collect images for training and testing?

If I put only purple and blue shirts, I guess it won't recognize red shirts?

Should I gray scale all images to detect the types of clothes? :)


texts:
It is known that SGD iteration has huge variance.
Given the iteration update:
$$
w^{k+1} := w^k - \underbrace{\alpha \ g_i(w^k)}_{p^k},
$$
where $w$ are model weights and $g_i(w^k)$ is gradient of loss function evaluated for sample $i$. How do I compute variance of each update $p^k$?
I would like to plot it for each iteration and study its behavior during minimization process.


texts:
I have a dataset with most columns having Boolean values and categorical values.
A sample of it is:

Name    Country approved political
bbc.com  US       true   True
stackoverflow.com US true False
Number.com US      False False

...

Based of values above, I would like to determine if other websites have been approved or not.

My questions are:

does any heat/correlation map/matrix sense with categorical variables?

Would it be possible to predict if a website is approved or not (target variable), using caterogical values?

Is there any other model that should be preferable?

Thanks


texts:
I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with missing value techniques. Also cannot give some number because even 0 and -1 are valid values for temperature. How to deal with such data?

I am thinking of putting very big value for such columns which is not possible as temperature. Please suggest.


texts: How to split up my dataset in a train and testset, in order to prevent data leakage?

texts:
I realize that this could be considered a duplicate of this question, Is using samples from the same person in both trainset and testset considers being a data leakage?, where it is stated that "The testing data should not be linked to the training data in any way" in order to prevent data leakage.
However, how should I continue if it is not possible to split the dataset, in a train and testset, where there is no link between them in any way?
My dataset exists of 839 time series samples which i want to classify according to 14 classes. However, this is an unbalanced dataset as can be seen in Figure 1.
The dataset is composed of the recordings of 16 volunteers. Unfortunately, for a number of these classes, data is only available for a number of volunteers. Figure 2 shows that for some classes only data of one specific volunteer is available. In the figure, the number gives the amount of volunteers of which samples are available for that specific class.
I want to validate my machine learning models using a seperate testset. My initial idea was to leave the data of one volunteer out for the testset. But as stated above, this is not possible since for two classes there is only data available from one volunteer.
The second best approach that I could come up with is leaving the data from one volunteer (which has samples available for 12 of the 4 classes) out alongside a selected few samples of the two minority classes that are not present in the samples of this specific volunteer.
This means that my testset would consist of samples which are completely seperated from the trainset for 12 of the 14 classes. The samples of the testset for the 2 other classes would be from the same volunteer on which the models are trained (since there is only data available of one volunteer for these classes).
As stated above, I know that this will introduce some form of data leakage. Is there a good alternative that won't introduce data leakage (besides adding more data)?
If not, would the StratifiedKFold method from sklearn be a good alternative for validating my model?
(With the StratifiedKFold i would use a pipeline in order to perform preprocessing only on the training part of the folds.)
I would be really glad for an answer and some more explanation on data leakage for this specific case.
Class imbalance
Data availability of volunteers per class


texts: How to use scikit metrics for a statsmodel or vice versa?

texts:
Am working on binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3558.

I did refer this post but not sure whether it is updated now or anyone has any way to compute this metrics

Currently I use logit model as shown below

model = smm.Logit(y_train, X_train_std)
result=model.fit()
y_pred = result.predict(X_test_std)
print("Accuracy is ", accuracy_score(X_test_std, y_pred))  #throws error from here and all the line below
print(classification_report(X_test_std, y_pred))
print("ACU score is ",roc_auc_score(X_test_std, y_pred))
print("Recall score is",recall_score(X_test_std,y_pred))
print("Precision score is",precision_score(X_test_std,y_pred))
print("F1 score is",f1_score(X_test_std,y_pred))

The reason why I am trying to do this is because statsmodel has p-values, coeff, intervals etc and I was hoping to get the usual metrics through scikit metrics as shown above but it isn't accepted.

On the other hand, Through scikit logistic regression I can get usual metrics and coeff, but what about p-values, conf intervals? Is there anyway to do the reverse?

Can someone help me with this?


texts: NLP: how to organize this research into Tensorflow

texts:
I need a network with the following organization:

the input text, it is constant for training, for example "mama soap frame sponge" or the formulas of physics, math, chemistry and others
a series of questions (who, where, when, what, whom, what, for what), for example "who washed the frame?"
A series of answers (labels), for example, the answer "Mom".

How to do it with TensorFlow?

Thank you


texts: XGBoost predicting everything as null when sample weights are passed

texts:
I am trying to build an Uplift model using observational data. The data is consists of collections calls to customers and my objective is to predict the incremental probability due to the treatment (collection calls). I am using XGBoost algorithm for the same, and as I am using observational data, I need to do an Inverse Probability Weighting (IPW) to remove the bias caused due to the unavailability of Randomized Test and Control groups. Hence, I have predicted the probability of a customer being contacted using a logistic regression and used the probabilities to calculate IPW. Now, to remove the bias, I need to weigh each instance of my data with the IPW while building the final uplift model. Here's where I am facing the issue, when I pass these weights while training, post training xgboost predicts everything as either null or .5 for every data point in both test and training data. When I don't use these weights, the model runs fine and gives out proper probabilities. Below is the code I am using

X_train, X_test, Y_train, Y_test = model_selection.train_test_split(X, Y, train_size = 0.8,random_state = 42,stratify  = Y)
train_weights = X_train.sipw
test_weights = X_test.sipw
X_train.drop('sipw',axis=1,inplace=True)
X_test.drop('sipw',axis=1,inplace=True)
xgbmat_train = xgb.DMatrix(X_train,Y_train,missing=np.nan,weight=train_weights.values)
xgbmat_test = xgb.DMatrix(X_test,Y_test,missing=np.nan,weight=test_weights.values)
initial_params = {'learning_rate':0.1,'n_estimators':1000,'objective':'binary:logistic','booster':'gbtree','reg_alpha':0,
                'reg_lambda':1,'max_depth':5,'min_child_weight':1,'gamma':0,'subsample':0.8,'colsample_bytree':0.8,
                'scale_pos_weight':1,'missing':np.nan,'seed':27,'eval_metric':'aucpr','n_jobs':8,'silent':True}
xgb_1 = xgb.train(initial_params,xgbmat_train)
prob = xgb_1.predict(xgbmat_test)

I am going crazy thinking about why this is happening, as I had previously built a model (for a different country) using the same code as above (passing IPW as weights) and it gave me perfect results. While for this particular dataset, I am unable figure out why the model is returning such results. Any help will be deeply appreciated.


texts: issue with early-stopping on f1 score with imbalanced data

texts:
I have a highly imbalanced dataset with less than 0.5% of the minor class. Using Keras, I'm training DNN on the training set and evaluate performance on validation set. Loss function is binary_crossentropy.

I'm setting my early-stopping on f1 score, instead of validation loss. What I observe during training is f1 score fluctuate wildly up and down while validation loss is decreasing. I actually end up with a very low f1 score with early-stopping, although f1 score was many epochs back...

I'm confused about it. Am I supposed to do early-stopping on the performance metrics? Should we alway use validation loss for early-stopping criterion? thanks.


texts: loss/val_loss are decreasing but accuracies are the same in LSTM!

texts: I am trying to train a LSTM model, but the problem is that the loss and val_loss are decreasing from 12 and 5 to less than 0.01, but the training set acc = 0.024 and validation set acc = 0.0000e+00 and they remain constant during the training. Why this happening and how can I fix it?


texts: This might sound a weird question, but I could not find enough details in sklearn documentation about 'class_weight'. Can we first oversample the dataset using SMOTE and then call the classifier with the 'class_weight' option? As my testing set is highly imbalanced, I want to penalize misclassifications for minority classes. Thank you!


texts: Which method is implemented when using the periodogram method from the 'TSA' package on R? Is it Welch or Bartlett?


texts: Running DenseNet from cmd line and jupyter but vastly Different loss and accuracy

texts:
I’m running a DenseNet121 from Pytorch with the same exact code, same exact hyper parameters and same exact image sizes, once from a jupyter notebook and once using the command line via a python script. The notebook version gives me very good results (I have also cross checked this with the current SOTA and those results are in line with that) but the script version gives me very, very bad results. This isn’t about different results due to the randomness in the initialization - the script results are so bad, they simply cannot be true, based on my research. To get an idea of the difference:

Notebook: accuracy on test set ~0.95, training loss ~0.14
script: accuracy on test set ~ 0.62, training loss ~0.7

Does anyone have any idea why this discrepancy is happening?


texts: how to work with NLP with other features

texts:
My dataset looks like this

Sport_Type       City         Report_Text                                                 Labels
Ball             Toronto      Messi has been announced the best soccer player...          Soccer
Swimming         London       Todays new records in Butterfly Stroke & Backstroke...      Butterfly Swimming, Backstroke Swimming, Front Crawl
Ball             Chicago      Tennis and basketball along with football has...            Tennis, basketball, Soccer
Fighting         Sydney       Todays matches include boxing, judo, and...                 Boxing, Judo, Karate
Horse            Melbourne    Melbourne Cup is the race that stops the nation...          Horse Racing

I can build multi label model to identify labels in each Report_Text field.

but is there a way I can consider Sport_Type and City field in my model as it will help in improving results.

How can I use other features such as Sport_Type and City in NLP multi label model?


texts:
I was watching Andrew Ng's video on ResNets, and he mentioned that "And in theory, as you make a neural network deeper, it should only do better and better on the training set."
Here is my understanding of neural networks, as the model progresses through the model, the parameters it will learn will become more and more sophisticated, correct? Intuitively, it should be able to recognize/discover more detailed pattern information about the training set. Is my understanding correct?

Then why, in practice, does adding excessive layers to Neural Networks actually harm the performance of the model?
Thanks in advance.


texts:
Is it possible to implement an RNN layer with no nonlinearity in Pytorch like in Keras where one can set the activation to linear?
By removing the nonlinearlity, I want to implement a first-order infinite-impulse-response (IIR) filter with a differentiable parameter and integrate it into my model for end-to-end learning. I can obviously implement the filter in Pytorch but I thought using an inbuilt function may be more efficient.


texts: R: Calculations based on frequencies / grouped / aggregate data

texts:
I am trying to do simple calculations in R when no raw data but grouped data with frequencies is available only. This is the case when I have a large amount of records in a database, say a large SQL table, and then for given reasons GROUP BY and COUNT to aggregate instead of downloading the original table for analysis in R. As I understand, one could say in R that I'm talking about data in a table format.

To give a simple example:

> original=c(1,2,2,3)
> aggregate=table(original)
original
1 2 3
1 2 1

In the case I describe, only data structured like aggregate would be available as the result of my database query, usually in a data.frame similar to this:

|value|frequency|
|1    |1        |
|2    |2        |
|3    |1        |

What are standard approaches in R to work with this kind of data, particulary when it's the aim to run analysis on value? E.g., I may want to

Calculate Mean, Median, Quantiles
Plot a boxplot which should give the same results as when I would run boxplot(original)

I feel I'm just missing a basic concept of aggregated data processing in R, cannot imagine it's not something being build in a native way in R.

The only idea I could find googling this was basically an approach to let R create original based on the frequency information.


texts: Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator?

texts:
So, I am having a play around with some tree based algorithms from Spark's mllib. The code I have is here;

from pyspark import SparkConf
from pyspark import SparkContext
from pyspark.sql import SparkSession
from pyspark.sql.functions import mean
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

from pyspark.ml import Pipeline
from pyspark.ml.classification import (RandomForestClassifier, GBTClassifier, DecisionTreeClassifier)
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

conf = SparkConf()
conf.set('spark.logConf', 'true').set("spark.ui.port", "4060")

spark = SparkSession.builder.config(conf=conf).appName("Gradient Boosted Tree").getOrCreate()

data = spark.read.parquet('/mydata/location)

def yt_func(x):
    if x <= 10:
        yt = 0
    else:
        yt = 1
    return yt

yt_udf = udf(yt_func, IntegerType())

data = data.withColumn('yt_1',yt_udf(data['count']))

datasub = data.select('feature1', 'feature2',
                      'feature3', 'feature4',
                      'feature5', 'feature6',
                      'feature7', 'feature8',
                      'feature9', 'feature10',
                      'feature11','feature12',
                      'feature13')

datasub = datasub.na.fill(0)

from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols = ['feature1', 'feature2',
                      'feature3', 'feature4',
                      'feature5', 'feature6',
                      'feature7', 'feature8',
                      'feature9', 'feature10',
                      'feature11','feature12',
                      'feature13'], outputCol = 'features')

output = assembler.transform(datasub)

finaldata = output.select('features','yt_1')

train_data,test_data = finaldata.randomSplit([0.7,0.3])

finaldata.show(20)

dtc = DecisionTreeClassifier(featuresCol='features',labelCol='yt_1')
rfc = RandomForestClassifier(featuresCol='features',labelCol='yt_1', numTrees=70)
gbt = GBTClassifier(featuresCol='features',labelCol='yt_1')

dtc_model = dtc.fit(train_data)
rfc_model = rfc.fit(train_data)
gbt_model = gbt.fit(train_data)

dtc_preds = dtc_model.transform(test_data)
rfc_preds = rfc_model.transform(test_data)
gbt_preds = gbt_model.transform(test_data)

dtc_preds.show()
rfc_preds.show()
gbt_preds.show()

accuracy_eval = MulticlassClassificationEvaluator(metricName = 'accuracy', labelCol='yt_1')
recall_eval = MulticlassClassificationEvaluator(metricName = 'weightedRecall', labelCol='yt_1')

print 'dtc accuracy:', accuracy_eval.evaluate(dtc_preds)
print 'dtc recall', recall_eval.evaluate(dtc_preds)

print 'rfc accuracy:', accuracy_eval.evaluate(rfc_preds)
print 'rfc recall', recall_eval.evaluate(rfc_preds)

print 'gbt accuracy:', accuracy_eval.evaluate(gbt_preds)
print 'gbt recall', recall_eval.evaluate(gbt_preds)

When I run this I get the following;

dtc accuracy: 0.98596755767033761
dtc recall: 0.98596755767033761

rfc accuracy: 0.98551077243825225
rfc recall: 0.98551077243825225

gbt accuracy: 0.98624595624862965
gbt recall: 0.98624595624862965

What is confusing me here is why I am getting the same values for the accuracy and the recall.... they are EXACTLY the same. Surely this isn't correct....??

Any ideas?


texts: Pandas Excel writer update existing excel file

texts:
How do I update an excel file and not overwrite it?

#Run function

query_data(atlantic_stones)
query_data(aircool_north)
query_data(aircool_south)

#Create some Pandas dataframes

dfastones = query_data(atlantic_stones)
dfanorth = query_data(aircool_north)
dfasouth = query_data(aircool_south)

#Create a Pandas Excel writer using XlsxWriter as the engine.

writer = pd.ExcelWriter('data_repository.xlsx', engine='xlsxwriter')

dfastones.to_excel(writer, sheet_name='atlantic_stones')
dfanorth.to_excel(writer, sheet_name='aircool_north')
dfasouth.to_excel(writer, sheet_name='aircool_south')

#Close the Pandas Excel writer and output the Excel file.

writer.save()


texts: Better way of writing following Pandas Python code

texts:
Following code is given, we need to sum up the medals.

import pandas as pd

# Defining the three dataframes indicating the gold, silver, and bronze medal counts
# of different countries
gold = pd.DataFrame({'Country': ['USA', 'France', 'Russia'],
                         'Medals': [15, 13, 9]}
                    )
silver = pd.DataFrame({'Country': ['USA', 'Germany', 'Russia'],
                        'Medals': [29, 20, 16]}
                    )
bronze = pd.DataFrame({'Country': ['France', 'USA', 'UK'],
                        'Medals': [40, 28, 27]}
                    )

I wrote this working solution. However, it feels very un-pythonic:
And I feel there is a better way of approaching this.

df = gold.set_index('Country')["Medals"].add(silver.set_index('Country')["Medals"], fill_value=0).add(bronze.set_index('Country')["Medals"], fill_value=0)
df = pd.DataFrame(df.sort_values(ascending=False))
print(df)```


texts: How to determine individual contribution when data is aggregated by team

texts:
I want to know the individual contribution of each salesperson in terms of actual sales where the dataset I have aggregates sales data by sales "team".

The dataset looks as follows:

Month | # of Sales | SalesPerson1 | SalesPerson2 | SalesPerson3
1 | 25 | Fred | Susan | Tom
1 | 12 | John | Wayne | Connor
2 | 16 | Fred | John | Wayne
2 | 11 | Susan | Tom | Winona
3 | 34 | Connor | Susan | Tom

Is there a way to predict the individual contribution of any given salesperson based on the data above where the membership of any given sales team can change from one month to the next?

I want to be able to say that for any given sales person, they close x # of sales per month.

Thanks!


texts:
If I want to fit a nonlinear regression model with some parameters like $\sigma^2$(where $\sigma$ is the standard deviation, which is positive), how can I guarantee that $\hat{\sigma}$ is positive?

I mean, if I use maximum likelihood(and the model only have square term of $\sigma$), how does the optimization method know $\sigma$ is positive? (well, it seems OK that the result estimation of $\sigma$ is negative, but it looks weird, and this is why I ask this question.)


texts: Dealing with frequency in neural network input

texts:
I have a name database with frequency of a name. Eg:

Madura,1064

I want to use this frequency parameter along with the name, as an input to my neural network since it indicates how strong a particular name is. But this frequency value is only available in the training dataset.

How can I provide frequency, something like a weight for my input, to the neural network?


texts: What is the best Big-Data framework for stream processing?

texts:
I found that Apache-Storm, Apache-Spark, Apache-Flink and TIBCO StreamBase are some powerful frameworks for stream processing. but I don't know which one of them has the best performance and capabilities.

I know Apache-Spark and Apache-Flink are two general purpose frameworks that support stream processing. but Apache-Storm and TIBCO StreamBase are built for stream processing specially. Is there any considerable advantage between these frameworks?

Thanks


texts: Should I prevent individuals mating with themselves in EAs?

texts:
In evolutionary algorithms, should I always avoid individuals mating (crossover) with themselves? That is, should I prevent the selection algorithm from selecting a single parent twice to produce a child with itself?

I'm interested in the general case as well as for tournament selection.


texts:
I am learning linear regression right now.
In the most of the examples of implementation of this method, which I found, gradient descent is used.

Is there a better way to optimize linear regression than gradient descent?


texts: Visualization methods in R to examine correlation of labels against response

texts:
Question
What are some good plotting methods in R for examining the relationship between a target variable and various explanatory variables? In particular, I'm looking for visualization techniques that scale to more variables than the traditional scatterplot matrix.
More details
The scatterplot matrix is a great tool for visualizing pairwise relationships between variables. For example, with the swiss dataset in R, we can easily plot a matrix of scatterplots.
library(datasets)
data(swiss)
plot(swiss[1:3])

which yields

I am interested in the case where I want to predict some response, say Fertility using some combination of explanatory variables. I want to closely examine how each explanatory variable correlates with Fertility. If I have many columns in my dataframe, using plot(swiss) becomes unwieldy.
For example, the following plot (generated following instructions here) shows pairwise correlations for all columns in a dataframe. If I could plot something like this but only showing correlations between Fertility and other columns, that would be useful.
library(datasets)
data(swiss)
plot(swiss[1:3])

library(devtools)
library(inspectdf)
library(tidyverse)
library(readr)

show_plot(inspect_cor(swiss))

which yields


texts: Trade-off between number of channels and size of convolutional filters

texts:
As far as I understand, the common practice in the modern CNN architectures is to use a smaller convolutional filters, but deeper networks with more channels.
One of the reason behind this is that one can have less parameters with the same receptive field. For instance, one can have a $5 \times 5$ receptive field in the following cases:

$5 \times 5$ convolutional filter : $25 + 1 = 26$ parameters
two subsequent $3 \times 3$ convolutional layers : $2 (3 \times 3 + 1) = 20$ parameters
two subsequent $5 \times 1$ and $1 \times 5$ convolutional layers $2 (5 \times 1 + 1)  = 12$ parameters

I the latter 2 cases, one can also add a non-linearity between the layers. However, I wonder, whether the deeper architectures would be always advantegeous - because the larger filter may extract more sophsitcated interaction between features on particular scale and is more expressive. Also , probably, one can achieve the same quality with larger filters with the use of smaller number of them.
There is a recent paper - https://arxiv.org/pdf/2101.10143.pdf - where larger filter are preferred in terms of spectral characteristic of the data and reduction of some spectral leakages and artifacts.
Is there theory, with a certain level of rigor, which strategy - larger filters/shallower networks - smaller filters/deeper networks is preferable for a particular kind of problem?
I would strongly appreciate references to papers and research in this field!


texts: Making sense of loss and accuracy curves

texts:
This is an issue that I have come across over and over again. Loss (cross-entropy in this case) and accuracy plots that do not make sense. Here is an example:

Here, I’m training a ReNet18 on CIFAR10. Optimizer is SGD with 0.1 learning rate, 0.9 Nesterov momentum, 1e-4 weight decay. The learning rate is decreased to a ⅕ at epochs 60, 120, 160.

Initially the curves are all nice and dandy; meaning training and
validation loss are decreasing and accuracies are increasing.
Around epoch 65~70 you see signs of overfitting; as val. loss starts
increasing and val. accuracy starts decreasing (the red box). There
is still nothing strange here.

Now there are two things that don’t make sense to me:

After epoch 120 (where LR is decreased) val. loss and accuracy start
improving for a couple of epochs (the green box). Why would
decreasing the learning rate suddenly improve validation performance
of a model that was already overfitting?! I would expect the drop in
LR to actually accelerate overfitting.

After epoch ~125 (the blue box) loss starts going up but accuracy keeps
improving. I understand that loss could go up while accuracy stays
constant (by the model getting more confident in its wrong
predictions or less confident in its correct predictions). But I
don’t get how accuracy can improve while loss goes up.


texts:
I have a dataset where each row is a interaction of a user with a content.
I have user's features to represent the user (each user is uniquely represented through user.id):

user.id, user.nationality, user.company, user.role

and content's feature:

content.id, content.type, content.activity..

My goal is to use ML techniques to predict a content (given by a user.id).
My main problem is that, for a new prediction, I have the features of the user, but I do not have the features of the new content.


texts: (De-)Scaling/normalizing input and output data inside Keras model as layer

texts:
I am building a 2-hidden layer MLP using Keras.
I'm using a SciKit learn wrapper to be able to use the GridSearchCV functionality.
My sample-size is limited, forcing me to use K-fold verification as well for trustworthy results.

However, it is to my understanding that for every iteration in the K-fold validation, input data should be scaled (and output descaled) only using the training data. This requires the scaling to be performed inside the Keras model.

In order to have understandable results, the output should than be transformed back (using previously found scaling parameters) in order to calculate the metrics.

Is it possible to

Z-score standardize my input data (X & Y) in a normalization layer (batchnormalization for example)
Transform the output layer back (before calculating metrics), using scaling parameters found in 1.

I've looked at the batchnormalization functionality in Keras, but the documentation mentions: "During training time, BatchNormalization.inverse and BatchNormalization.forward are not guaranteed to be inverses of each other because inverse(y) uses statistics of the current minibatch, while forward(x) uses running-average statistics accumulated from training." Which seems like this prevents the functionality from being used in this manner.

Does anybody have a known solution or function for this?


texts:
I'm trying to find out how AI can help with efficient customer service, in fact call routing to the right agent. My usecase is given context of a query from a customer and agents' expertise, how can we do the matching?

Generally, how is this problem solved? What sub-topic within AI is suitable for this problems? Classification, recommender systems, ...? Any pointers to open-source projects would be very helpful.


texts: Printing all files' names from a folder (Kaggle kernel)

texts:
train_dir= '../input/train'
for file in os.listdir(train_dir):
    print(file)

I am trying to get all the file names in a directory called "train", from a Kaggle dataset.

However, using the above code I just get a return of the folder "train", instead of all the images in the folder "train".

If I repeat the code above using:

data_dir = '../input'
for file in os.listdir(data_dir):
    print(file)

Then, it works correctly, it outputs the folder "train", the file "train.csv", the file "sample_submission.csv" and the folder "test" correctly.

Hence, I am quite puzzled why it doesn't work in the first case.

Thanks a lot. I am working directly on the Kaggle kernel by the way.


texts: LSTM predicting expected value and standard deviation with one-hot encoding

texts:
I am implementing LSTM to a time series prediction which requires predicting both the expected value (i.e. the mean of the prediction), and the standard deviation (i.e. the interval that the future value should fall into).

I have considered that one-hot encoding can return the possibility of the prediction of each category, which can then be used to both find the mean and the S.D. of the prediction, for example if the output of the LSTM is

Category  | Possibility
[0,5)     |    0.1
[5,10)    |    0.2
[10,15)   |    0.4
[15,20)   |    0.3

I then conclude that the expected value of the prediction is
(2.5x0.1 + 7.5x0.2 + 12.5x0.4 + 17.5x0.3 = 12)

and the S.D. is find in the similar manner.

My question is, is this method a valid method to find the mean and the S.D. of the prediction, why / why not?


texts:
I have a large dataset which can't be loaded in memory, hence I decided to use incremental learning using Xgboost. What I have done currently is:

Tuned num_boosting_rounds using a chunk of data
Set early stopping rounds to a value (60) << num_boosting_rounds to prevent overfitting

When my training code runs, it runs for a 1000 rounds on the first two chunks optimizing the loss function. It then stops at 60 rounds for each subsequent chunk as the best value for loss function was observed in the 1000th round in the 2nd Chunk. Is this the correct way to configure the Incremental model? Would this result in my model being sub-par owing to early stop for a majority of training chunks.

for idx,df in enumerate(df_pointer):
    num_round = 1000
    early_stopping_rounds = 60
    param = {'max_depth':5, 'eta':0.02, 'silent':1, 'objective':'binary:logistic', 'eval_metric':'logloss', 'max_delta_step':4, 'scale_pos_weight': 4}
    dtrain, deval = getDMatrixSplit(df)
    watchlist  = [(deval,'eval')]
    bst = xgb.train(param, dtrain, num_round, watchlist,
    early_stopping_rounds=early_stopping_rounds, xgb_model=xgb_model)
    xgb_model = self.model_path +'/xgb_%s_%s.model'%(ml_algo, idx)
    bst.save_model(xgb_model)


texts:
I am currently doing a course in tensorflow in which they used tf.one_hot(indices, depth). Now I don't understand how these indices change into that binary sequence.

Can somebody please explain to me the exact process???


texts: Measuring the similarity between a numeric data matrix and one or more categorical variables?

texts:
Given a numeric data matrix $A$ of size $n \times p$, which each row represents an observation along $p$ variables, and a second categorical data matrix $M$ of size $n \times z$, where each row represents some categorical description of the data in matrix $A$ (i.e. metadata), how can we measure the correspondence between the variables in $M$ and the data in $A$?

The goal is to determine which of the variables in $M$ best reflects the relationship of the data points in matrix $A$, where each variable in $M$ is some imperfect / estimated feature of the data matrix matrix observations.

So if one were to imagine clustering the data in matrix $A$, the goal is to find which of the variables in $M$ best "fits" the clusters such that the levels of the variable separate well across the clusters in $A$.

One more consideration: the variables in $M$ may contain different numbers of levels, e.g. $M_1$ may take on one of three values while $M_2$ may take on one of ten values, and so on.

A couple approaches I've considered are:

Multinomial logistic regression
Clustering the data in $A$ and measuring the similarity between the data partitioning and each variable in $M$.

For example, if we pretend that the first four columns in the iris dataset (Sepal Length, Petal Width, etc.) are our numeric data matrix, and the last column (Species) is one of the categorical variables we are interested in comparing the data to, we could do something like:

Multinomial Logistic regression

If I train a logistic regression model using only a single variable from $X$ (or, something like PC1 from a PCA-projected version of the data), then I can compute a pseudo-$R^2$ and get something reasonable like (R example):

null <- glm(factor(iris$Species) ~ 1, family = 'binomial')

mod <- glm(Species ~ Sepal.Length, data = iris, family = 'binomial', maxit = 100)

# pseudo-rsquared
max(as.numeric(1 - logLik(mod) / logLik(null)), 0)

# 0.6238031

(In the actual analysis, this process would be repeated for each of the categorical variables in $M$ using a for-loop..)

However, when I attempt to train a model using the full data (or a large enough number of principle components from the PCA-projected version of the data), then the model seems to overfit the data, resulting in a pseudo-$R^2$ value of "1":

mod <- glm(Species ~ ., data = iris, family = 'binomial', maxit = 100)
#Warning message:
#glm.fit: fitted probabilities numerically 0 or 1 occurred

max(as.numeric(1 - logLik(mod) / logLik(null)), 0)
# 1

On the other hand, by first clustering the numeric data, the problem then becomes one of measuring similarity between two categorical variables with differing levels, although this is also not ideal since we are losing a lot of information in the process of going from a numeric data matrix with $p$ columns, to vector of cluster assignments.

Finally, I could perhaps try and "brute force" the problem by clustering the data into some number of clusters and then computing ratio of variable levels within each cluster, compared with the expected ratio of levels if they were randomly distributed across clusters, however, this seems like overkill.

Does anyone know if there is a better way to approach this problem?


texts: ARMA/ARIMA on energy forecasts timeseries: strange prediction

texts:
I'm trying to use ARMA/ARIMA with the statsmodel Python package, in order to predict the gas consumption. I tried with a dataset of this format:

Using only the gas column.

from pandas.tseries.offsets import *

arma_mod20 = sm.tsa.ARMA(januaryFeb[['gas [m3]']], (5,3)).fit()
predict_sunspots = arma_mod20.predict('2012-01-13', '2012-01-14', dynamic=True)
ax = januaryFeb.ix['2012-01-13 00:00:00':'2012-01-15 22:00:00']['gas [m3]'].plot(figsize=(12,8))
ax = predict_sunspots.plot(ax=ax, style='r--', label='Dynamic Prediction');
ax.legend();

Why is the prediction so bad?


texts: How does an encoder-decoder network work?

texts:
Let's say I trained an encoder-decoder network on a cat dataset using reconstruction error as loss function. The network is fully trained and the decoder is able to reconstruct good cat images.

Now what if I use the same network and input a dog image. Will the network be able to reconstruct dog image or not?


texts: Evaluating model with categorical target variables

texts:
I converted all the numeric target variables of MNIST dataset into categorical variables. So, 0 became zero, and so on. Next, I tried the following model on this dataset:
from sklearn.linear_model import SGDClassifier
from sklearn.multiclass import OneVsRestClassifier

sgd_clf = OneVsRestClassifier(SGDClassifier(random_state=42))
sgd_clf.fit(X_train, y_train)

When I try sgd_clf.predict(X_test), I get an array of categorical values. I want to understand how to evaluate my model since confusion_matrix is not working on this.


texts: How to correctly label images with multiple objects

texts:
I have 3 types of images:
A: Images of apples
B: Images of bananas, however some of these banana images also contain apples in the observable background
Is it enough to just label the bounding boxes of the individual items or MUST I label images with both in?
Rephrasing the above question, do I have to label all label-able objects in a scene, or can I leave some items out? How will this effect accuracy?
Will my model (YOLO) still be able to detect multiple objects in scene, having only been trained on images with one class label in scene?


texts: I have 17k vectors, each having 6 points, I want to cluster the vectors based on nature of points, eg. linearly increasing into one cluster, convex in another cluster, concave in another, decreasing in another and so on. What would be a good strategy to do this? I need it to detect outlier vectors.


texts: Simple Markov Chains Memoryless Property Question

texts:
I have a sequential data from time T1 to T6. The rows contain the sequence of states for 50 customers. There are only 3 states in my data. For example, it looks like this:

        T1  T2  T3  T4  T5  T6
Cust1   C   B   C   A   A   C

My transition matrix X looks like this:

    A   B   C
A   0.3 0.6 0.1
B   0.5 0.2 0.3
C   0.7 0.1 0.2

Now, we see that at time T6 the state is at C which corresponds to c=[0 0 1] vector.
I am now predicting T7 by doing the matrix multiplication: c * X which gives me [0.7 0.1 0.2]. Based on this, I decide that the state at T7 would be A (highest prob. value).

For T8, I use the result of the probability vector I got above and do [0.7 0.1 0.2]*X = [0.4 0.44 0.14]  and decide that the state is B.

My question is: Am I doing something wrong? Am I contradicting the memoryless property of the Markov Chains?


texts: Data transformations in hierarchical classification

texts:
I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy
as described in the A survey of hierarchical classification across different application domains:
E.g. if we have the classes 1.1, 1.2, 2.1, 2.2, 2.3 then in the first level we use all the training set to train a classifier to distinguish between class 1 (1.1,1.2) and 2 (2.1,2.2,2.3),
at the second level we use two multiclass classifier the first one to classify between 1.1 and 1.2 using as training set only the data belonging to these classes and the second classifier for the rest.
Should any data transformation (e.g. scaling, tf-idf) that we do to the data happen at each level of the classifier?
I.e. since at the first level the tf-idf vectors are created by fitting to the whole training set, can we use them at the second level or should we fit to the new training subsets?


texts:
I am doing a project and I am facing a problem I have no idea how to solve.
I have a data series of stock prices, and, based on this series I obtained another series, of so-called rentabilities (which technically are relative change indexes (Rn/R(n-1) - 1).

The thing is the following: when applying to this series the coefficient of variance I obtain the value of 123%. Has it any sense?

My series look in the following manner: most of my data (for rentabilities) is focused around the mean, which is pretty plain to see on a box-plot (leptokurtic and normal distribution), but it has A LOT of outliers, in both directions.

Then I thought about applying the index Rn/R(n-1) for the calculation of CV, and I obtain 2,3%.

The fact is, both  indexes show modification. I understand that mathematically, the standard deviation does not change, only the mean which gets translated with a unit. But, as both indexes show modification in my variable stock-prices, why should CV be so different?

Any ideas with my 123% CV? What should I do?


texts: which loss function (if any) optimizes the calibration graph

texts: The calibration graph is the predicted versus actual probability(see http://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html). Is it possible to optimize the linearity of that curve in terms of a loss function? Does the log-loss actually optimizes this curve in terms of KL divergence?


texts:
How to write "OR" in this example?

DATA a1;
set a;
if var1=1 OR 2;
run;

P.S. va1 is the categorial (with categories: 1, 2, 3)


texts: Should I update action value functions when there is no change?

texts:
Suppose there is a website and the decision-maker wants to recommend some products to each customer visiting the website. Customers visit the websites in which the time interval between two consecutive visits is a random variable (in some cases it might take one time unit, in some cases it might take several time units, like the concepts in SMDP). When a customer visits the website but there is no purchase, there would be no reward. Should I update the $Q$ function even when there is no reward? In this situation, I should set $R_{t+1}=0$ and use any updating method like Q-learning? Moreover, if I consider eligibility trace and $\tau$ represents the time units between two-arrival, should I update eligibility trace like $E(s,a)=\gamma^\tau \lambda E(s,a)+1$?

Thanks


texts: what is difference between the DDQN and DQN?

texts:
I think I did not understand what is the difference between DQN and DDQN in implementation.
I understand that we change the traget network during the running of DDQN but I do not understand how it is done in this code.

We put the self.target_model.set_weights(self.model.get_weights())
In implementation of DDQN and this is added when action is finished for DQN  https://github.com/keon/deep-q-learning self.target_model.set_weights(self.model.get_weights()) is added to DQN in order to change the DQN to DDQN! But this happens when we are going out from running by break! There for there is no difference between them!

What is wrong in my mind? (Maybe the difference will be at test? Is this code for train and the test is done with setting the exploration rate=0  and then run this for just one episode with new weight we found? Is it right?

Therefore, what is difference between the presented DQN and DDQN of this link.


texts: How to make tool that's robust to user-generated typos?

texts:
Background: A client is generating datasets (excel files). They asked I make an app to analyze the datasets, e.g. summary tables and figures. I'm doing this in R shiny.

Problem: There are many typos in the user-generated data. For example, Alice is sometimes entered as Alce, alice, alice., Aalice, or Jennifer (middle name). Some errors are easy to correct for, like trailing whitespace and upper/lowercase. Others are nearly impossible, like knowing Jennifer (middle name) is really Alice.

How can I make my tool robust to errors in the data when I don't have control over data entry?


texts:
I've just embarked on a project to recommend similar candidates to employers for a website that offers employers the ability to search its CV database.

My background is in engineering research on biomedical images - from simple predefined feature extraction to CNNs.

I'm about to start with a simple bag of words approach standardised by the frequency of appearance and see where that gets me.

Has anyone done anything similar before and has suggestions for which algorithms might give good results?

The data is very well structured, separated into seperate sections for each previous employment etc.


texts: Removing correlation between independent variables

texts: If there are two continuous independent variables that show a high amount of correlation between them, can we remove this correlation by multiplying or dividing the values of one of the variables with random factors (E.g., multiplying the first value with 2, the second value with 3, etc.). We would be keeping a  copy of the original values of the independent variable whose values have been transformed for proper comparison of the original values with the prediction results received. Can this be considered as an alternative to dropping one of the highly correlated variables?


texts: Do batch GD and stochastic GD give the same results?

texts:
If a neural network is trained on a dataset of M samples for N epochs, do batch GD and SGD give the same result? Is SGD is faster because utilize the hardware better?
I am asking because I figured out that both (batch GD & SGD) give mathematically the same result, but I read SGD avoid local minima, how can this can be true if SGD & batch GD give the same result!?


texts: I see that there is an XGBoost package as well as XGBoost option in the "train" function in Caret. What is the difference between these two options, is it that they provide different settings? More generally, how would one choose between using the "train" function and a standalone package for the method?


texts: Visualization: Changing group memberships over time

texts:
I am looking for an interactive visualization. I have store classification based on their sales (sales buckets) for multiple quarters. I want to visualize

The size of each bucket for every quarter
The contribution towards
sales from each of the buckets in the previous quarter (essentially
capturing upgrade/downgrade of stores from one bucket to the other)

I tried Sankey diagrams but the issue is that it only helps me visualize the count of stores moving (changing buckets) over time.

Edit 2

The numbers in the example below and the diagram might not match

I want to capture the contribution of sales from previous bucket to the next as well.

Q1 buckets|Q1 Sales
0-5tr|1831
5-10tr|561
10-20tr|452
20-50tr|400

Q2 buckets|Q2 Sales
0-5tr|1600
5-10tr|200
10-20tr|50
20-50tr|69

Here, tr is an abbreviation for turnover
Similarly, I will have the number of stores in each bucket for both quarters. Although, I can visualize the movement of stores (using sankey) from one bucket to the other by creating a nXn matrix (n being the number of buckets) and each value i,j representing the number of stores moving from bucket i (in Q1) to bucket j (in Q2), I am not able to do so for its sales.


texts: What does the Ip mean in the Bayesian Ridge Regression formula?

texts:
From http://scikit-learn.org/stable/modules/linear_model.html#bayesian-ridge-regression, they gave the bayesian ridge distribution as this:

$p(w|\lambda) = \mathcal{N}(w|0,\lambda^{-1}{I_{p}})$

And there is a variable $I_p$ but it's unexplained what does the $I_p$ refer to?

Also, the variable $\mathcal{N}$ is unexplained but I'm not sure whether I've guessed correctly but is that the Gaussian prior as described in the Bayesian regression section above the Bayesian Ridge?


texts: What is the intuition of using clustering for performing feature engineering in machine learning tasks?

texts:
I am trying to implement the research paper Combining Boosted Trees with Metafeature Engineering for Predictive Maintenance. The paper has a section called meta feature engineering where they have used hierarchical clustering to create features. The paper says:

The third method we used to analyze the outliers in the dataset is based on an
hierarchical Agglomerative Clustering algorithm [5].
Hierarchical Agglomerative Clustering starts with Z groups (Z being the
number of observations), each initially containing one object, and then at each
step it merges the two most similar groups until there is only one single group,
containing all data.
The rationale for this method is that the last observation that are merged
might still be significantly different from the group they are merged into. By
definition outliers are different cases and will typically not fit well into a cluster,
unless that cluster is comprised by other outliers itself. Yet again, since these
are not ordinary data points, we do not expect them to form large groups.

I am unable to understand the authors' intuition behind doing this.
The problem I am trying to solve and the paper is related to is the IDA-2016 competition dataset. You can find more about the competition here


texts: Usage of KL divergence to improve BOW model

texts:
For a university project I chose to do sentiment analysis on a google playstore reviews dataset. I have obtained decent results classifying the data using BOW model and an ADALINE classifier.
I want to improve my model by incorporating bigrams relevant to the topic (Negative or Postive) in my features set, and I have found this paper  which uses KL divergence to measure relevance of unigrams/bigrams relative to a topic.
Only I have trouble understanding what C refers to in the (2.2) equation, does it refer to the unique words associated with topic C, the set of documents on a topic, the words in a document?


texts: using dataset to classifying and labelling another unlabeded dataset

texts:
I collect a collection of posts from Facebook and I use a published sentiment datset to labeling my collected dataset.
is this a right technique and what its name is this transfer-learning ?


texts:
I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?

# Univariate Selection
import pandas
import numpy
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

# feature extraction
test = SelectKBest(score_func=chi2, k=4)
fit = test.fit(X, y)
# summarize scores
numpy.set_printoptions(precision=3)
print(fit.scores_)

# Feature Extraction with RFE
from sklearn.feature_selection import RFE

# feature extraction
model = LogisticRegression()
rfe = RFE(model, 15)
fit = rfe.fit(X, y)
print("Num Features: %d" % fit.n_features_)
print("Selected Features: %s" % fit.support_)
print("Feature Ranking: %s" % fit.ranking_)

# VarianceThreshold
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
sel.fit_transform(X)

idxs = sel.get_support(indices=True)
np.array(X)[:, idxs]


texts: Can we use both validation loss and cross validation at the on CNN?

texts:
I have seen that validation loss is used for avoiding overfitting of the training set and cross-validation is used for generalized the models' results.

Are they use for similar purposes or results? If not, then how can I use both validation loss and cross-validation combined on CNN?


texts: Question on decision tree in the book Programming Collective Intelligence

texts:
I'm currently studying Chapter 7 ("Modeling with Decision Trees") of the book "Programming Collective intelligence".

I find the output of the function mdclassify() p.157 confusing. The function deals with missing data. The explanation provided is:

  In the basic decision tree, everything has an implied weight of 1,
  meaning that the observations count fully for the probability that an
  item fits into a certain category. If you are following multiple
  branches instead, you can give each branch a weight equal to the
  fraction of all the other rows that are on that side.

From what I understand, an instance is then split between branches.

Hence, I simply don't understand how we can obtain:

{'None': 0.125, 'Premium': 2.25, 'Basic': 0.125}

as 0.125+0.125+2.25 does not sum to 1 nor even an integer. How was the new observation split?

The code is here:

https://github.com/arthur-e/Programming-Collective-Intelligence/blob/master/chapter7/treepredict.py

Using the original dataset, I obtain the tree shown here:

Can anyone please explain me precisely what the numbers precisely mean and how they were exactly obtained?

PS : The 1st example of the book is wrong as described on their errata page but just explaining the second example (mentioned above) would be nice.


texts: PyTorch Faster R-CNN ResNet50 - support for occlusion?

texts:
Currently I'm using the PyTorch model Faster R-CNN ResNet50.  I instantiate this as follows:

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

Then I more or less perform standard object detection, similar to this example on the PyTorch site:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I'm getting great results with this model so far.  However, many annotation programs now support marking bounding boxes as "occluded".  I'm not sure if I can make this model take advantage of or incorporate this into it's training/calculations, can anybody provide a code example or link to an example that demonstrates this?  Can this model be used to take advantage of the "occlusion" tag in most annotation programs today?  Or do I need to choose a different model more specific to this purpose?


texts: Do RL agents learn the optimal "degree" of an action to take?

texts:
I have a game environment I want to train an RL model on. This environment has 2 fundamental actions that the agent can take; "Left" or "Right" (say, 0 or 1).

However, the actions "Left" or "Right" can be taken in a discrete number of "degrees". For example, I can take action "Left" with degree 70% , or take action "Right" with degree 16%.

Assuming a discrete action space between 0-100% for each "Left" or "Right", making the total action space a discrete size of 201 (0-200 in increments of 1), does an agent learn the optimal degree to take either "Left" or "Right" in any given state?


texts:
I read that in a hidden layer of a convolutional neural network, all neurons share the same weights and bias. As a result, all the neurons detect the same feature and hence ConvNets become invariant to translation.

And then I read that in 2003 Simard, Steinkraus and Platt achieved better classification in MNIST classification by artificially expanding the data using translation among other operations.

So, my question is are ConvNets really invariant to translation?


texts: Which one first: algorithm benchmarking, feature selection, parameter tuning?

texts:
When trying to do e.g. a classification, my approach currently is to

try out various algorithm first and benchmark them
perform feature selection on the best algorithm from 1 above
tune the parameters using the selected features and algorithm

However, I often cannot convince myself that there may be a better algorithm then the selected one, if the other algorithms have been optimized with the best parameter / most suitable features. At the same time, doing a search across all algorithms * parameters * features are just too time-consuming.

Any suggestion on the right approach / sequence?


texts: What should we do with the construction of a classifier (e.g., NN) if we have more number of input features?

texts:
For example, I consider an NN with n number of input units. What should the construction of NN be changed if the number of input units increases to 10*n?
I think the number of hidden units and hidden layer should be increased. And, we should have a higher number of samples in the dataset (e.g., using data augmentation).
I also think dimensionality reduction could be used (like word embedding).
What do you think about this problem?


texts: Can't load custom Keras metrics using mlflow.pyfunc

texts:
I have a DNN in Keras, which includes a custom metric function and which I want to pipeline with some SKlearn preprocessing. I further want to persist the model using MLFlow for easy deployment. The requirement to pipeline with sklearn means that I can't use the mlflow.keras versions of .log_model() and .load_model(), and have to instead use the mlflow.pyfunc versions, which is fine.

Saving the model seems to work fine, but when I try to use mlflow.pyfunc.load_model() to reimport the saved model I get this error message (full stack trace at link):

  ValueError: Unknown metric function:custom_mse

To try and make sure that the custom function makes its way through to MLFlow I'm persisting it in a helper_functions.py file and passing that file to the code_path parameter of .log_model(), and then attempting to import that function in .load_context() before using keras.models.load_model() to reimport the saved keras model.

helper_functions.py:

import keras.backend as K

def custom_mse(y_true, y_pred):
    return K.mean((y_pred - y_true) ** 2)

The PythonModel I'm trying to persist is this:

class ProductRecommender(PythonModel):

    def __init__(self, pipeline):
        self.pipeline = pipeline

    def load_context(self, context):

        from helper_functions import custom_mse

        self.keras_model = keras.models.load_model(context.artifacts["keras_model"], custom_objects={'custom_mse', custom_mse})
        self.sklearn_preprocessor = joblib.load(context.artifacts["sklearn_preprocessor"])

        self.sklearn_model = KerasModelRegressor(self.keras_model, epochs=5, validation_split=0.2)

        self.pipeline = Pipeline(steps=[
            ('preprocessor', self.sklearn_preprocessor),
            ('estimator', self.sklearn_model)
        ])

    def fit(self, X, y):
        self.pipeline.fit(X, y)
        self.pipeline.named_steps.estimator.model.save('artifacts/keras_model.h5')
        joblib.dump(pr.pipeline.named_steps.preprocessor, 'artifacts/sklearn_preprocessor.joblib')

    def predict(self, context, X):
        return self.pipeline.predict(X)

Note I import custom_mse function from the helper_functions module and pass it as part of custom_objects to keras.models.load_model()

Here's the mlflow.pyfunc.log_model() call:

with mlflow.start_run() as run:

    run_id = run.info.run_id

    conda_env = {
        'name': 'mlflow-env',
        'channels': [
            'defaults',
            'anaconda',
            'conda-forge'
        ],
        'dependencies': [
            'python=3.7.0',
            'cloudpickle',
            'keras==2.2.5',
            'joblib==0.13.2',
            'scikit-learn==0.20.3'
        ]
    }

    artifacts = {
        'keras_model':'artifacts/keras_model.h5',
        'sklearn_preprocessor':'artifacts/sklearn_preprocessor.joblib'
    }

    mlflow.pyfunc.log_model(
        artifact_path='Model',
        code_path=['artifacts/example_sklearn_wrapper.py', 'artifacts/helper_functions.py'],
        python_model=pr,
        conda_env=conda_env,
        artifacts=artifacts
    )

What's happening here? Why isn't keras seeing my custom_mse function?


texts: Fitting as much points as possible on a line

texts: I want to fit a set of points on a 2D plane that looks like the blue points in the following picture. Instead of having a least square fit (the yellow, dotted line), I want a line that looks like the red line that either crosses the blue points as much as possible or is perpendicular to them. How can I achieve this?


texts: What are the key differences between FPGA and GPUs for Deep Learning?

texts:
I'm trying to investigate the ways in which FPGAs differ to GPUs for the purpose of deep learning. I understand this is a complex question and not necessarily easy to answer in one go, however what I'm looking for are the key differences between the two technologies for this domain.

Ideally, it would be nice to see the benefits and cons of both technologies with regards to Deep Learning.

In addition, from my understanding, and please correct me if I'm wrong, is that with FPGAs, those undertaking a DL project would need someone who is able to configure the FPGA according to the type of project they want through the use of languages such as Verilog or VDHL. Furthermore, if they want to change the type of DL project they want to do, they have to reconfigure the FPGA to follow suit. Have I got this correct, or is something like Intel's Deep Learning Inference Accelerator (https://www-ssl.intel.com/content/www/us/en/servers/accelerators/deep-learning-inference-accelerator-product-detail.html) a lot more similar to GPUs than I thought in terms of configuration?


texts: what is the representation/meaning/implication in real life of $P(\text{+})$ in the wiki Drug testing Example about Bayes' theorem

texts:
In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule) describes the probability of an event, based on prior knowledge of conditions that might be related to the event.

wiki gives this example to illustrate Bayes' theorem

  Suppose that a test for using a particular drug is 99% sensitive and
  99% specific. That is, the test will produce 99% true positive results
  for drug users and 99% true negative results for non-drug users.
  Suppose that 0.5% of people are users of the drug. What is the
  probability that a randomly selected individual with a positive test
  is a drug user?

${\displaystyle {\begin{aligned}P({\text{User}}\mid {\text{+}})&={\frac {P({\text{+}}\mid {\text{User}})P({\text{User}})}{P(+)}}\\&={\frac {P({\text{+}}\mid {\text{User}})P({\text{User}})}{P({\text{+}}\mid {\text{User}})P({\text{User}})+P({\text{+}}\mid {\text{Non-user}})P({\text{Non-user}})}}\\[8pt]&={\frac {0.99\times 0.005}{0.99\times 0.005+0.01\times 0.995}}\\[8pt]&\approx 33.2\%\end{aligned}}}$

to make this discussion more concrete, assume there are 10,000,000 people in the computation.

0.5% of people are users of the drug means there are 50,000 people are drug users.

$P({\text{+}}\mid {\text{User}})$ represents that the test will produce 99% true positive results for the 50,000 drug users, that is, 4,950 people

in this context, what does "$P(\text{+})$" mean?


texts: Various algorithms performance in a problem and what can be deduced about data and problem?

texts:
HI I am currently trying to apply various algorithms to a classification problem to assess which could be better and then try to fine tune the bests of the first approach. I am a beginner so I use Weka for now. I have basic ML concept understanding but am not in the details of algorithms yet.

I observed that on my problem, RBF networks performed vastly worse than IBK and other K methods.

From what I read about RBF networks, "it implements a normalized Gaussian radial basisbasis function network. It uses the k-means clustering algorithm to provide the basis functions and learns either a logistic regression (discrete class problems) or linear regression (numeric class problems) on top of that. Symmetric multivariate Gaussians are fit to the data from each cluster. If the class is nominal it uses the given number of clusters per class.It standardizes all numeric attributes to zero mean and unit variance."

So basically, it also use k means to classify at first. But for some reason, I get the worst results with it using my metrics (ROC), while K methods are among the bests. Can I deduce from that fact something important about my data, like the fact that it has not a gaussian distribution, or is not fitted for logistic regression, or whatever I can't figure out?

I also observed that random forests get similar results to K methods, and that adding a filter to reduce dimensionality improved these RF, random projection being better than PCA?

Can this last point means that there is much randomness in my data so random dimension reduction is better than "ruled" dimension reduction like PCA? What can I deduce from the fact that RF perform equally to K methods?

I feel there is some signification here, but I am not skilled enough to understand what, and I would be very glad for any insights. Thanks by advance.


texts:
Let's consider that I give an agent a reward of -1 (minimum reward) every time it performs an action which leads to the premature end of the episode (i.e., the agent dies). Besides, I also give a negative reward in the [-1, 0) interval when the agent performs an action that I want to avoid to repeat too much. However, these actions do not terminate an episode.

Is it possible that the agent will learn to take the "-1" action given that it is possible to collect many negative rewards which in the end will sum less than -1? This is, the agent will prefer to commit suicide rather than experience many negative rewards.

PD: the agent also receives positive rewards in the (0, 1] interval, which tend to be of higher magnitude than the negative rewards but the agent receives them less often when it is exploring.


texts: What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data

texts:
I am trying to find a formula, method, or model to use to analyze the likelihood that a specific event influenced some longitudinal data. I am having difficultly figuring out what to search for on Google.

Here is an example scenario:

Image you own a business that has an average of 100 walk-in customers every day. One day, you decide you want to increase the number of walk-in customers arriving at your store each day, so you pull a crazy stunt outside your store to get attention. Over the next week, you see on average 125 customers a day.

Over the next few months, you again decide that you want to get some more business, and perhaps sustain it a bit longer, so you try some other random things to get more customers in your store. Unfortunately, you are not the best marketer, and some of your tactics have little or no effect, and others even have a negative impact.

What methodology could I use to determine the probability that any one individual event positively or negatively impacted the number of walk-in customers? I am fully aware that correlation does not necessarily equal causation, but what methods could I use to determine the likely increase or decrease in your business's daily walk in client's following a specific event?

I am not interested in analyzing whether or not there is a correlation between your attempts to increase the number of walk-in customers, but rather whether or not any one single event, independent of all others, was impactful.

I realize that this example is rather contrived and simplistic, so I will also give you a brief description of the actual data that I am using:

I am attempting to determine the impact that a particular marketing agency has on their client's website when they publish new content, perform social media campaigns, etc. For any one specific agency, they may have anywhere from 1 to 500 clients. Each client has websites ranging in size from 5 pages to well over 1 million. Over the course of the past 5 year, each agency has annotated all of their work for each client, including the type of work that was done, the number of webpages on a website that were influenced, the number of hours spent, etc.

Using the above data, which I have assembled into a data warehouse (placed into a bunch of star/snowflake schemas), I need to determine how likely it was that any one piece of work (any one event in time) had an impact on the traffic hitting any/all pages influenced by a specific piece of work. I have created models for 40 different types of content that are found on a website that describes the typical traffic pattern a page with said content type might experience from launch date until present. Normalized relative to the appropriate model, I need to determine the highest and lowest number of increased or decreased visitors a specific page received as the result of a specific piece of work.

While I have experience with basic data analysis (linear and multiple regression, correlation, etc), I am at a loss for how to approach solving this problem. Whereas in the past I have typically analyzed data with multiple measurements for a given axis (for example temperature vs thirst vs animal and determined the impact on thirst that increased temperate has across animals), I feel that above, I am attempting to analyze the impact of a single event at some point in time for a non-linear, but predictable (or at least model-able), longitudinal dataset. I am stumped :(

Any help, tips, pointers, recommendations, or directions would be extremely helpful and I would be eternally grateful!


texts: How to frame this data science problem to have ML algorithm optimize input function?

texts:
I have data generated using production machine. Different components of these machine have certain initial vague functions assigned to them through which we maintain response of this machine.

For example

component 1 = function f1 (could be temperature)
component 2 = function f2 (could be pressure)
.
.
.
.
and it continues till f20.

All these functions have priorities which are maintained in the graph, which is followed for production machine's actions.

For example, f1 (temperature should not exceed 100 degrees) has more priority than f2 (pressure should not exceed certain threshold). If f1 is achieved, machine should stop working to prevent damage but if f2 is reached, machine does not need to be stopped but speed of production should be reduced.

Is there any good way to frame this approach where I can update and optimize these functions and their priorities, assigned to these components using data generated through these components? Can you let me know if there could be something missing here which can not achieve this goal or may be better way to frame this problem?


texts: Which learning model should I use to find out the most relevant applicants?

texts: Let say for one job application we received 10,000 applications. But after a long exercise we found only 10 resumes are relevant fit. Which ML model would be the right choice to determine the most relevant candidates?


texts: Hello, when i'm training my model with 80% data and testing with 20% data the accuracy is 49% and without split it's 99%

texts:
Hello, when i'm training my model with 80% data and testing with 20% data the accuracy is 49%. And when i'm training my data without splitting it's giving around 99%. I'm confused. Please help me with this
The below code is with split which got 49% accuracy
data = pd.read_csv(r"dataset.csv")

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = data.select_dtypes(include = "object").columns

for feat in objList:
    data[feat] = le.fit_transform(data[feat].astype(str))

X = data.iloc[:, data.columns != 'Outcome'].values
y = data.iloc[:, data.columns == 'Outcome'].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

from sklearn.ensemble import RandomForestClassifier
randomforest = RandomForestClassifier(n_estimators = 10, criterion = 'entropy', random_state = 0)
randomforest.fit(X_train, y_train)

y_pred = randomforest.predict(X_test)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred))
cm = confusion_matrix(y_test, y_pred)
print(cm)

The below code is without split which got 99% accuracy
data = pd.read_csv(r'dataset.csv')

le = LabelEncoder()

objList = data.select_dtypes(include = "object").columns

for feat in objList:
    data[feat] = le.fit_transform(data[feat].astype(str))

X = data.iloc[:, data.columns != 'Outcome'].values
y = data.iloc[:, data.columns == 'Outcome'].values

file = open(r'finalized_model.pkl', 'rb')

data = pickle.load(file)

y_pred = data.predict(X)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y, y_pred))
cm = confusion_matrix(y, y_pred)
print(cm)

Total data is the same used for 2 codes.


texts: Same TF-IDF Vectorizer for 2 data inputs

texts:
I am trying to work on Dataset released by quora, to identify if Question1 has similar intent as of Question2
The dataset looks like:

id|question1|question2|is_duplicate
0|What is the step by step guide to invest in share market in india|What is the step by step guide to invest in share market?|0

I am trying to refer to Abhishek Thakur's feature to get started.
It says:

As per my understanding the python code for sklearn would be:
    from sklearn.feature_extraction.text import TfidfVectorizer
    import pandas as pd
    tfidf_vectorizer = TfidfVectorizer()
    data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data.question1)
    data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data.question2)

data['tf_idf_q1] and data['tf_idf_q2] will refer to 2 models for each question as in 1st part of the image.
I am not sure how would i achieve second part? Do I fit_transform the vectorizer with first question and then transform the second question?
Or do I merge 2 questions and then get a vectorizer?
Something like below:
merged_questions = pd.DataFrame(data['question1'].map(str) + data['question2'].map(str))
data['tf_idf_q1_q2'] = tfidf_vectorizer.fit_transform(merged_questions)

Any inputs are greatly appreciated.