texts: Why does Caret featurePlot returns returns NULL?

texts:
I am trying to use featurePlot of caret in R.

This is the code I am using:

trellis.par.set(theme = col.whitebg(), warn = FALSE)
caret::featurePlot(x = df[, 3:6],
            y = df$ud,
            plot = "scatter")

where all the columns of df[, 3:6]are of numeric type, while df$ud is a binary factor.

I also have installed the library ellipse as suggested elsewhere.

However, I keep getting NULL.

Any idea why?


texts: I keep getting an error message in R while using twitteR

texts:
here is what I am doing:
tweets=searchTwitter("walmart",n=3000, lang="en",since="2021-01-08",until="2021-01-10")

And this is my error message:
Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit,  :
  3000 tweets were requested but the API can only return 0

Any help would be appreciated.


texts: When classifying documents with naive Bayes, which probabilities do you multiply in case of repeated terms?

texts:
In short: For naive Bayes and text classification, do you multiply a probability for each instance of a word in a document or once if the word occurs?

In more detail: The question is how to calculate naive Bayes for a document. We have a corpus of text from which we can calculate the frequency of documents with class $C$ having a word $w$. We then calculate the probability $P(w_1|C)...P(w_n|C)$ and choose the $C$ that maximizes this number. But I'm not sure whether we multiply a probability for each word whenever it occurs or just once if it occurs in the document? Like, if the text is "$w_1 \ w_2 \ w_1$", is the estimated probability $P(w_1|C)^2 P(w_2|C)$ or is it $P(w_1|C)P(w_2|C)$?


texts: Using GridSearchCV and a Random Forest Regressor with the same parameters gives different results

texts:
As the huge title says I'm trying to use GridSearchCV to find the best parameters for a Random Forest Regressor and I'm measuring my results with mse.

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

estimator = RandomForestRegressor()
para_grids = {
            "n_estimators" : [10,50,100],
            "max_features" : ["auto", "log2", "sqrt"],
            "bootstrap"    : [True, False]
        }

grid = GridSearchCV(estimator, para_grids, scoring = 'mean_squared_error')
grid.fit(Inputs_Treino, Outputs_Treino)
forest = grid.best_estimator_

reg_prediction=forest.predict(Inputs_Teste)

print (grid.best_score_, grid.best_params_)

mse = mean_absolute_error(Outputs_Teste, reg_prediction)

This is the gist of the code (nothing too complex I know, just getting started with it all)

When I print the result of grid.best_estimator_ I get this

RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

The problem is if I try to create a regressor with these parameters (without using grid search at all) and train it the same way I get a waaaay bigger MSE on the testing set (5.483837301587303 vs 43.801520165079467)

Inputs_Treino = dataset.iloc[:253,1:4].values
Outputs_Treino = dataset.iloc[:253,-1].values
Inputs_Teste = dataset.iloc[254:,1:4].values
Outputs_Teste = dataset.iloc[254:,-1].values

regressor = RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

regressor.fit(Inputs_Treino,Outputs_Treino)

#fazer as predictions
Teste_Prediction = regressor.predict(Inputs_Teste);

mse = mean_squared_error(Outputs_Teste, Teste_Prediction);

Does this have to do with the cross validation GridSearchCV performs ? What am I missing here ?


texts:
My data contains a set of start times and duration for an action. I would like to plot this so that for a given time slice I can see how many actions are active. I'm currently thinking of this as a histogram with time on the x axis and number of active actions on the y axis.

My question is, how should I adjust the data so that this is able to be plotted?

The times for an action can be between 2 seconds and a minute. And, at any given time I would estimate there could be about 100 actions taking place. Ideally a single plot would be able to show hours of data. The accuracy of the data is in milliseconds.

In the past the way that I have done this is to count for each second how many actions started , ended, or were active. This gave me a count of active actions for each second. The issue I found with this technique was that it made it difficult to adjust the time slice that I was looking at. Looking at a time slice of a minute was difficult to compute and looking at time slices of less than a second was impossible.

I'm open to any advice on how to think about this issue.

Thanks in advance!


texts:
There a lot of information on how to handle categorical variables when preprocessing data for ML classification. However, I cannot find any feedback on how to handle categorical variables, where each sample can belong to more than one label.

I'm working on bug detection classifier. I've got many features like who contributed to the source code. There are about 200 unique labels and creating so many dummy variables makes my model overfit.

So are there any alternatives for this method. Something like target-based encoders (ex. CatBoost)


texts:
We are capturing emotions as survey responses. We need to assign values for the responses(emotions) for analysis purposes. Is there an optimum range that can be assigned to achieve this? (like from -100 to 100). An example of a question and a set of answers are as follow.

Question: "How are you feeling today?"
Answers: Terrible, Sad, Ok, Good, Great

A suitable approach we could think of is to assign values from 1 to 100 with equal distance. Is this statistically valid? What are the things that we should consider when achieving this? In this case, only positive integers are assigned because we need to calculate statistics like weighted average. Can't we assign negative numbers as well?


texts: How to get the collective score from different test methods

texts:
im very new to this - I have a dataset of some patients with certain symptoms of sleep apnea. They were assessed for the risk of Sleep Apnea using ESS, BQ, SBQ and SACS (different testing methodologies) .
So we really do not know which test method is better than the other, so was thinking of creating one collective score based on a patient's individual ESS,BQ,SBQ,SACS scores.
Question is how to achieve this single collective score? if there would be some disadvantages to consider?


texts: train half model in one time and other in another time

texts: I am using keras library to build the convolutional neural network model and tensorflow as a backend so what I wanted to do is that to train the model in one point time then save that model and again use that trained model to train itself to improve the accuracy. So is it possible then how should I do?


texts: metric performances varies with each training run of my deep learning model

texts:
My model (a biLSTM model in Keras) gives different results for my test metric (precision) each time I train.
My metric varies between 80% and 92%. That is really bad.
What reason besides a bad kernel initialization could this be?


texts: How would you teach multiplications to a neural network?

texts:
Let's say you train a neural network on Input = 1/Output = 2; Input = 4/Output = 8.

How would you train a machine to recognize that it needs to multiply the input by 2, from scratch?


texts: Why is deep learning used in recommender systems?

texts:
I am currently reading a lot about recommender systems (RS) and came across that many RS are based on deep learning.
However, I never find a good scientific article why deep learning is used in RS and why it is more successful compared to other methods.
Now my question is:

Why is deep learning used in RS?

Why would it be better to use deep learning in RS?

How does deep learning work in RS?

I would be very happy if you could help me to answer my unanswered questions. I would be very happy if you would add a source (that is scientific, no medium etc.) that I can read more about it.


texts: Migrate Dash App to separate Data Studio Reports

texts:
I would like to know how can I do this and if it is makes sense to you.

I have a Dash app written in Python in which:

1) I make the connection to my Databases (Python).
2) Perform several queries in SQL to obtain the data I need (Python).
3) If need, manipulate the query response with Python to obtain the
   desired metric (Python).
4) Build the plots (Python, Plotly, Dash)

This Dash App is on a server and I pay for it.

On the other hand, I am evaluating to migrate this App to something else, replacing the previous items with:

1) Make the connection to the databases with Google BigQuery
2) Perform the queries in SQL to obtain the data I need with Google
   BigQuery.
3) If I need, manipulate the query response with Python to obtain the
   desired metric.
4) Build the plots or dashboard with Google Data Studio.

So, is it possible? I have some question about each point:

1) To make the connection to my databases with Google BigQuery? Do I
   have to migrate my datasets to Google Cloud to use them?
2) I assume that you can perform queries with SQL normally with Google
   BigQuery, I need to do several JOINS,etc.
3) Can I add a Python script to manipulate my data in Google BigQuery?

So Instead of having an app I will have several reports on Google Data Studio.


texts: What would be the best machine learning approach for sets of varying sizes?

texts:
I have the following problem: I have two differents sets of labels (extracted using N.E.R) and given a combination of labels of the first set (a,b,c or d) I have a supervised set of best combination of the second (x,y,z) as an "answer".
The problem is, both can vary in size.
A hypothetical training data would be something like:
{a1,b2,c4,d1} -> {x2,y4,z5}
{a1,b1,c1} -> {x2,y2,z1}
{a4,b2,c4,d1} -> {x1,y3,z5}
...
{a4,b2,c4,d3} -> {x1,y3,z5,w2}

Of course new types of combination of the first set would appear and, using ML, I'd expect to like to give the best prediction.
So, what would be the best machine learning approach for that situation?


texts: Time series analysis model evaluation performance metrics integration in time series application

texts:
After study in time series analysis, I recognized RMSE and MAPE  are the best evaluation metrics for used model in real time series application. But my queries are below as this is my first practice application project in time series analysis:
If I use past 1 year data for training and forecast for next 30 days but we will get actual data after 30days in real time. How could we integrate  these RMSE and MAPE with time-series application?
As I heard from some data scientist if I skip first training cycle and calculate these RMSE and MAPE in next retraining cycle for past training cycle where I have actual data and forecast data, Is this approach useful ?
I concerned about calculated RMSE and MAPE could help to improve model performance in real time.
Could the evaluation metrics calculation every time on retaining useful from production point of view?
My thanks in advance


texts: What machine learning model should be used to predict coincidence factor

texts:
I have coincidence  factor for different sizes of groups and the associated attributes (e.g. building usage type and floor area) for each consumer in the group. I want to predict coincidence  factor for new groups using the attribute for each consumer in the new group and the number of consumers in the group. What machine learning method can solve the problem? :)

Defintions:
Coincidence factor is the peak of a system divided by the sum of peak loads of its individual components


texts: Replacing missing values with mean of feature calculated from previously replaced values

texts:
I am not sure how to ask this, but I will try my best.

I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all missing values were replaced because some categories of the second feature did not have any values in the first feature, so its mean cannot be calculated. Then, I tried again by replacing the leftover missing values with mean conditional on a third feature. The third feature also contains values that were just replaced with the mean of the first feature. So, is it alright to replace missing values with the mean calculated from values that include some that have once replaced other missing values?


texts: Checking linearity for a linear regression model?

texts:
I've read that there are various assumptions associated with a multiple linear regression model which you should check/validate before getting too excited about your model results.
One of these is the assumption of linearity. I get that you would plot the dependent variable against the independent variable and visually check for linearity, but is there a more scientific way to do this?
I have the two plots below. Looking at the first, I can see some linearity by removing the outliers. The second however, is much harder. I can *maybe* see something, but I'm not sure if this is my eyes playing tricks on me.

If I determine the second plot does not satisfy linearity, what do I do? Exclude the feature from the model?


texts: Online service for crowdsource text labbling

texts:
I'm trying to make a text classificator for short texts.
For that task, I need a labeled samples dataset. I already have samples but most of them are not labeled. I'm tired to label them in excel alone =)

I wonder if there are any free/open source services for crowdsource text labeling so I can ask my friends to help me.
Are there any services for that?


texts: Non-brute force approach to finding permissible English word anagrams

texts:
I am interested in proposed base R/Python solutions (i.e. without relying on external packages/libraries) to the following problem:

  You have a large dictionary containing millions of valid English
  words. You are given an input “word”, which may or may not be an
  anagram of one or more words in the dictionary. Find all possible
  valid words in the dictionary of which the input may be an anagram.
  Note, the input “word” may be meaningless. For example,

  input: lloeh output: hello

  input: cksli output: slick, licks

  Rather than use a brute force technique that computes all possible
  character combinations for each input target and compares them to a
  dictionary, you want to use a more efficient technique.

The approach I was considering was to pre-compute impossible string onsets in the target language. For example, in English, the onset sequences ^ck and ^ng are not permissable because they violate the orthographic and phonotactic constraints of the language. Given an off-line list of impossible_sequences at run time you would incrementally build possible "words" from the target. During each iteration the possible word is checked against the impossible_sequences list and if it matches the search
down that search branch is terminated. The following pseudo-code in R uses for loops to illustrate my idea.

target <- "cksli"
impossible_list <- c("ck", "ng", "rt", "zz") # pre-computed not-possible English onsets

for (i in 1:nchar(target)) {
  word <- substr(target, i, i)
  if (word %in% impossible_list) break
  for (j in 2:nchar(target)) {
    word <- paste0(word, substr(target, j, j))
    if (word %in% impossible_list) break
    for (k in 3:nchar(target)) {
      ### same logic
      ### goes deeper and deeper
    }
  }
}

(1) Are there any issues with the underlying logic of my proposal?
(2) Obviously nested for loops are inelegant - Could one do this recursively and if so, how?
(3) What other approaches should be considered?


texts: First layer weights for transfer learning with new input tensor in keras.applications models?

texts:
In the pre-implemented models in keras (VGG16 ect) it is specified that we can change shape of the inputs of the models and still load the pre-trained imagenet weights.

What I am confused about is then what happens to the first layer weights? If the input tensor  has a different shape, then the number of weights will be different than for the pre-trained models isn't it?

Here is the implementation of the Keras VGG16 model for reference.


texts: Validation loss keeps fluctuating about training loss

texts: I am training a Keras model for multi-target regression by using a custom loss function with the goal of getting predictions accurate to below 0.01 with respect to that loss function. As can be seen from the below plot of the loss functions, both the training and validation loss quickly get below the target value and the training loss seems to converge rather quickly while the validation loss keeps fluctuating about the training loss value. Although the loss is below the target threshold I'm wondering whether such fluctuations could reflect some problems with the model fit? Or could it be that the validation set size is too small (compared to the training size); they are |training| = 13500; |validation| = 3400 respectively (so about 80% and 20%). I am using batch learning with a batch size of 16.


texts: How to include class features to linear SVM

texts:
I am planning to do a simple classification with a linear SVM. One feature I have is another classification of some sort done previously.
Can I just use this class feature as a 1-hot encoded array? So, e.g. for 3 different classes, I'd have 3 binary features being 0 or 1?

The problem I see is that this feature is not linear but binary. Will this pose a problem? And if yes, how can I somehow transform a binary feature into a linear one?


texts: How to handle missing data for machine learning

texts:
I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation.

As an example, I would like to get weather data such as rain and temperature for the past 7 days of an event. I got this data from BigQuery which seems to work fine. But sometimes it does not find data for all the past 7 days. How would I handle such a case? What would be a good way to structure my features? I thought something like:

dayofyear,temp,water_temp,temp-1,rain-1,dayofyear-1,temp-2,rain-2,dayofyear-2,....,temp-7,rain-7,dayofyear-7,visibility

While the last one would be the field I like to predict at some point.


texts: How does Decision Tree with Gini Impurity Calculate Root Node?

texts:
I couldn't figure out how it selected the root node with with <=7.5 and it's gini impurity is 0.45 but I tried to manually calculate it but the weighted gini impurity I got for it was 0.27.

Can anyone explain me how the calculation is done here just for the Root Node?

Here's a small dataset that I've generated,

import pandas as pd
import numpy as np

a = [5, 6, 7, 8, 9]
a1 = [1, 0, 1, 0, 0]
df = pd.DataFrame(np.c_[a, a1], columns=['val','target'])

   val  target
0    5       1
1    6       0
2    7       1
3    8       0
4    9       0

Here my code,

from sklearn.tree import DecisionTreeClassifier
from IPython.display import Image
import pydot

dt = DecisionTreeClassifier()
dt.fit(df.val.to_frame(), df.target.to_frame())

data_dot = export_graphviz(dt, out_file=None, class_names=['0', '1'])
graph= pydot.graph_from_dot_data(data_dot)
Image(graph[0].create_png())


texts: Is it possible to share and merge classification models?

texts:
I'm starting my machine learning study and I'm trying to figure out a simple question:

Let's say that I have two models, one that recognizes cats, and another one that recognize dogs.

Now I have a camera and I want to recognize both cats and dogs using my models.

Obviously, I don't want to create a third model to recognize both, having all the work to label each animal, so, is there a method to "merge" both models into one?

I'm asking this question because I want to understand one thing: why ML engineers don't share their models so then we can create aggregated models? For example, if a person A has a model that classifies people and a person B has a model for animals. Why they can't just share their models so then each one will have a more powerful model without needing to re-train everything?

I'm sorry if this is a too basic question, but I didn't see on Google any clear explanation. Thanks.


texts: How to apply an RNN to forecast non-stationary time series?

texts:
Is it possible to predict a time series which is non-stationary, in the sense that, the dependent variable Y have an increasing trend? Therefore, the highest value of $Y$ in the training set may be lower than $Y$'s in the test set. Af far as I know, RNNs output some scaled-down version of the dependent variable. Is it possible to scale it back, when the range of values is also changing?

(One of) the independent variables (after moving avg smoothing):

The dependent variable (after moving avg smoothing):


texts: What does Conv1d do in a sentiment analysis?


texts: best NN architecture for point prediction

texts:
I'm training to predict a single value y (continuos in [0,1]) based on a number of variables x1, ..., x45. These are variables related to various measurements of the same value using different techniques -  some of them may be irrelevant.

1) Would it make sense to use a CNN? I read that it works best for things like image recognition/classification, and this is clearly not the case

2) Which NN would work best? The benchmarks would be the naive linear regression and the average x


texts: How can precision be less than one in Leave-One-Subject-Out binary classification if each subject contains only one class

texts:
Say I'm trying to classify a medical condition.
Theres only two classes: Sick and Healthy.
I build a model and I can't split the data because I don't want data from the same patient being in training and test set. So I elect to use Leave-One-Subject-Out, training the model on all subject except one and testing on the left out subject.
So for each test set I have one subject and they are either healthy or sick. So the confusion matrix only contains one class where precision is technically one every time and recall equals accuracy.
I've been reading some papers that claim to use leave-subject-out training and test splits for tasks where patients either have a medical condition or do not. I've seen papers that report accuracy, recall, and precision but I don't understand how you could have precision be less than one if each subject only contains one class. I doubt these papers are lying because I've seen this more than once.
I just want to know whats going on here for them to get precision values that are less than one. Are they doing some kind of averaging or am I missing something and thinking about this in the wrong way? None of the papers explain this either.


texts: Prediction count seems to be way in logistic regression despite good accuracy, precision and recall score

texts:
I built a Logistic Regression Model to predict Loan Acceptors. The dataset is 94% non acceptors and 6% acceptors. I've run several logistic regression models one with the original dataset, one after upsampling to 50/50 and removing some predictor variables, and one without the upsampling, but after removing some predictor variables.
Model 1: Better than 90% accuracy, precision and recall on 25 feature columns. After running the model, I output the predicting to a different CSV (same people as original csv though) and it's returning 10,000 acceptors. My guess was this could be caused by overfitting? Wasn't sure, but then tried it on the same 94% non-acceptors and 6% acceptors, but with fewer variables (19 feature columns). This time the accuracy is 81%, but the precision is only 21%, while recall is 765 (for training and test). This time it only returns 8 total acceptors ( out of 18,000)
Finally, I tried upsampling and upsampled to a balanced set. The accuracy is only 68% (which I can work with) and the precision and recall is 66% for both training and test. Ran the model then outputted the prediction to the csv file (again same people, different CSV file, not sure if that's messing it up) and this time it returned 3 acceptors.
Additionally, it's returning either 0 or 1's, which I thought logistic was supposed to return probability, so .78 or .55, etc.THat could just be me misunderstanding it though.
Does anyone have any advice on what is causing this and how I can fix this?
I'm not sure which regression code would be most beneficial, so I'm posting the upsampled code because I feel ultimately, that's what I need to use since the data is so far off from being balanced.
y = upsampled.OpenedLCInd.values
# Drop our target feature from our features dataframe
X =upsampled.drop(['OpenedLCInd', 'LCSent', 'Age','DisposableIncome', 'AvgDDSum', 'AvgDDCount', 'AvgTransactionCount', 'LetterCheckCount','sharecount','CheckingCount','loancount','IndirectMemberInd'], axis = 1)
cols = X.columns

mm = MinMaxScaler()
# Fit and transform our feature dataframe
X = pd.DataFrame(mm.fit_transform(X))
# Reassign column names so new dataframe has corresponding names
X.columns = cols

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .30, random_state = 33)

logreg = LogisticRegression(fit_intercept = False, C = 1e12, solver = 'liblinear')
# Fit the model to our X and y training sets
logreg.fit(X_train, y_train)

# Generate model prediction data for train and test sets
y_hat_train = logreg.predict(X_train)
y_hat_test = logreg.predict(X_test)

residuals = np.abs(y_train - y_hat_train)
# Print value counts of our predicted values
print(pd.Series(residuals).value_counts())
print('----------------------------------')
# Print normalized value counts of our predicted values
print(pd.Series(residuals).value_counts(normalize = True))

residuals = np.abs(y_test - y_hat_test)
print(pd.Series(residuals).value_counts())
print('---------------------------------')
print(pd.Series(residuals).value_counts(normalize = True))

cnf_matrix = confusion_matrix(y_test, y_hat_test)
print('Confusion Matrix: \n', cnf_matrix)

### Beginning of Output to Test Dataset

test=pd.read_csv(r'/testdata from sql.csv')
test.drop(['OpenedLCInd', 'LCSent', 'Age','DisposableIncome', 'AvgDDSum', 'AvgDDCount', 'AvgTransactionCount', 'LetterCheckCount','sharecount','CheckingCount','loancount','IndirectMemberInd'], axis = 1)

##TEst Initial Regression Model
predictions = logreg.predict(X_test)

test_predictions = logreg.predict(test.drop('OpenedLCInd', axis = 1))

test["predictions"] = test_predictions

test.to_csv(r'/finaltest.csv')

````


texts: what's the best way to plot a confusion matrix in a multilabel setting?

texts:
In a multilabel setting a training example could be a, b, (a, b), d, c, (d, c), etc. This makes it a bit hard to come up with a helpful confusion matrix because the number of columns or rows could be very large - as I understand it, you wouldn't have a column or row for just a or b, but also (a, b).

What do people usually do in these cases? Do they usually create a column or row for every possible combination, or do they simplify somehow?

In case you'd like to provide code I am using Weka, but my question is primarily about the best practice.


texts: I am working with CoreML and ARKit for Face recognition. But i don't want to build CoreML model with app. I have make a coreML model with python Turicate. I want this model to be put on server and it give us a API then i will sent face pictures to this model through this API and then server get back a response in form of JSON.I want this because CoreML size is very high and if we want to more train this model then this user has to download this high size model and compile on device.I have research and found that apple adds a feature to compile CoreML model on device but every time for small training upgradation in model user has to download this high size model again.


texts: What is "GOAL" in terms of Reinforcement Learning specified in these papers?

texts:
I have a question regarding Reinforcement Learning. I've been reading the Horde and the UVFA paper extensively.

Take the Horde paper, there is this GVF, General Value Function Approximators which have a pseudo termination function and a pseudo reward. If I understand correctly, multiple demons with different behavioral policies, each of them having a GVF, spawned and off-policy learning takes place with GQ probability distribution. What is the pseudo termination function here? Is it a state?

Similar to this, there is this "goal" in UVFA paper. Take this paragraph for example, at the end of the second page:

  A two-stream architecture, on the other hand, assumes that
  the problem has a factorized structure and computes its output
  from two components φ : S → R and ψ : G → R, that both map into an n-dimensional vector space of embeddings and output function h : R × R → R that maps two such embedding vectors to a single scalar regression output.

The novelty of this paper is that the value function generalizes not just over states but also goals, as introduced in the abstract. But I'm having hard time understanding the goal definition here. Again, is it a state that an agent wants to achieve?

So, can anyone give me a clue about the meaning behind these two concepts?

Thanks.


texts: Interpreting fraction of zero weights in TensorFlow

texts:
I am using the TensorFlow to do a simple linear classification using logistic regression. The graph included from the TensorBoard displays what they call the fraction of zero weights. How do I interpret this in terms of model evaluation? I am assuming this is good since I got the good results in terms of loss, precision, recall, etc but not sure.

Thank you.


texts: Where in the workflow should we deal with missing data?

texts:
I'm building a workflow for creating machine learning models (in my case, using Python's pandas and sklearn packages) from data pulled from a very large database (here, Vertica by way of SQL and pyodbc), and a critical step in that process involves imputing missing values of the predictors. This is straightforward within a single analytics or stats platform---be it Python, R, Stata, etc.---but I'm curious where best to locate this step in a multi-platform workflow.

It's simple enough to do this in Python, either with the sklearn.preprocessing.Imputer class, using the pandas.DataFrame.fillna method, or by hand (depending upon the complexity of the imputation method used). But since I'm going to be using this for dozens or hundreds of columns across hundreds of millions of records, I wonder if there's a more efficient way to do this directly through SQL ahead of time. Aside from the potential efficiencies of doing this in a distributed platform like Vertica, this would have the added benefit of allowing us to create an automated pipeline for building "complete" versions of tables, so we don't need to fill in a new set of missing values from scratch every time we want to run a model.

I haven't been able to find much guidance about this, but I imagine that we could:

create a table of substitute values (e.g., mean/median/mode, either overall or by group) for each incomplete column
join the substitute value table with the original table to assign a substitute value for each row and incomplete column
use a series of case statements to take the original value if available and the substitute value otherwise

Is this a reasonable thing to do in Vertica/SQL, or is there a good reason not to bother and just handle it in Python instead? And if the latter, is there a strong case for doing this in pandas rather than sklearn or vice-versa? Thanks!


texts: PyTorch: How to use pytorch pretrained for single channel image

texts: If I have to create a model in pytorch for images having only single channel. How can I transform my model to adopt to this new architecture without having the need to compromise the pre-trained weights on which it has been trained upon.


texts: How to arrange the image dataset in CNN?

texts:
How do I arrange the image dataset in CNN? Should I put each image category in a separate folder? Or all of them in the same folder? Should the image name be the category name?

I would like to see an example for an image dataset (other than MNIST).

Thank you.


texts:
Please provide me with excellent resources to learn deep learning with Keras, imbalance images classes and using imageGenerator, Transfer learning with Keras.

Thank you


texts: How is "relevance" defined in information retrieval outside the context of systems with user feedback?

texts:
I've seen information retrieval systems that return some results from a query, and then the user rates these results as either "relevant" or "not relevant".

What can you do if you do not have user feedback?

E.g. suppose your system returns some ranked results from a query. Suppose you have no pre-defined notion of what is relevant, and suppose you cannot receive any kind of user feedback. What can you do?

This is important, because information retrieval evaluation metrics are based on relevance. Maybe it isn't possible to define relevance without user feedback, if so, maybe some information retrieval evaluation metrics without dependency on user feedback can be suggested?


texts: predicted labels or predicted probabilities in a ROC plot?

texts:
What should be taken as a parameter to plot ROC curves , for example in a classification model, I can get predicted labels and predicted probabilities.
I have softmax layer in the output layer.

Should I use predicted labels or predicted probabilities to plot ROC curve in a classification problem.


texts:
I have a question about memory usage.

I want to do 4 things:

1) make a dataframe from one of several columns from a datasource, say a json string
2) make the third column of the original dataset the index to the dataframe
3) change the name of another column
4) change the series i've created to a dataframe

My question is about memory efficiency. It seems that for step 1), I am first loading a whole dataframe, then run a concat command to concatenate the columns I want.

For step 2, I again need to resave the new dataframe as another object.

For step 3, it seems to stick so nothing there.

Please advise on a more efficient way to go about this, if that exists.

Command:

   df = pd.DataFrame(jsonobject)
   df = df.set_index("columnC")
   df.index.names= ["foo"]
   df1 = df["foo"].map(lambda x:x["id"])
   df2 = pd.DataFrame(df1)


texts: How Do Machine Learning Models Work and Remember?

texts:
I am new to machine learning. I am confused about how a machine learning model remembers what it learns. And how it learns.

I know the basic workflow of machine learning: first is data gathering, then data preprocessing, then creating training and test data sets, then training the ML learning algorithm. After training, we test the accuracy of our model with test data. But I am confused about how the algorithm is trained and how it remembers what it learns.


texts:
I created a GridSearchCV for a Random Forest Regressor. Now I want to check the feature importance. I searched around and I found this:
rf_gridsearch.best_estimator_.named_steps.feature_importances_

This already works, but my training data is huge, 669 attributes. Therefore, I need the attribute names. So I found this code:
rf_gridsearch.best_estimator_.named_steps["step_name"].feature_importances_

But I don't know what the "named_steps["step_name"]" are.
I tried something like this:
named_steps = X_train.columns

But this doesn't work. Could somebody explain me what named_steps["step_name"] is?


texts: Initial embeddings for unknown, padding?

texts:
Last time I've been passing pretrained word embeddings into LSTM to solve text classification problems. Usually, there are additional <pad>, <unk> replacements for padding and unknown types. Of course, there are no pretrained vectors for them.

Solutions I've come up with are

Fill them with random values
Fill them with zeros

Which approach is better/wrong? What is the common practice?

Note: I use pytorch+torchtext if it matters.


texts: Where to find statistically relevant documentation of common Python packages?

texts:
I am trying to entice my lab to transition from Matlab and R to Python. The main objection at this point seems to be that Python's analytical libraries are not sufficiently well documented. Given how prolific Python is, I suspect that sufficiently detailed documentation exists and we simply cannot find it.

Example

Recently, I needed to upsample a signal (1D vector) and found a Python function called resample. In contrast to Matlab and R, pandas documentation doesn't tell me what kind of interpolation resample uses (linear, cubic splines, pchip?).

Is there any place where I could find such information about this and other libraries, preferably with equations or references to papers? I understand that I could analyze source code but this isn't most time efficient. If you know R, I am basically looking for a Python equivalent of CRAN (PDF warning).

Thanks!


texts:
Can anyone mathematically prove this equation given the values of $dz^{[2]}$, $W^{[2]}$, $z^{[1]}$ and the activation function $g^{[1]}$

$dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]})$


texts: 80-20 better than full dataset for LightGBM

texts:
Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables.

I have two modes, 1) Production and 2) Testing. The first one just trains a model with the whole dataset. The second does the same with an 80% of the dataset and tests over the remaining 20% (80-20 done with train_test_split, from sklearn.model_selection, no seed used).

In both cases I show the neg_mean_squared_error at the end of the execution. In the first case for the whole dataset, in the second I get two values (training and testing)

I am shocked to see that while in the first case my error is around -10, in the second the values are -5 (training) and -5.3 (testing). An average value of my objective variable can be 80. I would expect to lose accuracy since I train with less data and then I test over a different sample.

Question: There is any theoretical reason that explains that a 80-20 test gets a "better" neg_mean_squared_error than in the full data case? Or it has to be a (sneaky) bug in my code?


texts: Learning to Rank vs Reinforcement Learning in Information Retrieval - which one is preferable and why?

texts:
I am trying to create an information retrieval system which can benefit from user feedback (either implicit, through e.g., click-through data) or explicit (e.g., binary feedback on irrelevant suggested results).
More specifically, my plan is to use an ElasticSearch index to reduce the space by providing a 'bucket' of candidate relevant results and then re-rank the results in the bucket by a more sophisticated ML model that can benefit from feedback (both implicit and explicit, as described above).
My sense is that for the ML component I should use a Learning to Rank method, but other colleagues have suggested that I should take a look in Reinforcement Learning instead.
I want to know what is your viewpoint on the matter, i.e., which of the above two methods (Learning to Rank vs Reinforcement Learning) seems more relevant for the task?


texts: How regroup modality in R and improve inertia on MFA

texts:
I collected some data on wine like this:

There are a lot of regions, there are 100 rows. I tried to do an MFA on my data, using the following:
res.mfa <- MFA(data2,
group = c(4, 2),
type = c("n", "s"),
ncp=2,
           name.group = c("pays","prix"),

           graph = TRUE)

But I don't know why I got only 7% of inertia on my 2 axes. What did I do wrong? As I expected to get 70-80% on 2 first axes but me I got 7%.
One idea to improve results would be to regroup province, variety or wineries which are similar.


texts: Feature scaling data with missing values

texts: am interested in doing some feature scaling to try and tease out something from my data (box plots by outcome show that the 25/50/75 quantiles are very similar; certain variables have more "outliers" than other by class. The issue I have however is that there is a lot of missing data. I would like to replace these NAs (they are numerical) with something like -9999. Should I first scale my data and then assign -9999 to NAs or first assign then scale?


texts:
I'm trying to train a classifier to recognize my own signature. This is how I built my classifier

How did I collect data?

Signed on a piece of paper for 50 times and created 50 images out of it.
for negative test cases, I downloaded IAM Handwriting database. which contains around 600MB of handwriting data. this is to negate other possible matches.

How did I do Feature Engineering?

Step 1 : Read and convert image in gray scale. Perform median blur.

img = cv2.imread(image,0)
img = cv2.medianBlur(img,5)

Step 2 : Perform adaptive threshold followed by morphological opening to reduce noise in the image.

edges = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY_INV,11,2)
kernel = np.ones((2,2),np.uint8)
dilation = cv2.morphologyEx(edges, cv2.MORPH_OPEN, kernel)

How did I do model training?

Extracted ORB features from all the sample images for training (matrix size
RX32) and used a RandomForestClassifier for training.

The problem

The accuracy of my classifier is a whopping 0.9874066374996978, but it fails to recognize almost all of my new signature samples, signed on same paper under same lighting conditions. I'm new to applied ML. What do you experts think I should check??


texts: LSTM prediction for many multivariate time series

texts: Let's say we have 8,000 different time series where each of them has 10,000 samples and 25 features. The goal is to have an LSTM sequence to sequence model (using Keras) where one can use a sequence of 7 elements for each of these time series in order to predict the following 4 elements of that series. I believe to do this for one of the time series: one has to create the 7 lags of the series and the 7 lags of each of the features and the input data would have the shape (10000, 7, 25x7). How would the input and output shapes look like if one wanted to create a single model for all the series at the same time?


texts: Retraining a text classifier with new data that has new labels

texts:
I have a text classifier model built on AutoML Natural Language. It currently does a great job classifying text into the set of labels it was trained for. (One of the labels it is trained for is "Uncategorized")

Now, I'd like to make the model to start classfying some of the "Uncategorized" text into additional new labels. I have new data to train the model on the new labels.

How do I go about this given that i don't want to necesarrily re-train the model for the labels it is already doing a good job, but just want it to start classifying the new labels, and reduce the occurence of the "Uncategorized" label.

Any insight on how to do it on AutoML? I can train a new model with AGGREGATED data but don't want to do that unless it is necessary.

(Note: The new labels are independent of the old ones)


texts:
I have data, which looks like this:

These data are only for one subject. I will have a lot more.
These data will be analyzed in R.

Now I'm storing them like this:

subject <- rep(1, times = 24)
measurement <- factor(x = rep(x = 1:3, each = 8),
                      labels = c("Distance", "Frequency", "Energy"))
speed <- factor(x = rep(x = 1:2, each = 4, times = 3),
                labels = c("speed1", "speed2"))
condition <- factor(x = rep(x = 1:2, each = 2, times = 6),
                    labels = c("Control", "Experm"))
Try <- factor(x = rep(x = 1:2, times = 12),
              labels = c("Try1", "Try2"))
result <- c(1:8,
            11:18,
            21:28)

dt <- data.frame(subject, measurement, speed, condition, Try, result)

What is the appropriate way to store these data in R (in a data frame)?


texts: Find the most relevant columns for each single class in pandas

texts:
The following question (this one) did not help me.

I have a big dataset, and I want to know which Columns are the most relevant for the Target Variable. I know that, in my case, for each class in the Target Variable, different Columns have a different impact.

In that question, the suggested answer recomend using LDA. From what I understood, it looks like a normal classification algorithm, so it's not what I need

What I what is something like

In :
    magic_function("name_of_target_variable_1")
Out :
    ["really_important_column_a", "really_important_column_b" ...]
In :
    magic_function("name_of_target_variable_2")
Out :
    ["really_important_column_a", "really_important_column_f" ...]

How can I obtain this result? Is there a way, in the first place?


texts: What is the best reference for multi-class SVM?

texts: Can someone suggest some papers about the multi-classification methods by SVM? One against all? A good survey or paper which clearly describes all of the steps.


texts: I recently came across the terms Word2Vec, Sentence2Vec and Doc2Vec and kind of confused as I am new to vector semantics. Can someone please elaborate the differences in these methods in simple words. What are the most suitable tasks for each method?


texts:
I have implemented KNN using a custom DTW metric with sci-kit learn and as shown below:
def dtw(t1, t2):
    distance = fastdtw(t1, t2)[0]
    return distance
model = KNeighborsClassifier(n_neighbors=1, metric=dtw)

Additionally, I also implemented standalone DTW for template matching as shown below:
 def calc_dtw():
    predictions = []
    for i in range(len(X)): #X is an array containing X values to be predicted
        distances = []
        for j in range(len(self.X_)):
            distances.append(fastdtw(X[i], self.X_[j])[0]) #self.X_ are the templates
        closest = distances.index(min(distances))
        predictions.append(self.y_[closest])

     return predictions

Both implementations of DTW use the same 'fast dtw' library. I ran a test to calculate how long it takes for each algorithm to compute a prediction and results showed that kNN-DTW is faster than standalone DTW. Shouldn't standalone DTW (quadratic time complexity) be faster than kNN-DTW (cubic time complexity)? Is this because of an error in my DTW implementation?
Thank you in advance for your help.


texts: How to classify very short text for spend analytics?

texts:
I have a very small description of the item, example below-

ACCELATOR SHAFT
ADAPTOR –EGR PIPE & EGR VALVE(125KVA)
ADAPTOR-EGR COOLER AND VALVE-125 KVA
ADJUSTING LATCH - ALTERNATOR-125 KVA

Some items are just one word long and maximum number of words in a sentence are around 6.

These items need to be classified into category and a sub category for further spend analysis.

This is a generic problem and many of our clients have similar problems and it takes a lot of time in classifying this manually so we are trying to develop a ML/NLP model to expedite this process.

For problem at hand we have around 2000+ unique items into 6 categories, each category having around 300 items.

We plan to tag around 100 items in each category and expect the model to classify the rest.

How do we approach the problem, If someone already dealt with such a problem in past, guidance will be helpful
Does it make sense to go with tokenization/TFIDF approach with such a small description
What other Neural Network based methods we can look at (LSTM, RNN etc, but will 100 tags per category be sufficient for training)


texts:
I am a beginner in deep learning and recently I am trying to understand the structure of Wavenet. (for more information, please refer to the paper http://sergeiturukin.com/2017/03/02/wavenet.html) Nevertheless, I had some questions over the overall architecture. As can be seen from the graph, the residual blocks are stacked together, and my first question is, why are they stacked together?

Moreover, I am not so sure how the skip-connection works. From my understanding, in a very deep neural network, we use skip-connection to skip several layers to directly add to the result several layers later. In this case, we can see several layers skip-connects to the part that processes the result in order to produce the output. Does this imply that, without skip-connections, we have to go from the bottom layer of the neural network to the top layer in order to produce the output? If not, how does the skip-connection work in this case?

Finally, I had a question about the top arrow in the graph (around the "Residual"). I am confused by what this arrow is trying to do, and how does it contribute to the overall process? Thanks in advance!


texts: LSTM input layer shape in Keras using functional API

texts:
0
I am trying to implement LSTMs on drug data the shape of the data is given below also the model but it throws an error which is "Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2".
If I use return_sequnce = True then it gives me this error "ValueError: Error when checking input: expected Inputlayer to have 3 dimensions, but got an array with shape (29790, 1144)"
Can please somebody help that what will be the input shapes for the model thanks in advance.
############### Shape of Data ############
x_train shape (29790, 1144)
y_train_one_hot shape (29790, 65)
x_test shape (7474, 1144)
y_test_one_hot shape (7474, 65)
############# code for the model ###########
event_num = 65
droprate = 0.3
vector_size = 572

def DNN():
    train_input=Input(shape=(29790,1144),name='Inputlayer')
    train_in=LSTM(512)(train_input)
    # train_in=BatchNormalization()(train_in)
    train_in=Dropout(droprate)(train_in)
    train_in=LSTM(256)(train_in)
    # train_in=BatchNormalization()(train_in)
    train_in=Dropout(droprate)(train_in)
    train_in=Flatten()(train_in)
    train_in=Dense(event_num)(train_in)
    out=Activation('softmax')(train_in)
    model=Model(input=train_input,output=out)
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model
dnn.fit(x_train, y_train_one_hot, batch_size=128, epochs=1, validation_data=(x_test, y_test_one_hot),
                    callbacks=[early_stopping])


texts: I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find keywords arranged in a cluster. I tried to auto summarize using markovify library in python but the summary it created or the sentences did not make much sense. Any suitable suggestions are welcome. Thank you


texts: How to turn a series that contains pandas.datetime.timedelta to floats e.g. hours or days?

texts:
I want to plot some data that contains timedeltas. However, the numpy plot() and errorbar() return:

TypeError: float() argument must be a string or a number, not 'Timedelta'

The question is how can I efficiently convert the timedeltas to a float or integer number. E.g. when I use minutes as base:

0 days 01:30:00 -> 90

or hours

0 days 01:30:00 -> 1.5

Similar to Series.dt.days which would return 0 in this case.


texts: What's the point with neural networks if you can only predict linear test data?

texts:
So, I have tried all the different activation functions listed on https://keras.io/api/layers/activations/. I can indeed approximate any nonlinear function in the training range perfectly well - but for any data outside the training range, I have a model that is limited to linear functions. For example, I tried to approximate the sinus function and had great results within the training range but was left with a linear function outside this range. I used a network with 3 hidden ReLU layers(with 16 units per layer) and an affine output layer. Here is the good approximation within the training range:

and the predictions outside the training range:

The same appends when trying to approximate any other nonlinear function. This is unfortunate because you would rather want your model to generalize well than to perform good exclusively on training data :(
I could of course use a sine explicitly as an activation function but that seems to ridicule(maybe a strong word but you get the idea) the "self-learning" of neural networks.
This seems to make neural networks look very limited in my eyes - am I missing something?
I really appreciate your time!


texts: How to read in all text files from UNIX bash directory in Cloudera's Python API

texts:
I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like this:

import glob
files = glob.glob("*.txt")

Is there an equivalent way to do this in the UNIX environment? I know I can read in one file by

infile=sys.stdin

but as far as reading all in from one directory I'm not sure.

Thanks!


texts: Strong overfitting accompanying strong class imbalance

texts:
I'm training an xgboost binary classification model. The data I have is around 600k and positive is only 0.1% of it. I tried to use all overfitting prevention techniques xgboost has to offer (tune eta, gamma, min_child_weight, subsample feature/data, early stopping etc). However, my model either strongly overfits but with decent metric on test set or does not overfit too strong but general performances are bad. I've listed two training results (metric is auc-pr) here,

AUC-PR
130 trees
3800 trees

train
0.18
0.98

test
0.05
0.45

validation(early stopping)
0.06
0.46

I checked for data leakage but doesn't seem to have any. My question is:

Can I use the model with 3800 trees, even though it's overfitting, it's getting decent results on test set?
Do I absolutely have to be strict in preventing overfitting for this strong class imbalance scenario? If so, what are other techniques?
Anything I did wrong?


texts: Tool for finding correlations between data after some event

texts: I'm looking for a product that allows us to take in a collection of datastreams, and then after some event, will find any data that changes or correlates with that event. (For example, having a headache, and identifying that I drank too much beer last night and didn't drink enough water)


texts: linear regression with "partitioned" data

texts:
I am not a stats/math expert by any stretch of the imagination, but have been trying some linear regression with census data and think I have run across a fundamental problem/obstacle.

The response variable is a percent. The independent variables are percents by state. Here is an example of what I only know to refer to as  "partitioned" data. One of the groups of independent variables is "lighting" and has the following variables and values for one city (for example):

electricity - 40%
kerosene - 10%
candle - 25%
generator - 15%
solar - 5%
other - 5%

When added together, those six variables = 100%. There are only six choices and a household can only have one type of "lighting". Using all six of these variables, causes my model to blow up producing coefficients that are off the charts and clearly something is wrong.

Any insight on how to handle or at least still make some use of partitioned data with linear regression? Is there another name for these types of variables?


texts: Building Stacking machine learning model using three base classifiers

texts:
I did a stacking using three base classifiers RF, NB, KN N and metamodel random forest or SVM using sklearn library

But which is strange each time i change the metamodel i got the same results. Is it normal ?????


texts: Properties of embeddings that work best in recurrent layers

texts:
Let's assume that you have a simple recurrent layer that works on top of sequences of dense embeddings, predicting next embedding. The embeddings are generated by an autoencoder.
What properties should the embeddings have to work best in recurrent (LSTM) layers? Is there any general knowledge regarding such properties?
For example: the embeddings should be uncorrelated, the autoencoder weights should be orthogonal, you should use regularizations on weights/activations,..


texts: Get an equation from the relationship among variables

texts:
I have an agricultural dataset of crop yield (in kilograms) under environmental conditions such as humidity, rainfall, sunlight over few years. I want to find out relationship between the environmental factors and the crop yield.

A score is assigned based on the crop yield each term.

The closer the environmental conditions to the ideal conditions, higher the crop yield.
The highest crop yield under ideal conditions is scored as 10.
For non-ideal environmental conditions, the score can be between 1-10.

The output I'm expecting is an equation which anyone can plug-in the environmental conditions, and get a score between 1-10 on how ideal the conditions are.

Could you help me on choosing an approach to achieve this?


texts: Context based Named Entity Disambiguation and Extraction

texts:
Named Entity Disambiguation generally deals with the same entity meaning different in different contexts. For example, Tesla can be a company, Tesla can be a car, Tesla can be a person.
But the NED task I am looking at is a little different. I want to disambiguate between the named entities in a sentence based on the context in which it belongs.
Let's take the following sentence as an example

Elon Mask sent a car to space which Bill Gates applauded.

Now, with a trained model I can extract both 'Elon Mask' and 'Bill Gates' as Named Entities. However, I have a specific context in mind when doing the extraction - which is 'sending cars into space'. So, the only named entity I am concerned with is 'Elon Mask'. So, is there a way to disambiguate between the extraction of Elon Mask and Bill Gates?
One option I am looking at is giving enough examples to train the model because I will be working with a fixed set of contexts. But is there something else, something more elegant that I can look at?


texts: How can I implement Double-Exponential Smoothing for my data?

texts:
I have a time series problem that I'm dealing with and I'm looking for guidance on how to implement Double/Triple-Exponential Smoothing algorithm to predict my data. My problem is Taxi demand prediction. I have Jan-2015 data.

About my data:

I have number of taxi pickups in $10$ minutes time-bins. $i.e$ each no_pickups entry is in 10 minutes bin and time-bins are from,
$\text{01-01-2015 00:00:00}$ to $\text{31-01-2015 00:00:00}.$
There are about 4464 data-points because (31(days) * 24(hours) * 60(mins)) / 10(mins) = 4464 (time-bins)

How can I use these previous values to forecast it for Feb-2015?

I would really appreciate if you could tell me how to proceed with such problem and really looking forward to detailed answer so I can follow along. If anyone can help me out do let me know.

Example of data-set:

# Here's a sample data head.

               time_bin                          n_pickups

(01-01-2015 00:00:00 - 01-01-2015 00:10:00]           50
(01-01-2015 00:10:00 - 01-01-2015 00:20:00]          108
(01-01-2015 00:20:00 - 01-01-2015 00:30:00]          160
                .........                            ...
                .........                            ...
                .........                            ...
(31-01-2015 23:50:00 - 01-02-2015 00:00:00]           32


texts:
So I wrote Linear regression from scratch using y = mx+b and ran the algorithm for 50 epochs (times) to minimize the cost and get the best parameters.

When I use Scikit Learn, I just call the Linear Regression method and fit the data-set to it then start predicting. How many epochs does fit method run ? Not only for Linear Regression but also other ML methods in general.


texts:
Simplified question:
I have a dataset of how well certain agents perform on certain tasks, and based on this I've trained a model that can make predictions for newly incoming tasks. I'd like to make an algorithm that distributes tasks amongst agents to maximise performance.

One way would be to make a prediction for each scenario - each agent assigned to each new set of tasks (note that one agent can work on more than one task at a time) -, thus iterating through all possible solutions, and summing performance and finding the max.

This, however, would be very complex - to iterate through every possible combination an finding a maximum.

Is there a better way to optimise (maximise performance) without having to iterate through all possible solutions? How could I do that?

Original question:

Let's take a look at the following setup:

There are a number of "agents"
Agents work on "jobs"
Each agent might be better at certain jobs, but the jobs get assigned by random, and not based on performance (good performance = the least amount of time spent working)

I'd like to know if there's any ML algorithm that can learn how to assign jobs to agents as to minimise time spent working (maximise performance).

I know that with methods like regression or classification, or even deep learning you can teach the algorithm to predict a target variable, but in this case I do not want to reach a target variable, but want to increase performance (minimise time spent working). Is there such an algorithm/method that could "learn" from past performance reviews and assign new jobs to agents to maximize performance?

Edit:

A bit more formalised:

I teach an SVM (or a regressor, or a neural net), that will be able to predict how well a certain agent A1 performs on a certain (type of) task T1 (call this performance P1). So when a new task comes in, I'll be able to predict P1 based on A1 and T1. BUT! And here comes the question. Now that I can predict P1 based on A1 and T1, how do I use this knowledge to actually assign ALL the tasks - T1...Tn amongst all the agents A1...Am as to maximise the performance: sum(P1, P2, ..., Pi)?


texts: Triplet optimization producing a weird diagonal line?

texts:
I'm pretty sure this is the right forum for this, or let me know otherwise, I'll happily move this to a better place.
I have a strange problem. I've written an algorithm designed to take three files of UNIX timestamps, and produce a list of triplets in order of closeness. Each triplet is unique (no two triplets share an element), each triplet has one element from each file, and each triplet {x,y,z} is created so as to minimize max(x,y,z) - min(x,y,z).
When I run the algorithm and did a visual examination of the output, everything looked great. But, when I plotted the data, something weird happened. I plotted the resulting triplets on a 2-dimensional histogram. The horizontal axis for a triplet {x,y,z} was x-y, and the horizontal axis was x-z. I ended up with a weird, perfect diagonal line running from the bottom left hand corner to the upper right hand corner: https://filebin.net/3d0qfqi7ice1uik0/lines1.pdf?t=derlloqu .
I thought something might've been wrong with my algorithm, so I tried two additional algorithms. First, I wrote a binary search algorithm. Instead of finding a given element, the binary search terminated by comparing the closest found element to it's neighbors. I then saved the closest element in a list and, when finished, sorted the list in order of closeness and removed non-unique triplets. I ended up with the same, odd diagonal line.
I tried another algorithm, the most common for this particular task. Since the three files are sorted, one can use that property to find the closest possible triplet from the three files. Simply pick a starting point, and move forward or back in each of the three triplets based upon which is the largest or smallest element (I probably did a terrible job of explaining that, a better explanation can be found at: ). Again, I ended up with the same, odd diagonal line.
I thought, perhaps it's something with the data. But, when I match one file of "live" data with two files of completely random data (uniformly distributed), I get the same, odd diagonal line. Moreover, when I match three files of completely random, uniformly distributed data, I get multiple odd diagonal lines: https://filebin.net/cqw31xysci7kaddn/lines2.pdf?t=jv9srf2t .
Any idea what's going on here? Is this some artifact of this particular kind of analysis? I'm pretty certain at this point it's not my algorithms. All three were completely different, yet all three produced the same result.
If you're interested in the source code: https://pastebin.com/8gL1D6Bw


texts: Adding document vector doc2ec to seq2seq model

texts:
In general we use word2vec for word embedding in seq2seq model, is it possible to add the document vector from Doc2vec with the input words , I mean using the tag of the document as a word and its vector for emending : the picture will explain mu point a view

default seq2seq

My opinion

the doc_tag is followed by its original words so will that improve my seq2seq model ( logically ) ?


texts: Using dynamic time warping (DTW) to match only one point in reference series?

texts:
In a standard implementation of Dynamic Time Warping (DTW) or fast DTW such as those here and here every point in the two series is matched.
Question: is there a requirement for every point to be matched for the method to work or can one define a specific point in series A and only query to find the corresponding point in series B? It is unclear to me if this only works by matching all points across two series.
What I would like to do is isolate a specific section of the time series (the start and end of that section) without having to calculate all the points that come before and after as a way to reduce computational time.


texts: Explain forward filling and backward filling (data filling)

texts:
Can I understand in this way? Let me know if any statement is wrong or not accurate.

Reason of data filling:

Assume I have a consecutive data (e.g., daily log data), and partial data are missing. In order to make some calculation (e.g., mean value), we first need to assign values to the missing parts (e.g., equal to existing data)
Forward filling and backward filling are two data filling methods.

The difference is the filling direction? E.g., Tuesday data (missing) equals to Monday data (existing) is forward filling. The opposite is backward filling.


texts: I am looking for a library which converts pdf to json. Basically in that json the paragraph heading is the and the value is the content of paragraph. Is there any python library for that ? I am already using pdfminer but that just converts to plain text. It cannot persist the structure/organisation of the document. For now it is ok to not read images and table although if there is a library to do that would be great.


texts: VAR model ValueError: x already contains a constant

texts:
I'm using VAR model for multivariate time series. The structure is that although each variable is a linear function of past lags of itself and past lags of the other variables, one and/or two of the variables MAY NOT alter within the period under investigation. Out of 10 variables.

Below is a similar dataframe to the one I'm working on. The actual dataset has 190 rows.

x0 = [0,0,0,0,0]
x1 = [0.011866,0.013380,0.015357,0.024451,0.030889]
x2 = [0,2,2,3,3]
x3 = [1,1,2,3,3]
x4 = [0,0,0,0,0]

T = ['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04', '2000-01-05']
TDT = pd.to_datetime(T)
df = pd.DataFrame({'X0': x0, 'X1': x1, 'X2': x2, 'X3': x3, 'X4': x4})
df.index = TDT
df

model = VAR(df)
result = model.fit(1)

**ValueError: x already contains a constant**

Is there a way to fix this?


texts: Need for doing grad.zero_() after setting torch.no_grad() in pytorch

texts:
In the following lines of code,

# Adjust weights & reset gradients
with torch.no_grad():
    w -= w.grad * 1e-5
    b -= b.grad * 1e-5
    w.grad.zero_()
    b.grad.zero_()

we already specify torch.no_grad() so why is there a need for doing w.grad.zero_() ? The no_grad() will set the .requires_grad to be false, so I can't see why .grad.zero_() is needed.


texts: XGboost classification with very small data set

texts:
I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets.

Normally I tune the n_rounds parameters by cross-validation, but what if you have too less observations to do proper CV? For example if I have 30 variables and 4000 observations in my training data, how can I find a nice value for n_round which is not over/underfitting the training data?

Are there any "best practices" for parameter tuning (also max_depth etc.) having small datasets?


texts: What is meant by Average Content Distance in Videos generated by GANs?

texts:
I'm reading a research paper on generating/synthesizing videos:
MoCoGAN: Decomposing Motion and Content for Video Generation

To evaluate the generated videos, they have used a metric called 'Average Content Distance'. I couldn't find any material on google related to this. Can anyone please explain what Average Content Distance means?

Here is the snippet from the paper

  we first computed
  the average color of the generated shape in each
  frame. Each frame was then represented by a 3-dimensional
  vector. The ACD is then given by the average pairwise L2
  distance of the per-frame average color vectors.

What I understood from this is as follows:
For each frame, convert rgb to gray (average of color). Then for successive frame, calculate the l2 distance.
$$\frac{1}{MN} \sum_{x=1}^{M}\sum_{y=1}^{N}{(Frame_i(x,y) - Frame_{i+1}(x,y))^2}$$
This gives ACD. Have I understood it correctly?

Also, how does this metric represents quality of a video? How can this be used to compare qualities of different generated videos? You can also point me towards some references.

Thanks!


texts: Sampling trying to keep as much multivariate variance as possible

texts:
I was thinking if anyone considered a sampling technique that would try to aim keeping as much of the variance as possible (e.g. as many unique values, or very widely distributed continuous variables).

The benefit might be that it will allow development of code around the sample, and really work with the edge cases in the data.

You can then later always take a representative sample.

So, I am wondering if people have tried to sample for maximum variance before and if there is a clever way to sample with as high possible variance (of course an approximation is just fine).


texts: TF-IDF vectorizer doesn't work better than countvectorizer

texts:
I am working on a multilabel text classification problem with 10 labels. The dataset is small, +- 7000 items and +-7500 labels in total. I am using python sci-kit learn and something strange came up in the results. As a baseline, I started out with using the countvectorizer and was actually planning on using the tfidf vectorizer which I thought would work better. But it doesn't.. with the countvectorizer I get a performance of a 0.1 higher f1score. (0.76 vs 0.65)

I cannot wrap my head around why this could be the case? There are 10 categories and one is called miscellaneous. Especially this one gets a much lower performance with tfidf.

Does anyone know when or why tfidf could perform worse than count?
I need to formalize this for my thesis.


texts: advice on distance metric for knn w/image recognition

texts:
I'm getting my feet wet with machine learning and am implementing a knn algorithm on a dataset that i've created. I've created a set of images of circles and squares and want the knn algorithm to classify the test set. It does this by feature extracting the border of the circle/square with a canny algorithm from scikit-image library. This returns a numpy array of booleans. Right now my distance metric looks like so.

distance = 1024
for x in range(length):
    for y in range(length):
        if(instance1[0][x,y]==instance2[0][x,y]):
            distance -=1

I understand it's poor for a bunch of reasons, but i'm having a difficult time figuring out how to do better since

1.) my images are black/white
2.) the canny edge detection algorithm returns a M x M matrix of booleans that determine if something is an edge or not.

Because of number two, i'm checking every pixel and seeing if they match, if they do I mark them as closer in distance. As you can imagine, my accuracy rate is pretty close to 50 percent, hovering right around 58. Are there any distance algorithms I'm missing that are accurate for this type of condition?


texts: Why does increasing the training set size not improve the results?

texts:
I have trained a model on a training set, which is not that big (overall around 120 true positives, and of course lots of negative examples). What I am trying to do is to improve the results by increasing the data size. I tried two approaches:

I added data from a different domain and concatenated the data with the existing one. It increased the F-score from 0.13 to 0.14.
I added the same extra data instances, but this time with a domain adaptation technique (feature augmentation). This time, I got an improvement of around 6 percent which was significant.

My question is why does the first approach help a little bit?

The additional data set was around 10 times bigger than the main data set. That is, adding around 1000 instances from a different domain increased the results 1 percent. If it didn't help at all, I think, that would be easier to explain. But now, I don't understand. If the data is biased and test set and training set are so close to each other such that the additional data set can not help, why do I get 1 percent improvement?

On the other hand, how do I get almost the same result when the main data set in this situation is just around 1/10 of the data? I don't understand the behaviour of the model. I am using a Naive Bayse classifier.


texts:
I'm looking for a quite little/medium dataset (from 50MB to 500MB) that contains photos of famous people organized by folder.

The tree structure have to bee something like this:

├── bfegan
      └── ...
├── chris
      └── ...
├── dhawley
      └── ...
├── graeme
      └──...
├── heather
      └──...

Do you know some dataset that have this tree structure?

I want to understand how my very basic neural network is able to detect face in a 'real-world' situation.

https://github.com/alessiosavi/PyRecognizer


texts: Does resampling imbalanced data decrease the precision of a model?

texts:
I have a model with an imbalanced dataset, lets say 5% of the rows are from the positive class. If I resample my data using something like SMOTE, or removing rows from the larger class (downsampling), I can change that imbalance to ~40%.
This works well and the performance in training is good (good performance on train/valid data). But when this model makes predictions in production, the precision is much lower (25% compared with 75% in training) and the recall is much higher (99% compared with 80%). As such, I am wondering whether this is because of the resampling.. The model thinks that the positive case is much more likely to occur than it actually is.
I have checked for data leakage, and cant find any issues. I am also stopping my model once performance plateaus (although many of the predictions are either close to 0 or 100% class probability, which seems odd to me). Any ideas?


texts:
I have a question about One-Hot-Encoding, something confusing me.:\
I have this sample dataset. My dataset is categorical:

F1
F2
F3
F4
Target

1
Blue
3
Car
Yes

4
Red
6
Ship
No

3
Pink
3
Cow
Yes

9
Black
6
Fish
Yes

question:
Should I use the One-Hot-Encoding for F2 and F4 too?
This is clear for me, that must be used one hot encoding for column Target. But what about the Features?
If the answer is yes why and if the answer is no why?
Thanks for your support.


texts: RL PPO Algorithm: Understanding the Value Function Loss term in PPO by OpenAI

texts:
In the Schulman 2017 PPO Paper, there is a value function loss term in the final loss in equation 9, where they state that the value function loss is the MSE of the target value and predicted value.
My question is, how do you compute the $V_t^{Target}$ term? I'm guessing it's the return or collected sum of rewards. Would that be discounted like
$V_t^{target} = \sum_{i=t}^T \gamma^{(i-t)} r_i$,
or $V_t^{target} = \sum_{i=t}^T  r_i$,
or neither?


texts: Example of binary classifier with numerical features using deep learning

texts:
I would like to get more understanding of deep learning. Browsing the web I find applications in speech recognition and hand-written digits. However I would be interested to get some guidance on how to apply this in the classical setting:

binary classifier
numerical features (each sample is a numerical vector of $K$ entries, no 2D pixels or such).

I am doing my own experiments choosing learning rates, number of hidden neurons and so on, but I would be happy to see an application by somebody more experienced.

The software that I use offers weight initialziation using Restricted Boltzmann Machines (RBMs). I wonder whether this is useful in this context and whether the other special techniques that one encounters in the literature (convolutional NN) are useful here to.

Could anybody share a blog post, a paper or personal experience?


texts: Does shuffling data for time series forecasting help?

texts:
So I am trying time series forecasting using LSTM's. The aim is to predict $Y$ given $X$ using regression.

I had already converted the input data into a sliding window format such that if my input data was of the form:

X = [x0, x1, x2,.....]
Y = [y0, y1, y2,.....]

Then I converted it into:

Xnew = [(x0, x1, x2), (x1, x2, x3), (x2, x3, x4),...]
Y    = [         y2,           y3,           y4,...]

Still, upon training my data I find a very high validation_loss.

Since validation_split takes only the fraction of the data from the end, I thought maybe I should try and randomize the data before training it. However, in that way, will time series have any meaning?

I found a similar question, but I had apparently already tried what was suggested:  Is it valid to shuffle time-series data for a prediction task?


texts: LSTM Train accuracy decreases in training time during the epoch

texts:
I'm training an LSTM model for sentiment analysis on a text corpus. There's a thing that I believe is not normal because I never have seen it in training my models. At the start of the epoch, the training accuracy is something around 80 and then during the epoch, it decreases to 40. And this is happening in every epoch. The model starts with high accuracy and then the accuracy decreases!
I never have seen such behaviour from a model while training. I believe if it's something is wrong it must be the data because I've been using this exact model architecture for topic classification for another dataset and the training behaviour was normal.
My question is this behaviour of the model a normal thing? If it's not, what is wrong with the data that causes this behaviour?


texts: How to build the following the string in R?

texts:
I am trying to build a long string of the form.....\'name1\',\'name2\',\'name3\',..etc. How do I do it ? I am trying to do it using R ? By hand is impossible as there are currently over 200 such names and in future I may have more. So I need to automate the process. However, every time I tried to use R's paste0 function I am getting a different result. Here is what I did:

t <- c()
for( nms in name){ t <- paste0(t, paste0("\\'",nms,"\\'", sep = ",") ) }

However, the result turns out to include double backslashes instead of one, like this "\\'name1\\',\\'name2\\',\\'name3\\' etc. I couldn't get a single backslash as I want.

Any help would be appreciated.

Thanks.


texts: what is the difference between "fully developed decision trees" and "shallow decision trees"?

texts:
As reading Ensemble methods on scikit-learn docs, it says that

  bagging methods work best with strong and complex models (e.g., fully
  developed decision trees), in contrast with boosting methods which
  usually work best with weak models (e.g., shallow decision trees).

But search on google it always return information about Decision Tree.

I'd like to know the detail of the two trees mentioned in the doc, what's the fully developed and shallow meanings.

Update:

About why bagging work best with fully developed and why boosting work best with shallow.

First, I think complex models (e.g., fully developed decision trees) means such a data set has a complex format be called as fully developed decision trees.

After I read above quote over 20 times and rapaio's answer, I think my poor English lead me to the wrong road(misunderstand). I also mistake shallow as shadow , which make me confusing a long time....Now I understand the meanings of fully developed and shallow .

I think the quote is saying bagging work best with a models(already trained) which algorithm is complex. And boosting only need simple model.
Both bagging and boosting need  many estimators, as n_estimators=100 in scikit-learn examples.

If n_estimators=100 :

bagging need 100 fully developed decision trees estimators(models)
boosting need 100 shallow decision trees estimators(models)

Does my thoughts is right?
Hope my update can help non-native speakers.
e.g. means for example, so there are other models can use for bagging and boosting. How about changing the model to svm or something else? Or both of they need a tree base model?


texts:
I was wondering what percentage of Machine Learning Algorithms are iterative in nature; i.e. I can run them sequentially, on different set of data and the model refines itself. Are there any algorithms which support this mathematically, moreover, is there some tradeoff in runtime or accuracy of these models.

Iteration here not to be confused with : What does iteration in Machine Learning mean?


texts: Is there a python code to perform the following task in python

texts:
Is there a way by which I can find how many people rated below 60% or how much percentage of people gave a rating of less than 60%.

This is the sample rating for the product. Where a 6 represents 60%. The data is currently stored in a pandas dataframe.

  9
  6
  2
  2
  8
  5
  4
  4
  5
  4
  4
  2
  4
  5
  3
  5
  7
  3
  5
  4
  1

dataset['Z32'].describe() # gives me a output

count        24114.000000
mean         4.327445
std          2.086072
min          1.000000
25%          3.000000
50%          4.000000
75%          6.000000
max         10.000000

counts = dataset['Z32'].value_counts()
print(counts)

5     5831
4     3722
3     3384
6     2874
1     2836
2     2218
7     1451
8     1040
9      416
10     342


texts: Is removing poorly predicted data points a valid approach?

texts:
I'm getting my feet wet with data science and machine learning. Please bear with me as I try to explain my problem, I haven't been able to find anything about this method but I suspect there's a simple name for what I'm trying to do. To give some idea about my level of knowledge, I've had some basic statistics training in my university studies (social sciences), and I work as a programmer at the moment. So when it comes to statistics I'm not a complete layperson, but my understanding leaves a lot to be desired.

I need to make predictions on a certain dataset, all features and the target are continuous variables (interval/ratio level), and I'm using a regression multi-layer perceptron with a single hidden layer. The $R^{\textrm{2}}$ is relatively low when fitting on the entire dataset, so I was hoping to improve the predictive power by doing cluster analysis and fitting multiple regressors, on each cluster separately. This didn't work for me, the highest cluster $R^{\textrm{2}}$ was the same as for the entire set, so I dropped that approach. I've tried multiple clustering algorithms, haven't noticed much of a difference among them, but I haven't done an exhaustive search, so I may have missed something.

What I ended up doing is, I fit the model on the training subset, identify the data point for which the prediction error in the test subset was greatest, and remove that one data point. The train-test split is done randomly, so the probability of some "bad" data point remaining by chance is pretty low. I repeat this process until I get an $R^{\textrm{2}}$ I'm satisfied with, after which I designate the remaining data points as belonging to e.g. "Group 1". This whole process is then repeated on the entire dataset minus Group 1, and ultimately all of the data should be divided into groups inside which reasonably reliable predictions can be made. To give some idea about the data: $R^{\textrm{2}}$ on the entire set of about 11000 data points hovers around 0.7. In Group 1, I kept 7000 data points for which I can get up to 0.9. The remaining groups also have acceptable (for my standards) $R^{\textrm{2}}$s.

After all of the data is split into groups in this way, I expect to be able to train a classifier on the features, to predict the group that a certain data point belongs to, and use the appropriate regression model for that group to predict the final target variable. My question is, is there some methodological flaw in this approach, specifically the removal of certain data points based on  prediction errors? Am I introducing some artefacts into the data, or something like that? As far as I can tell there's no leakage of information about the target. What this looks like to me is, some roundabout way of doing cluster analysis, as it seems to exclude "outliers" in a certain sense, but I couldn't be more precise, and that may not be the case at all.

Note: all mentions of $R^{\textrm{2}}$ refer to predictions, i.e. the score I get on the test set, not training.

EDIT: I remove data points only from test subset (which changes randomly in each iteration), but now that I think about it I guess there's no reason to limit exclusions to the test subset, as a worse prediction may happen in the training subset. I'll update when I try it. Also, I haven't yet tried fitting the classifier, so I suppose I may not get any better results with this final model; regardless of the results I get, I'm interested in the validity of this approach. Also, if someone knows of a theoretical limitation here, if there's a reason why I couldn't get better results in principle with this approach, I'd like to know.


texts: Random Forests with complementary features

texts:
In my dataset, I have 2 features that are not only correlated but that makes sense only in the presence of each other. For instance, one would be the number of times a task was attempted and the other one would be the number of successes.

As mentioned, it seems to be me that taken one of the 2 individually does not give any information. Should I do a scheme where if I pick one of them in a tree of my RF, I automatically include the other one?

And if so, is it possible to do so using the RF class from scikit-learn?

Thanks!


texts:
I have a feature that income of individual. It ranges from 10k to 116 Million. I have about 300k+ records.

Clearly, I cannot use this feature as is as it will distort the model output and there are outliers..

I was thinking of normalizing all values from 0 to 1. And also eleminating outliers. Like if most values from from 30k to 200k, then people with income of 200k would get a score around 1.

I have 2 questions?
1) Is this approach reasonable or should I think about it differently.
2) and suggestions on how to normalize and/or scale values to avoid outliers?


texts: What to do when testing data has less features than training data?


texts: What to replace outliers with? (supermarket transaction data)

texts:
I have a transaction dataset from a supermarket. Let's say the average spend is $50.
I want to get each customer's average spend and rank them based on where they fall based on this $50 average spend.
For example:
John Doe's average spend is 150% of the overall average spend = "Gold Customer".
Jane Doe's average spend is 25% of the overall average spend = "Bronze"
etc
etc
Now to calculate the overall average spend of the supermarket, I want to get rid of outliers. This is a grocery store, but they may have a TV on sale every now and then. So there are some single transactions that may be $600+. I want to get rid of these.
The question is, what do I replace them with?
I look for transactions 3 standard deviations above the mean. These are my outliers.
I don't want to replace them with the mean / median. I could be losing some of my "Gold" customers if I erase their bigger purchases and replace them with $50.
Could I replace the outliers with mean + 3*std_dev ?
I'm using python, so the current code is:
# set threshold above which transaction will be labeled an outlier
# this is the average spend plus 3 times standard dev

value_threshold = (df['amount'].mean()+(df['amount'].std()*3))

# now replace any outlier with the value threshold.
# this will ensure any big spenders stay big spenders so I can rank them accordingly

df['amount'] = np.where(df['amount']>value_threshold ,value_threshold ,df['amount'])

Does my approach make sense or am I breaking any rules?
Should I be using the median instead with interquartile ranges to find outliers?
Distribution:

After removing the outliers using my method above. Notice we still have big spenders in the data to the right of the chart. (my gold customers)

Bonus
Boxplot of my transaction data before fixing the outliers. It's horrific:


texts: How to calculate TPR and FPR for different threshold values for classification model?

texts: I have built a classification model to predict binary class. I can calculate precision, recall, and F1-Score. Now, I want to generate ROC for better understanding the classification performance of my classification model. I do not know how to calculate TPR and FPR for different threshold values.


texts: How does SQL Server Analysis Services compare to R?

texts:
This may be too broad of a question with heavy opinions, but I really am finding it hard to seek information about running various algorithms using SQL Server Analysis Service Data Mining projects versus using R. This is mainly because all the data science guys I work with don't have any idea about SSAS because no one seems to use it. :)

The Database Guy

Before I start, let me clarify. I am a database guy and not a data scientist. I work with people who are data scientist who mainly use R. I assist these guys with creating large data sets where they can analyze and crunch data.

My objective here is to leverage a tool that came with SQL Server that no one is really leveraging because no one seems to have a clue about how it works in comparison to other methods and tools such as R, SAS, SSPS and so forth in my camp.

SSAS

I have never really used SQL Server Analysis Services (SSAS) outside of creating OLAP cubes. Those who know SSAS, you can also perform data mining tasks on cubes or directly on the data in SQL Server.

SSAS Data Mining comes with a range of algorithm types:

Classification algorithms predict one or more discrete variables,
based on the other attributes in the dataset.
Regression algorithms predict one or more continuous variables, such
as profit or loss, based on other attributes in the dataset.
Segmentation algorithms divide data into groups, or clusters, of
items that have similar properties.
Association algorithms find correlations between different attributes
in a dataset. The most common application of this kind of algorithm
is for creating association rules, which can be used in a market
basket analysis.
Sequence analysis algorithms summarize frequent sequences or episodes
in data, such as a Web path flow.

Predicting Discrete Columns

With these different algorithm options, I can start making general predictions from the data such as finding out simply who is going to buy a bike based on a predictable column, Bike Buyers, against an input column, Age. The histogram shows that the age of a person helps distinguish whether that person will purchase a bicycle.

Predicting Continuous Columns

When the Microsoft Decision Trees algorithm builds a tree based on a continuous predictable column, each node contains a regression formula. A split occurs at a point of non-linearity in the regression formula. For example, consider the following diagram.

Comparison

With some of that said, it seems I can run a range of algorithms on the data and also have various functions available to me in SSAS to run against the data. It also seems I can develop my own algorithms in Visual Studio and deploy them to SSAS (if I'm not mistaken).

So, what am I missing here in regards to languages and tools from R? Is it just that they have more flexibility to deploy and edit complex algorithms versus SSAS etc?


texts: Multiclass Regression for density prediction

texts:
This is my first question in the DS-community, so I'm happily willing to accept any kind of (meta-) advice :).

I have time-series data for a set of users (~100), whereas each 15 min it is logged to which antenna (~80) they were connected (similar to cell phone connections).

Based on this data, I created a density vector, which for some time t (i.e. 01.01.2016 at 06:00) counts how many users are connected to which antenna.

Such a density object (for same day and time as given above) might look like this:

100: 5;
101: 2;
102: 3;
103: 0;

whereas the first number refers to some ID of the antenna, and the 2nd number refers to the number of users connected to the antenna.

I'm planning to feed this time-series data to a recurrent neural network.

The results should be the predicted number of users connected to the antenna at the next time step (so every 15 minutes). So it might predict for 01.01.2016 at 06:15:

100: 7;
101: 0;
102: 1;
103: 2;

Now I'm wondering what should the output layer be like? Regarding number of neurons and activation function especially. I've been reading quite a lot about multinomial logistic regression but some confirmation would be nice.

If it should output the predicted number of users per antenna, it should have probably the same number of neurons; so 80 it would be, just as in a multi-class classification scenario with a softmax activation function.

So what I need is a different activation function, but even after reading quite a bit, I couldn't wrap my head around it yet.

i.e. Get multiple output from Keras proposes to use a linear activation function, but in their case, they tried to predict the next 3 values, by using regression; whereas I am trying to predict the next 1 value for a set of antennas.

PS: For constructing the Neural Network, I'm using (Tensorflow-) Keras.

PPS: For feeding the neural network, I would generate the density vector for all time steps, and then feed batch-wise with batch_size = 80 (number of antennas). Out of curiosity: I happen to have only 1 feature, so the input_shape probably be (batch_size, 1, 1*80); If I was to have 2 features, would it then be (batch_size, 1, 2*80)?

PPPS: Not even quite sure how to name this problem. I think it would probably be called a (time-series) multiclass regression problem but I couldn't find any example with the same name (left aside the multinomial logistic regression).


texts:
I'm attempting to model a simple process using a Markov Decision Process.

Let $A$ be a set of $3$ actions : $ A \in \{b,s\}$.
$T(s,a,s')$ represents the probability of if in state $s$ , take action $a$ and end up in state $s'$

Notation for the MDP diagram is as follows :

Here is my MDP diagram which models 7 states:

The outgoing actions for each state sum to 1.

$T(1,b,2) = .7 $

$T(1,b,3) = .3 $

$T(1,s,4) = .9 $

$T(1,s,5) = .05 $

$T(1,s,6) = .05 $

I've tried to keep this as simple as possible to check my understanding. Are my representations & probabilities correct ?


texts: Data Science oriented dataset/research question for Statistics MSc thesis

texts:
I'd like to explore 'data science'. The term seems a little vague to me, but I expect it to require:

machine learning (rather than traditional statistics);
a large enough dataset that you have to run analyses on clusters.

What are some good datasets and problems, accessible to a statistician with some programming background, that I can use to explore the field of data science?

To keep this as narrow as possible, I'd ideally like links to open, well used datasets and example problems.


texts: How effective would this pseudo-LDA2Vec implementation be?

texts:
For my site I'm working on a chat recommender that would recommend chats to users. Each chat has a title and description and my corpus is composed of many of these title and description documents. I was curious about training an LDA2Vec model, but considering the fact that this is a very dynamic corpus that would be changing on a minute by minute basis, it's not doable. I was thinking of just doing standard LDA, because LDA being a probabilistic model, it doesn't require any training, at the cost of not leveraging local inter-word relationships. The other added benefit of LDA2Vec was that I could get accurate labeled topics. So I thought, what if I use standard LDA to generate the topics, but then I use a pre-trained word2vec model whether that be trained locally on my corpus or a global one, maybe there's a way to combine both.

The junk below draws heavily from the stuff in the lda2vec paper:

https://arxiv.org/pdf/1605.02019.pdf

topic 0: SpaceX, Nasa, Asteroids, Rover

w_j = target word

w_i = pivot word

d_j = document vector or what I'm calling the topic in my scenario

c_j = context vector

c_j = w_j + d_j

d_j = [v_1, v_2 ... v_n]

This is the loss function from the paper

I can then iterate each word w_j for each pivot w_i such that the loss is minimal:

so first iteration pivot word is w_i = SpaceX and w_j iterates over [Nasa, Asteroids, Rover]

second iteration pivot word is w_i = Nasa and w_j iterates over [SpaceX, Asteroids, Rover]

once it's done I'll have a document vector d_j, I'll find the closest word in the embedding space to d_j and I presume that's a good label for the topic. Either I'm being extremely naive or this maybe works. Any input would be much appreciated, thanks in advance.