texts: Decision Trees change result at every run, how can I trust of my results?

texts:
Given a database, I split the data in train and test.
I want to use a decision-tree classifier (sklearn) for a binary classification problem.
Considering I already found the best parameters for my model, if I run the model on the test set I obtain at each run (considering the same hyper-parameters) a different result. Why that?  Considering I am using as metric the accuracy score, I have variations from 0.5 to 0.8. Which result should I take as correct, because I am not sure If it's correct to take the best result on the test set, or maybe I should consider an average of the results... I dont know.
Example, scoring for gridsearchCV is accuracy. I obtain
Grid scores on development set:

0.627 (+/-0.129) for {'max_features': 2}
0.558 (+/-0.152) for {'max_features': 3}
--> Best parameters: {'max_features': 2}
    Best score: 0.626876876876877 (this is the accuracy)

Using the best estimator on the test set, I obtain accuracy 0.83.. which I think it is only due to the case. In fact I try again and the result is this

Grid scores on development set:
0.584 (+/-0.126) for {'max_features': 2}
0.572 (+/-0.168) for {'max_features': 3}
--> Best parameters: {'max_features': 2}
    Best score: 0.5840215215215215
On the test set accuracy: 0.62 !!

So, how can I trust my results?
Second, would not it be better to use CV on all data, instead of just splitting at the beginning the dataset in test and train just one time?

I read about random_state but the problem is that the results depend on which value I use.. example
random_state = 2 --> accuracy 0.6 (on test set)
random_state = 6 --> accuracy 0.79 (on test set)
so basically this does not resolve my problem.
How can I validate my model if I dont know which one to use?


texts: Text vectorizer that capture feature offset in the text?

texts:
I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification.
I believe the information I need tends to be in the beginning of the document, so I would like to somehow capture the offset of each feature per document (either of the first appearance, or the mean offset over of all appearances).
Is there some vectorizer that can do that? or some other method of extracting this information efficiently?

Thank you!


texts: What's the correct form of use the real coded genetic algorithm?

texts:
I'm new to genetic algorithms, but I haven't found specific info about real-coded GA's. I want to do antenna array optimization by using the real values of antenna position, phase, and amplitude, but I've only found binary encodings. Although the binary encodings seem easier during crossover, but I would like to use real continuous values, but I don't know how to apply crossover.
In MATLAB, I've been using an error value as a fitness function obtaining a vector with all the error data and using this to select the new population. I'd also appreciate a source describing the real continuous encoding process.


texts:
There is the opportunity to fit decision trees with other decision trees. For example:
adaclassification= AdaBoostClassifier(RandomForestClassifier(n_jobs=-1))
adaclassification.fit(X_train,y_train)

I got better results with random forest, so improved the result from adaboost with the random forest classifier. However I dont understand what´s happening here? It sounds easy: adaboost uses a random forest to fit it´s classification. But what´s mathematically going on here?
Adaboost is made of the residuals as a sequence (boosting). Random forest (bagging) built a forest out of trees.


texts: Performance and architecture of neural network for increased dimensions

texts:
I posted this question on Cross Validated before I realized that this existed. I think it is better suited here and got no answers over there so I have deleted other post. I have reproduced the question below:

I have been playing around with the neural network toolbox in MATLAB to develop an intuition for how the architectural requirements scale with feature dimension.

I put together a simple example, and the results have surprised me. I am hoping someone can point to either (a) an unrealistic expectation of mine, or (b) a mistake/misuse of the neural network toolbox.

The example is as follows: I have a simple un-normalized one-dimensional Gaussian that I am trying to learn. I do the following:

x = -5:0.2:5;
y = exp(-x.^2/2);

net = feedforwardnet(2);
net = configure(net, x, y);
net = train(net, x, y);
y2 = net(x);

plot(x, y, 'o', x, y2);
legend('Data', 'NN');

This gives me good results. I get the plot below.

Now, I try to extend this to 2 dimensions and this is where I run into trouble. I don't think I'm asking too much. My data is not noisy, or is it sparse. I figure if I double the number of neurons that should be sufficient for an increase in dimensionality. Here's my code:

x1 = -5:0.2:5;
x2 = -5:0.2:5;
[x1g, x2g] = meshgrid(x1, x2);
xv = [x1g(:)'; x2g(:)'];
yv = exp(-dot(xv,xv)/2);

net = feedforwardnet(4);
net = configure(net, xv, yv);
net = train(net, xv, yv);
y2v = net(xv);
plot3(xv(1,:), xv(2,:), yv, xv(1,:), xv(2,:), y2v, 'o');
legend('Data', 'NN');

The plot I get is this:

This is pretty poor. Perhaps I need more neurons? Maybe if I double the number of dimensions, I need to quadruple the number of neurons. I get this for 8 neurons:

Maybe with 8 neurons I have a lot of weights to fit, so let me try training with regularization. I get the plot below with trainbr:

It's only at around 16 neurons that I start getting something I would consider reasonable.

However, there are still oscillations which I don't like. Now I know I'm using it out of the box in a naïve manner, or perhaps I'm expecting too much. But this simple example resembles the real problem I want to tackle. I have the following questions:

Why is it that an increase from 1 to 2 dimensions increases the number of neurons required to get a decent fit considerably?
Even when I go to a larger number of neurons, I get oscillations that are going to be a problem in my real world application. How can I get rid of that?
Most resources on NN that I've read indicate a substantially lower number of neurons. They usually state something like "equal to or less than the number of input variables". Why is that? Is a multidimensional Gaussian a pathological case?
If I need to be more intelligent with how I treat my network for a given number of neurons, what do I need to do? I tried retraining the network to see if it was a local minima issue, but I generally get a similar fit.
Anything else that may be remotely useful to this issue is appreciated!


texts: How to create multi-hot encoding from a list column in dataframe?

texts:
I have dataframe like this
Label   IDs
    0   [10, 1]
    1   [15]
    0   [14]

I want to create a multihot encoding of the feature IDs. It should look like this
Label ID_10 ID_1 ID_15 ID_14
  0     1     1    0     0
  1     0     0    1     0
  0     0     0    0     1

The goal is to use them as features in Keras. So it is also acceptable to make this transformation using a Keras API.


texts: Human Activity Recognition - How to process gyroscope and magnetometer data

texts: I've found many articles (like this one) on HAR that mention using the gyroscope and magnetometer because it improves the precision of HAR performed just with the accelerometer. However no one mentions how this data is processed. For acceleration usually I calculate the mean, standard deviation, variance, energy and frequency domain entropy; but since the magnetometer just shows the degrees from magnetic or true North and it's not an acceleration value, does it make sense to use the same features for the magnetic field values? should I instead use the speed of variation of the magnetic field and gyroscope values? I can't find any article that explains how to process this data.


texts: Why is the memory of common GPUs so constrained?

texts: Often we find that the NN training process could be highly constrained by GPU memory size, like in object detection models the training batch size could be limited to 1 or 2, which has inflicted those working in such tasks for long and led to a number of tricks proposed sorely for the work-arounds on such limited memory/batch size scenarios. So why is the GPU memory limited to, e.g. 16GB on common P100/V100, compared to up to 256GB RAM for CPU on high performance workstations? What is the obstacle for Nvidia to release a, say, 128GB V100? And what would be the cost for that?


texts: Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

texts: Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP doesn't. Why is that so? And what are the differences, pros, and cons of both topic modelling methods?


texts: What is considered short and long text in NLP (document similarity)

texts:
What is considered short and long text in NLP?
I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a source which explicitly defines short and long text in NLP yet. The goal for my task is to find similar documents.


texts: How to obtain a typical object from each class of my neural network classifier?

texts:
I have trained a deep neural network to classify vectors of a given dimension over 4 classes. Is there a way to "reverse" this neural network in order to obtain what my network considers a typical vector for each of my classes ? In a way, I would like to extract knowledge from my network.

I know that the idea of reversing the network is in a way what is behind Google dreaming machines, but if I'm not mistaken, they are using GAN.


texts: A practical max-depth of LSTM layers, when trying to overfit?

texts:
When trying to overfit the network, what is the practical maximum depth of an LSTM Neural network before it will start to fall apart? If possible, what is the state-of-the-art depth in LSTMs produced by companies like Google?

I know that in real world number of layers is implementation-dependent & number of neurons can be estimated, [1], [2] but curious if my results are typical:

I've implemented a system in C++, using iRProp+ (here is the paper, section 3.1), however, I am struggling to get the error-propagation under control when going above 10 layers.

I am just trying to overfit the network, it successfully and very quickly converges with 4-6 layers (~100 iRProp iterations and the error is down to 0.00001, where it started at around 4.00000)

The network tries to predict the next character in the alphabet (made from 26 characters), so each layer has an LSTM that works with 27-dimensional vectors. The error-propagation happens after 25 timesteps

If I crank-up to 10-14 layers, the error is really hesitant to even begin climbing down, and seems to simply oscillate around the 3.6 value. In incredibly rare cases if a weight initialization was lucky, with 10-14 layers the error will decrease, but usually it will just oscillate. Is that usual? I am using float datatype, however, tested the double datatype and the oscilations still happen, so I doubt it's anything to do with precision. Adjusting the $n$ value (acceleration) doesn't seem to affect it either

Been looking at examples [3], but so far getting impression people use a maximum of 2-4 layers...


texts: Power BI - JSON file formatting doesn't work

texts:
I have created the following test report by Power BI Desktop with one Card visual.

I saved this formatting into the JSON file (View => Themes => Save current theme) because I want that the same formatting to be used in dashboard also. In Power BI cloud service I uploaded the same JSON file but nothing happens to the Card formatting. It still looks the same gray. Why is that?

The content of JSON file is as following:

Is it because the JSON file doesn't contain references to the Card visual? But if I saved the report's theme that contained the Card visual then why didn't it build complete JSON file that could contain all report elements? How can I know what is the right reference tag for each elements. Microsoft doesn't seem to have appropriate manual also.


texts: How to do Machine Learning the right way?

texts:
I have a basic understanding about Machine Learning in general. My question is how it is done in the practical applications of it.

If I take the following definition of ML

  A computer program is said to learn from experience E with respect to
  some class of tasks T and performance measure P, if its performance at
  tasks in T, as measured by P, improves with experience E.

It talks about Experience E. What I understand from the above is the more data you give to the algorithm E increases, and that in turn increases P.

Now let's consider a scenario where you build a decision tree model from 10,000 data rows available. Now I have the model, so can I say that my model has learned and just stop there!(use that model for prediction from that point on-wards forever)?

According to the definition, I need to feed more data so the experience increases, and in turn I get a performance improvement.

So is Machine Learning a continuous process so that You cannot build the model and just stop there. Do we need to feed more data to the algorithm time by time and improve the model so that the model actually LEARNS?


texts: Automatically Detect Valid/Interesting pivot/drill downs of a dataset

texts:
Imagine you are given a tabular data set with a limited set of columns and rows and you are asked to find the valid/interesting pivot configuration by exploring the data. The brute-force option is to calculate all possible pivot configuration and somehow score the result features (such as its sparsity, count,...) and pick the ones that are scored higher. This is obviously very time-consuming.

I understand that my definition of "Valid/Interesting" is fuzzy in here but is there a more science-based approach (say using correlations, column cardinality,...) to automatically find the good pivot configuration of a given dataset? Any pointer is highly appreciated.


texts: How to run XGBoostregressor using reg: tweedie as objective?

texts:
I installed XGBoost for anaconda on windows 10 based on the instructions provided here. It seems that xgboost 0.6 is already installed. It performs well using "reg:linear". However, if I use "reg:tweedie", this error is reported:

xgboost.core.XGBoostError: b'[23:08:27] src/objective/objective.cc:21: Unknown objective function reg:tweedie'


texts: Using Random Forest Probabilities for customer sensitivity

texts:
I am trying to build out a customer sensitivity model to price increase. My hypothesis is that as "Annual Fee" to a product subscription increases so will the probability of a customer to churn.

I am planning to initially build out a classifier model with output as 0 - (Does not churn) and 1 (Churn). For each percent increase in annual fee the probability of churn should also increase. Can I use a classifiers proba function for this kind of analysis?

For eg: If a customer is given 5% increase and the output of classifier is 1 and probability is 60% then for my next data point, keeping everything constant and just changing the increase from 5% to 10%, I want to see that output of the classifier is again 1 but has a higher probability (higher than 60%)

I will really appreciate any help input on this!


texts: Subset of training set produces good results while full training set produces poor results

texts:
I have an extremely unbalanced data set: around 200 positive samples and 70,000 negative samples. To overcome this problem I have tried to over-sample the minority class as suggested in previous questions here.

Since training the classifier (in my case I am using SVM) takes a long time on my computer I first trained a classifier on a small subset of my data picked randomly, the classifier gave solid results when tested on the test set, yet when I trained a classifier using the full training data the classifier produced very poor results. Does anyone have insight into why this is happening and what should i do? I have also tried using weights but it did not produce a different result.


texts: Database augmentation using Sobel filter (aka gradient)

texts:
I'm working in PyTorch and to improve a ProGAN training I'm augmenting a database by calculating, for each image, the horizontal and vertical gradient via the Sobel operator.
Do you have any reading to recommend to better understand why it makes sense to use the Sobel operator to augment a database?


texts: Extracting data from a data frame with conditions

texts:
Given the first data frame consisting of 4 columns

libId    studId    year    freq
 1000      3       2002     3
 1000      34      2002     2
 1000      52      2004     9
 1001      17      2003     5
 1001      34      2003     1
.
.
.
.

If I were to extract only data with maximum frequency based on year for each libId this would be the desired output.

 libId    studId    year    freq
 1000      3       2002     3
 1000      52      2004     9
 1001      17      2003     5
.
.
.
.

So far I've tried

data2<- by(data1, c(data1[c(1,2,3)]), function(df) max(df$Freq))

data2 <- as.data.frame(as.table(data2))

But my output is incorrect:

libId    studId    year    freq
 1000      3       2002     3
 1000      34      2002     2
 1000      52      2004     9
 1001      17      2003     5
 1001      34      2003     1
.
.
.
.


texts: Do Activation Functions map to Higher Dimensions?:

texts:
I just started learning tensorflow and I have a question regarding activation functions used in neural networks, I watched a 3b1b video a while ago and it seems it squished the value into an interval like sigmoid does so by squishing it between 0 and 1 so we could make more concrete comparisons however while watching a tutorial today the instructor said that it projects the data points into higher dimensional spaces. I didn't really get how that's the case as it seems it's being converted to a scalar. Is there an interpretation/example for the latter claim?
This is the timestamped URL he talks about it during the couple of minutes following this.


texts: Understanding declared parameters in my Conv2d layer of my convolutional neural network

texts:
I am trying to understand the architecture of my keras model implemented by the sequential model.
Here is a piece of the code :
model = Sequential([
    #block1
    layers.Conv2D(nfilter,(3,3),padding="same",name="block1_conv1",input_shape=(64,64,3)),
    layers.Activation("relu"),
    layers.BatchNormalization(), .......

My question is why the two parameters input_shape and name are declared in the layer conv2D while they are not included in the set of defined parmeters for Conv2D() in this link https://keras.io/api/layers/convolution_layers/convolution2d/


texts:
I would like to implement a Multi Input Multi Output filtering operation, acting as fast as possible on batches of data. Here is my current implementation:

def lfilter_mimo(b, a, u_in):
    batch_size, seq_len, in_ch = u_in.shape # [B, T, I]
    out_ch, _, _ = a.shape
    y_out = np.zeros_like(u_in, shape=(batch_size, seq_len, out_ch))
    for out_idx in range(out_ch):
        for in_idx in range(in_ch):
            y_out[:, :, out_idx] += scipy.signal.lfilter(b[out_idx, in_idx, :], a[out_idx, in_idx, :],
                                                         u_in[:, :, in_idx], axis=-1)
    return y_out  # [B, T, O]

For another use case I also need the individual components of the I/O response:

def lfilter_mimo_components(b, a, u_in):
    batch_size, seq_len, in_ch = u_in.shape
    out_ch, _, _ = a.shape
    y_comp_out = np.zeros_like(u_in, shape=(batch_size, seq_len, out_ch, in_ch))
    for out_idx in range(out_ch):
        for in_idx in range(in_ch):
            y_comp_out[:, :, out_idx, in_idx] = scipy.signal.lfilter(b[out_idx, in_idx, :], a[out_idx, in_idx, :], u_in[:, :, in_idx], axis=-1)
    return y_comp_out  # [B, T, O, I]

The implementations above are parallel on the batch index B, but they do require an explicit python loop on input and output channels. I tried to compile with tools like numba and jit without success.
I also tried to parallelize the various calls to lfilter with tools like multiprocessing.pool.ThreadPool or joblib.Parallel, but they introduce a certain overhead and this approach seems to be convenient only for pretty large time sequences (T > 512).

Can I do better than that? Should I write my own MIMO version on lfilter? Can I also exploit GPU
acceleration?

Thanks!


texts: Difference between MAP@K for recommendations, MAP from Precision Recall Curves and Macro-Precision

texts:
I have been using the 3 metrics independently for a while now, but trying to figure out if they are actually 3 separate things (with similar-looking definitions/names) or there is some underlying connection between them.
mAP - Mean Average Precision is the average of the Precision-Recall curve over various thresholds. 1 2
MAP@k -
Mean (over all data points) of the AP@K (which is average precision for K predictions) 3 4 5
Macro-Precision -
Un-weighted average of classwise-precision 6 7
It would be great to hear thoughts on how to reconcile these concepts. Please do check the links for more details on what I am referring to since nomenclature might vary from place to place.


texts: What are the disadvantages of using deep neural networks compared to a linear model?

texts: We have heard a lot about the advantages that artificial neural networks have over other models but what are the disadvantages of them in comparison to the simplest case of a linear model?


texts: How to continuously train a model with a stream of new incoming data

texts:
I have limited experience with machine learning, I trained a few networks, but nothing out of the ordinary. I have the following problem but I am not quite sure how to approach it and I'm hoping to get some advice here.
I have an series of time dependent data (timestamp + value) which I want to use to predict the next couple of outcomes. Usually this problem could be solved by using an LSTM. Every few minutes I get a new datapoint which I want to store and also take into consideration for future predictions.
The problem I think I have is that I'd need to train the model every few minutes with changing input size (I want to train the model with the entire history at once, I don't know if that makes sense) and also I don't want to overfit the model over time.
I'm not sure which architecture or concept is suited best for such a problem, so I hope to find some advice here. Thanks!


texts: 'DecisionTreeClassifier' object has no attribute 'importances_'

texts:
I've this code in order to visualize the most important feature of each model:

dtc = DecisionTreeClassifier(min_samples_split=7, random_state=111)
rfc = RandomForestClassifier(n_estimators=31, random_state=111)
trained_model = dtc.fit(features_train, labels_train)
trained_model.fit(features_train, labels_train)
predictions = trained_model.predict(features_test)
importances = trained_model.feature_importances_
    std = np.std([trained_model.feature_importances_ for trained_model in
 trained_model.estimators_], axis=0)
 indices = np.argsort(importances)[::-1]
    for f in range(features_train.shape[1]):
        print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))
        plt.figure()
        plt.title("Feature importances")
        plt.bar(range(features_train.shape[1]), importances[indices], color="r", yerr=std[indices], align="center")
        plt.xticks(range(features_train.shape[1]), indices)
        plt.xlim([-1, features_train.shape[1]])
        plt.show()

Using RandomForestClassifier this code runs good but when I try it using Decison Trees classifier I get the following error:

std = np.std([trained_model.feature_importances_ for trained_model in trained_model.estimators_], axis=0)

builtins.AttributeError: 'DecisionTreeClassifier' object has no attribute 'estimators_'

Which attribute should I use see the most important feature of each model?


texts:
I am working on a regression type problem. When if run to train the model i am facing the following outputs. Is there any problem in my code?
I also used adam, it gives numeric loss and mean_absolute_error.

Train on 2753307 samples, validate on 28094 samples
    Epoch 1/50
    1218816/2753307 [============>.................] - ETA: 47s - loss: nan - mean_absolute_error: nan


texts: What is the reason behind the minimum image size in the Keras InceptionResNetV2 model?

texts:
In Keras' documentation for the InceptionResNetV2 model, it says the following:

  input_shape: (...). It should have exactly 3 inputs channels, and width and height should be no smaller than 139.

I'm trying to figure out the reason behind the minimum size of 139px for this model, because I'm looking to train this model on smaller images (128px for starters).

I can't find any clues in the model's source code or in the paper which describes the model.

Assuming this number is not arbitrary, what is the reason behind this minimum size of 139px?


texts: Are there any paper for a closed domain conversational agent

texts:
i was trying to find a closed domain conversational agent/chatbot paper in Question and answering so not long conversation, and i don't think i see any.

All the paper i can find are related to an open domain conversational agent or a persona based neural conversational model, etc. I really need this kind of paper since i want to implement a chatbot with a quite niche topic and i can't seem to find any that can help me out.

I search most of the paper from arxiv and this https://github.com/snakeztc/NeuralDialogPapers#task-bots

Any kind of help will be greatly appreciated


texts: How data representation affects neural networks?

texts:
Suppose A's possible values are ON or OFF.

Suppose I represent it as: if A ON then feature f=1 else f=0

Or, suppose I represent it with 2 features, where:

-if A is ON then f1=1 and f2=0

-if A is OFF then f1=0 and f2=1

How this kind of representation affects neural networks?


texts: Sensitivity analysis in outlier explanation

texts:
I am trying to find the outlier explanation using the sensitivity analysis. Let’s consider that my dataset contains 19 different input values and 1 output value (So overall 20 different columns are there and values are numerical). I have already made a prediction model and I am considering the values with high prediction errors are outliers/ anomalies. I have done the sensitivity analysis for individual input values but in the dataset values are correlated with some other input values, e.g. value 1 is correlated with value 3,4,7; value 2 is correlated with 5,10,18 etc.

For outlier explanation, first I am checking if input values also contain any outlying inputs, if there are some then using sensitivity check I want to find if the values are more sensitive to the output value. Because the values are correlated with other inputs so individual sensitivity analysis does not make much sense, but in the end I want to find the most influential group of input values that makes the outlying value to normal. After that I will verify if it is valid for similar outliers and then I would provide the group as an explanation for the outlier. So my confusion is how I can check sensitivity of a group using in this case? If this approach does not sound logical, please let me know where does it sound confusing?


texts: How to best handle imbalanced text classification with Keras?

texts:
I implemented a text classification model using Keras.

Most of the datasets that I use are imbalanced. Therefore, I would like to use SMOTE to handle said imbalance.

I tried both on plain text, and once the text was vectorized, but I don't seem to be able to apply SMOTE on text data.

I use imblearn and received the following error:

Expected n_neighbors <= n_samples,  but n_samples = 3, n_neighbors = 6

How can I fix this error? And is SMOTE a good idea? If not, what other ways could I deal with class imbalance?


texts: Perform several different torchvision.transforms on ImageFolder object

texts:
I am using pytorch to build DCGAN which i aim to train on custom dataset.

I have already posted a question on training DCGAN on small dataset, and of course answer was data augmentation.

But i have a problem. I am coding in PyTorch and i want to perform several different transfomrations on existing ImageFolder object which represents my loaded dataset.

Idea was to perform, lets say around 5 different transformations, and after performing each transformation i want to expand my dataset by adding the newly transformed images to it. Here is pseudo code:

list_of_transformations = ['random_crop','center_crop','color_jitter','mirror','grayscale']
dataset = ImageFolder('path_to_dataset')
for t in list_of_transformations
       dataset += dataset.t()

I am guessing this might not be the best idea, or is even considered 'stupid', but i am new to Deep Learning and Python in general. Any help would be appreciated! Thank you in advance!


texts: Why should we use (or not) dropout on the input layer?

texts:
People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it?

Adding dropout (given that it's randomized it will probably end up acting like another regularizer) should make the model more robust. It will make it more independent of a given set of features, which matter always, and let the NN find other patterns too, and then the model generalizes better even though we might be missing some important features, but that's randomly decided per epoch.

Is this an incorrect interpretation? What am I missing?

Isn't this equivalent to what we generally do by removing features one by one and then rebuilding the non-NN-based model to see the importance of it?


texts: Can Reinforcement learning be applied in image classification?

texts: So my question is can Reinforcement learning be applied in image classification?


texts: Can FTRL be applied on linear least squares? or is it just for logistic regression models?

texts:
I'm exploring follow-the-regularized-leader FTRL proximal gradient descent:
paper, reference implementation.

Everywhere FTRL is mentioned, the loss surface for the gradient decent is the LogLoss, and the model for prediction is Logistic regression.

Can I use the same algorithm for a linear least squares model? I have a problem I want to model with a linear model and define the loss by least squares, and then do FTRL to find the optimal solution - do you see any problem with that?

Thanks.


texts: error in running exe file after changing .py to .exe

texts:
I have changed my python code to an .exe file using pyinstaller and anaconda. In my python script ("untitled.py") I import opencv which I have already installed from source using cmake. After creating exe file I run untitled.exe in cmd windows. However, I get the following error:

File "PyInstaller\loader\pyimod03_importers.py", line 531, in exec_module
File "cv2_init_.py", line 96, in
File "cv2_init_.py", line 58, in bootstrap
File "cv2_init_.py", line 56, in load_first_config
ImportError: OpenCV loader: missing configuration file: ['config.py']. Check OpenCV installation.

There is config.py in OpenCV folder, but it cannot be found. I tried this:
pyinstaller -F  --hidden-import ‘OpenCV’ untitled.py

But still does not work. I wanted to ask how I can solve this issue?


texts: Type of Test to Determine Correlation in R

texts:
I have a dataset of approximately 48,000 rows each one a click of a an article, some of these clicks were also comments. For each article I have the country and subject of the article and name of person who clicked on it these are all categorical variables with different levels from each other. I want to determine which of these categorical variables has the highest association or correlation with comments i.e. based the dataset is the best variable country, subject, or person to predict whether or not there will be a comment on an article.

Which statistical method would you use to determine correlation and how would you implement it in R?


texts: Machine learning model for determining whether a transaction is cheap, fair, or expensive,

texts:
I am currently working on a final project related to data science and would like some advice.

I have a second hand bike selling data set that consists of around 100,000 observations with the attributes: {Brand, Model, Machine Capacity, Mileage, Sold Date, and Price}.

Now, I need to decide on what to do with this data. Obviously, I could use a regression model to predict the price of a given bike. However, I would like to do something different, such as, for a particular type of bike getting the information on the price range that is considered cheap (e.g.: from 1000 - 2000 dollars), fair, and expensive.

I know that this can be solved by some simple statistics such as using normal distribution, and then using the empirical rule to divide the curve into 3 categories mentioned above.

However, if I want to take the machine learning approach, what kind of technique should I use to solve this problem?

Thank you.


texts: I'd like to define our app user churn,  Generally, people will define their app user churn by some simply way, like: if the user do not logon continually 7 days or 1 month, we will define them churn user. So is there a better way?


texts: Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

texts:
I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I found many outliers. So I tried to delete them from the dataset.

The accuracy and Cross-Validation dropped by approximately 5%. I had 80% accuracy and an Cross-Val-Score of 0.8

After removing the outliers from the 3 most important_features (RF's feature_importance) the accuracy and Cross-Val-Score dropped to 76% and 77% respectively.

Here is a part of the description of my dataset:

Here is an overview of my data:

Here are the boxplots before removing the outliers:

Here are the feature importances before removing outliers:

Here is the accuracy and Cross-Val-Score:

Accuracy score:  0.808388941849
Average Cross-Val-Score:  0.80710845698

Here is how I removed the outliers:

clean_model = basic_df.copy()
print('Clean model shape (before clearing out outliers): ', clean_model.shape)

# Drop 'num_likes' outliers
clean_model.drop(clean_model[clean_model.num_likes > (1938 + (1.5* (1938-125)))].index, inplace=True)
print('Clean model shape (after clearing out "num_likes" outliers): ', clean_model.shape)

# Drop 'num_shares' outliers
clean_model.drop(clean_model[clean_model.num_shares > (102 + (1.5* (102-6)))].index, inplace=True)
print('Clean model shape (after clearing out "num_shares" outliers): ', clean_model.shape)

# Drop 'num_comments' outliers
clean_model.drop(clean_model[clean_model.num_comments > (54 + (1.5* (54-6)))].index, inplace=True)
print('Clean model shape (after clearing out "num_comments" outliers): ', clean_model.shape)

Here are the shapes after removing the outliers:

Clean model shape (before clearing out outliers):  (6992, 20)
Clean model shape (after clearing out "num_likes" outliers):  (6282, 20)
Clean model shape (after clearing out "num_shares" outliers):  (6024, 20)
Clean model shape (after clearing out "num_comments" outliers):  (5744, 20)

Here are the boxplots after removing the outliers (still have outliers somehow.. If I delete these too, I will have really few datapoints):

Here is the accuracy and Cross-Val-Score after removing the outliers and using same model:

Accuracy score:  0.767981438515
Average Cross-Val-Score:  0.779092230906

How come is removing the outliers drops the accuracy and F1-score? Should I just leave them in the dataset? Or remove the outliers that are to see in the 2nd boxplot (after removing the 1st outliers as shown above)?

Here is my model:

model= RandomForestClassifier(n_estimators=120, criterion='entropy',
                              max_depth=7, min_samples_split=2,
                              #max_depth=None, min_samples_split=2,
                              min_samples_leaf=1, min_weight_fraction_leaf=0.0,
                              max_features=8, max_leaf_nodes=None,
                              min_impurity_decrease=0.0, min_impurity_split=None,
                              bootstrap=True, oob_score=False, n_jobs=1,
                              verbose=0, warm_start=False,
                              class_weight=None,
                              random_state=23)
model.fit(x_train, y_train)
print('Accuracy score: ', model.score(x_test,y_test))
print('Average Cross-Validation-Score: ', np.mean(cross_val_score(model, x_train, y_train, cv=5))) # 5-Fold Cross validation


texts: Why activation functions used in neural networks generally have limited range?

texts:
Why do we generally use activation functions with only limited range in neural networks? for e.g.

$sigmoid$ activation function has range $[0, 1]$
$tanh$ activation function has range $[-1, 1]$

Q1) Suppose I use some other non-linear activation function like $f(x)=x^2$, that don't have any such limited range then what can be potential problems in training such a neural network?

Q2) Suppose I use $f(x)=x^2$ as activation function in neural network and I am Normalizing the layers (to avoid values keep multiplying to higher values) then would such a Neural Network work? (This is again in reference to the question I posted in heading that "Why do we generally use activation functions with only limited range in neural networks?")


texts: Can I use LSTM models to evaluate multiple, independent time series?

texts:
Let's say that I would like to predict the temperature tomorrow. I could use the approach whereby I train a model based on a time-series dataset collected from a single location (for example, see this excellent walk-through:https://blogs.rstudio.com/tensorflow/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/).

However, let's say that I want to train a model that incorporated time-series from multiple weather recording sites. In this scenario, let us say that the recording observations from different sites are non-overlapping in time. Ideally, there would be a way to train models using non-consecutive observations (e.g. those collected from different sites), that also enabled us to quantify the influence of 'site' on our prediction.

Is this simply a case where one would train independent LSTM models for each site? Or are there alternative approaches whereby the total training data set can come from multiple observation sets (e.g. sites, non-consecutive observation blocks, etc.)?


texts: Pyspark code is not performant enough when compared to pure python alternative

texts:
I transformed the existing code which was in python pasted below was in pyspark.

Python code:

import json
import csv

def main():
    # create a simple JSON array
    with open('paytm_tweets_data_1495614657.json') as str:

        tweetsList = []
        # change the JSON string into a JSON object
        jsonObject = json.load(str)

        #print(jsonObject)

        # # print the keys and values
        for i in range(len(jsonObject)):
            tweetsList.insert(i,jsonObject[i]["text"])

        #print(tweetsList)
    displaySentiment(tweetsList)

def displaySentiment(tweetsList):
    aDict = {}

    from sentiment import sentiment_score

    for i in range(len(tweetsList)):
        aDict[tweetsList[i]] = sentiment_score(tweetsList[i])
    print (aDict)

    with open('PaytmtweetSentiment.csv', 'w') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames = ["Tweets", "Sentiment Value"])
        writer.writeheader()
        writer = csv.writer(csv_file)
        for key, value in aDict.items():
            writer.writerow([key, value])

if __name__ == '__main__':
    main()

Converted Pyspark Code:

import json
import csv
import os
from pyspark import SparkContext, SparkConf
from pyspark.python.pyspark.shell import spark

os.environ['PYSPARK_PYTHON'] = "/usr/local/bin/python3"

def main():
    path = "/Users/i322865/DeepInsights/bitbucket-code/ai-engine/twitter-sentiment-analysis/flipkart_tweets_data_1495601666.json"
    peopleDF = spark.read.json(path).rdd
    df = peopleDF.map(lambda row: row['text'])
    print(df.collect())
    displaySentiment(df.collect())

def displaySentiment(tweetsList):
    from sentiment import sentiment_score

    aDict = sentiment_score(tweetsList)

    #
    with open('paytmtweetSentiment.csv', 'w') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames = ["Tweets", "Sentiment Value"])
        writer.writeheader()
        writer = csv.writer(csv_file)
        for i in range(len(tweetsList)):
            writer.writerow([tweetsList[i], aDict[i]])
            print([tweetsList[i], aDict[i]])

if __name__ == '__main__':
    conf = SparkConf().setAppName("Test").setMaster("local")
    sc = SparkContext.getOrCreate(conf=conf)
    main()

I ran both programs but didn't see any significant performance improvement. What am I missing? Please could you shed some thoughts?


texts: Predicting sales in unobserved locations

texts:
I am building a model that is trying to predict the 5 year sales production of a salesperson in every zip code in the United States. I have only 7 years of data. This task is one piece of a larger model whose purpose is to determine the most optimal location (as determined by the zip code) to place a new salesperson.  My firm has approximately 20,000  sales people located in 10,000 zip codes. These 20,000 salespeople have customers in 35,000 different zip codes.

I have historic sales information and zip code demographics. My problem is I am trying to determine the “home zip code” to place a salesperson. Only 10,000 of the 43,000 zip codes are the “home zip code” to a salesperson. My company faces different constraints in every state, so our representation is not uniform, but we are in most states. I can build a machine learning model to predict the production of the salesperson in the 10,000 “home zips”, but how should I approach the other 33,000 zip codes. I’ve thought about building a similarity index of the zip codes based on demographics, sales history and regulatory constraints and building a unique model for each these groupings, but this seems like I’m adding a lot of uncertainty with the clustering/similarity grouping not to mention smaller data sets for learning.

My data is based on an annual contracts, i.e., the customer signs a one-year agreement on an annual basis.

Any ideas how to best approach this problem would be appreciated


texts:
I am working on a text classification problem, and plan on using Naive Bayes based model.

In which cases should I consider using Multinomial Naive Bayes?


texts:
What are the major differences between Kaggle notebook and Google Colab notebook?
To work on a dataset my first step is to start a Kaggle notebook but then I cant help thinking what could be the advantage of using Colab notebook instead.
I know few differences, correct me if I'm mistaken about any:

Kaggle has a console and Colab doesn't (but I still don't know what to do with the console).
Kaggle notebook allows collaboration with other users on Kaggle's site while Colab allows collaboration with anyone using the notebook's link.
Kaggle doesn't have "Stackoverflow" instant search like Colab does.
Kaggle requires uploading the data onto the Kaggle's site while Colab allows using data from Drive.
Kaggle doesn't have the feature of uploading the notebook directly to GitHub like Colab does.
Kaggle's notebook can be made public for all the users in Kaggle to view,vote,fork and discuss but Colab's notebook remains in your Drive until you decide to share with someone or upload somewhere.

Anything to add that is technically advantageous?


texts: How to use Deep Learning or Multi-Output Regression for tiles that change value

texts:
I need some help understanding how to put together the below problem in such a ways that I can use either multiple output regression or deep learning:
The problem:
I have a grid nxm where each value changes over time depending on wether they get activated or not. The more activated they get the larger their value.
so for a 2x2 gird:
Initial:
00
00

At time t1
01
00

At time t2
02
00

At time t3
12
00

At time t4
42
10

How can I predict the future value of the tiles?
I thought of this problem like a classical ml problem where I can treat the grid like a 2d image. Thus to feed this input into a ml algorithm I would typically transform the 2d array into a single vector such as [4210] for t4. Basically each tile in the grid is a feature. But instead of labelling a time as an object I want to out put the future value which is continuous.
I thought of this like weather forecasting problem and thought I could represent this data similarly and predict future values, but I would need my model to output a value for each tile. It would be great if there was a way to predict these values 2 or 3 hours into the future. I will have a set of historical values for the grid.
Any help, points, references, or ideas would be helpful. The only example I have found was from tensor flow website https://www.tensorflow.org/tutorials/structured_data/time_series that talks about weather temperature forecasting and outputting multiple outputs. If there is a simpler way I would like to learn about that as well.


texts:
I am Creating a Custom NER (named entity recognition ) Model using bi directional LSTM and CRF.

During Study on Ner i see all example includes Multiple entities per sentence.
For eample this sentence includes 2 entities

  (jhon lives in Us)  jhon = S-Per , US=S-Country

Question 1:

  Can we Create a model using (bi lstm crf) where we only want to predict 1 entity.?

Question 2:

  In CRF States of the neighbors affect the current prediction so
  predicting 1 entity per sentence seems difficult specially  with CRF?

Question 3 :

  if i Cannot achieve this with CRF can I use Bert to train a model
  having 1 entity per model?

Thanks In advance.


texts: how to remove unwanted characters from data

texts:
I need to write a function which identifies and removes the "*" character after some numeric values in a vector. I also need  that the resultant vector is a numeric vector.

c("21,34,99*", "56,90*", "45*")

I need to remove "*" which is unwanted.

I want to write function so whenever such data cleaning requirement I can use function and pass certain parameters.


texts: What does an image pose (camera to world) mean?


texts: What is the best way to find minima in Logistic regression?

texts:
In the Andrew NG's tutorial on Machine Learning, he takes the first derivative of the error function then takes small steps in direction of the derivative to find minima. (Gradient Descent basically)

In the elements of statistical leaning book, it found the first derivative of the error function, equated it to zero then used numerical methods to find out the root (In this case Newton Raphson)

In paper both the methods should yield similar results. But numerically are they different or one method is better than the other ?


texts: What is the advantage of using log softmax instead of softmax

texts: i am wondering if there are any advantages of log softmax over softmax. And also, when i should use softmax or log-softmax. is there any specific reason for choosing one over another?


texts: What is dimensionality reduction? What is the difference between feature selection and extraction?

texts:
From wikipedia:

dimensionality reduction or dimension reduction is the process of
reducing the number of random variables under consideration, and
can be divided into feature selection and feature extraction.

What is the difference between feature selection and feature extraction?
What is an example of dimensionality reduction in a Natural Language Processing task?


texts: Machine learning learn to work well on future data distribution?

texts:
This is based on my limited machine learning scope and experience, so correct me if I'm wrong. Many of the currently used machine learning models (SVMs, boosted trees, DNNs) work under the assumption that the training, validation and test data sets share the same distribution. They can work to some extent if the distributions differ but not by a lot. Here "can work" means that they work sub-optimally (i.e. can work better if the distributions are the same), not that their theory behind is supposed to deal w/ the distribution difference and can handle them like "nailing it".

Hence my question: is there work on predicting based on the assumption that the data sets are actually moving through a series of distribution changes? A crazy thought would be to observe the distribution difference between training and validation sets, and assume that the same diff will exist between validation and test sets and learn to predict well on test set. This will work great on time series where the nature of the data might change over time.


texts: Has anyone been able to run Tensorflow with GTX 1070 GPU on Ubuntu 16.04/15.10/15.04?

texts:
I've been tearing my hair out for the last 3 days and I just can't get it to work.

I recently got a GTX 1070 and I went to a fresh install with:

Ubuntu 16.04/15.10/15.04
PPA drivers nvidia-367
Cuda 7.5

Somehow every step of the way I couldn't get CUDA to detect my GPU despite following various instructions:

https://askubuntu.com/questions/693145/installing-cuda-7-5-toolkit-on-ubuntu-15-10

https://www.pugetsystems.com/labs/hpc/GTX-1080-CUDA-performance-on-Linux-Ubuntu-16-04-preliminary-results-nbody-and-NAMD-803/

Has anyone been able to get 1070/1080 running CUDA and tensorflow correctly and can guide me thru this painful process?

(I can't do Ubuntu 14.04 as it doesn't register my wifi card, which then I can't do anything)


texts: Building a search tool and classifying text using NLP and ML

texts:
Im a newbie in information Retrieval. Currently Im reading a book entitled "An Introduction to Information Retreival" by Christopher D. Manning and Prabhakar Raghavan. Im trying to implement an ai based search tool to search for some relevant information from a private dataset. ( say a chemical or mathematical dataset that invloves more number of numericals and unstructed messy representation of unit of measure)

Following is a row in my dataset in JSON format where keys are the coloumn name and value are the values corresponding to it.

  "ABC Project": {
    "In/Out diameter": " Both in and out are 1”  ",

    "Design Pressure (barG)": {
      "Max": "116 psiG (7.99 barG)",
      "Minimum": "79.7 psiG (5.49 barG)",
      "Design": "174 psiG (11.99 barG)"
    },

    "C02 %": "0.671",
    "MW": "16.68 kg/kmol",
    "TITLE": null,
    "Mothiram": "There is  very dense forest and their lived a king. The name of the king was Pandidhurai. He was very brave ",
    "Thooval delivery material": "- thooval delivery material is panam patta 316/316L  "

The sample search query that will fetch the above mentioned row from my dataset would be

" The project in which molecular wight is nearly 16 kg/kmol and Thooval delivery material panam patta 316l is used in which Mothiram is King Pandidhurai "

What I have done.

Preprocessing

Read each row from table(say $T$) as $R_{i}$ and search query as $Q$ do  the following preprocessing

convert_lower_case(data)
remove_punctuation(data)
remove_apostrophe(data)
remove_single_characters(data)
convert_numbers(data)
remove_stop_words(data)
stemming(data)
remove_punctuation(data)
Convert Word to Vector for coloumn $C_i$ and do it for each corresponding row in table $R_i$
Build Vector Space Model for coloumn $C_i$ and do it for each corresponding row in table $R_i$
Build a vector space model for query $Q$.
Compute cosine score to select Row having high score

Challenges I'm facing

When I do 'remove_punctuation(data)' Im loosing vital informations (say  $"$ denotes inches in the diameter coloumn)
A way to interpret unit of measure in the data
Unable to keep the relation between value ( say 16.78 kg/k mol  is stored in multi diamentions and unable to find it is related to MW).
I think it could be solved by building a ML classifier and train it to identify the entity MW. Say < 16.78 kg/k mol , MW > and using the unit of measure as a feature. But there are values like percentage composition of different chemical components, IN and OUT diameter , Inside and Outside Temperature. etc
Difficulty in interpreting scientific terms and mapping to one root form ( say MW, mol wt, molwt etc are all different ways of representing molecular weight)
Difficulty in finding the close proximity of neumerical values

Question

  Please suggest me a step by step approach to build a search tool using this dataset (Please also suggest me an apt algorithm that would
  be useful in each step)

I have read from here that some documents can be tagged as < document, class > and train a Navie Base Classifier Model for better search performance. Is it possible in this case , if so what could be class label you would be suggesting ? UPDATE : I think Navie Base cannot be used as the coloumn or key would  be greater than 1000. Im looking for a scalable approach to the problem.

  Is there any better approach to solve this problem than using a vector space model and  computing cosine similarity ?


texts: how to choose importance_type in lgbm.plot_importance

texts:
I have made a binary-classifier using lgbm.The classifier is made on unbalanced dataset.
I wanted to see the importance features of the model. There are two types of selecting importance_type -

importance_type (string, optional (default="split")) – How the
importance is calculated. If “split”, result contains numbers of times
the feature is used in a model. If “gain”, result contains total gains
of splits which use the feature.

both split and gain are giving me different feature sets.
My question is how to decide which is the better option between split and gain.
useful link - lightgbm.plot_importance


texts:
Is there any theory on the influence of skew in the data set on the performance of binary classifiers? At work, we are doing abuse detection, the negative population is regular logins, and the positive population is attack logins (account take over = ATO).

However, the frequency of ATO logins is 1/50,000 or less. So, we have a very skewed natural data set. Should I "unskew" my training data set by downsampling the legit logins? How much can I do that and still keep a model that will work well on the actual data? Any theory behind that?


texts: How to compute AUC in gridsearchSV (multiclass problem)

texts:
I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the most used metrics for this kind of problem.

I know how to use other metrics of scoring like accuracy etc. but the default ROC_AUC only works for binary class. Is there a method to use AUC in gridsearchCV for multiclass problems?


texts: How to customise cost function in Scikit learn's model？

texts: For example, when I have a problem that false negative should be penalised more, how can I incorporate that requirement in the algorithm such as SVM?


texts:
So I'm trying to create a simple bar chart of Survive vs Not Survive for the common Titanic data set in R. I keep getting just the number of No's and Yes's, and not the frequencies or counts associated with each no and yes. This is obviously not what is wanted. I am trying to just practice with ggplot2 and make some graphs. What am I doing wrong here?
The Bad Barchart:

#install.packages("tidyverse")
#install.packages("titanic")

library(tidyverse)
library(titanic)

view(Titanic)

titanic <- as.data.frame(Titanic)

titanic$Survived <- as.factor(titanic$Survived)

ggplot(titanic, aes(Survived, fill = Survived))+
  geom_bar()+
  ggtitle("Barplot to represent Passenger Count who Survived vs who Died")


texts: Multiple Linear Regression with k-fold Cross Validation

texts:
I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how well these models predict new observations by using 5-fold Cross Validation. From my understanding 5-fold Cross Validation shuffles then splits my data into 5 groups and chooses 1 for the testing set, and the other 4 for the training set. A given model is tested and the prediction error is recorded. This is repeated until all 5 groups are used as a testing set. Finally, the prediction errors are averaged.

My question is, when I am first determining the multiple regression equation (checking for assumptions, applying transformations, variable selection, etc.) which set of data should I use as my training set? Do I use the entire data set? Do I use one of the 5 training sets created by the 5-folds CV method? Do I repeatedly try to fit the regression model for all 5 training sets? If so, how would I extract each training set using the caret package?


texts: Loading own train data and labels in dataloader using pytorch?

texts:
I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader?

I have a dataset that I created and the training data has 20k samples and the labels are also separate. Lets say I want to load a dataset in the model, shuffle each time and use the batch size that I prefer. The Dataloader function does that. How can I combine and put them in the function so that I can train it in the model in pytorch?


texts: Python memory error: compute inverse of large sized matrix

texts: I have a big matrix of size 200000 x 200000. I need to compute its inverse. But it gives out of memory error on using numpy.linalg.inv. Is there any way to compute the inverse of a large sized matrix.


texts: assigning weight for attributes in dataset

texts: How can I assign weight for attributes in dataset before training model and which best method to achieve that in python?


texts: SGDClassifier: Online Learning/partial_fit with a previously unknown label

texts:
My training set contains about 50k entries with which I do an initial learning. On a weekly basis, ~ 5k entries are added; but the same amount "disappears" (as it is user data which has to be deleted after some time).

Therefore I use online learning because I do not have access to the full dataset at a later time. Currently I'm using an SGDClassifier which works, but my big problem: new categories are appearing and now I can't use my model any more as they were not in the initial fit.

Is there a way with SGDClassifier or some other model? Deep learning?

It doesn't matter if I have to start from scratch NOW (i.e. use something other than SGDClassifier), but I need something which enables online learning with new labels.


texts:
I am new to machine learning and have finished Andrew Ng's course on Coursera. I have just begun to tackle my first "real" ML problem - which is a binary classification problem. I was wondering if the steps I have taken is the "usual" way to work through a problem.

I have gathered my data which is in this case is some images (positive and negative images)

Preprocess the data - in my case read the images and convert these to grayscale images in order to simplify the data. I have then set up my arrays in terms of input and labels. No feature scaling is needed in my opinion or anyother preprocessing.

After this I split the data in training and test set (80-20 ratio). I used stratified split.

Model selection and evaluation. In this case I used three models: logistic reg., SGDClassifier and a decision tree.

Train the different models and evaluate them using k-fold cross validation sets. I evaluated the mean accuracy on the CV set and also the F1-score on the CV set. Based on this I selected the best modell (the scores for all the modells on the training set was 1 and on the CV sets 0.91 (log.reg) , 0,57 (dec.tree) and 0.73 (SGD))

NOTE : Shouldn't I plot some learning curves too see if there is a overfitt or underfitt problem (based on the result from the CV sets it seems like log.reg model is a good fit). Is there anything else to study or is it enough based on the scores?

Fine tune the selected model.

Evaluate on the test set.


texts:
I have generated a dataset of pairwise distances as follows:

id_1 id_2 dist_12
id_2 id_3 dist_23

I want to cluster this data so as to identify the pattern. I have been looking at Spectral clustering and DBSCAN, but I haven't been able to come to a conclusion and have been ambiguous on how to make use of the existing implementations of these algorithms. I have been looking at Python and Java implementations so far.

Could anyone point me to a tutorial or demo on how to make use of these clustering algorithms to handle the situation in hand?


texts: Can a decision tree learn to solve a xOR problem?

texts:
I have read online that decision trees can solve xOR type problems, as shown in images (xOR problem: 1) and (Possible solution as decision tree: 2).

My question is how can a decision tree learn to solve this problem in this scenario. I just don't see a way for any metric (Information gain, Gini Score, ...) to choose one of the splits in image 2 over any other random split.

Is it possible to solve the presented problem with a decision tree?
Would using a random forest solve the problem in any way?

Thank you in advance.


texts: building a 2-layer LSTM for time series prediction using tensorflow

texts:
From Tensorflow tutorials i am experimenting time series with LSTM

In the section 'multi-step prediction' using LSTM tutorial says

  Since the task here is a bit more complicated than the previous task, the model now consists of two LSTM layers. Finally, since 72 predictions are made, the dense layer outputs 72 predictions.

where previous task was prediction over a single point.

How do we know how many layers a problem requires (here, 2) ?

Then, from implementation point of view, using Python Tensorflow library,

multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))

why is there a need for the adding a Dense(72) layer ? what is the function Dense() doing ? (reading the doc doesn't help really)


texts:
I'm trying to create a model for when a shipped product will arrive at its destination. There are several stages the delivery goes through, so it's not just drive time from point A to point B. My first model looks at the status of the product on the first stage and uses average time deliveries for that product time to predict the # of minutes after the first stage that the product will be delivered.
I want to make another model that gives more of a continuous prediction, taking into account how long the delivery has been in a certain stage. For example, if most deliveries are done with the first stage after 15 minutes and it's been 10 minutes, the model should account for that in the eta.
How would I approach this? I can feed in the # of minutes it's been in the stage as an input but it seems like I can come up with a huge amount of data for each example.
Sorry if this isn't a clear question.


texts: Finding an appropriate price using thousands of data points

texts:
If i have a lot of data points describing the price of a used car. How would I find the market value of the car (assuming that the price points in the data set are the only determinant used, and the basis of determination will be the frequency [higher the frequency the better] price data point occurence for that particular car).

A count of absolute value recurrences will not work, as I want to bucket numbers that are similar (less than 5% difference of total price) into one category.


texts: Should we use one hot encoder class in data having 2 as maximum numeric representation of categorical feature in each column?

texts:
I am testing the Play Golf data set using Decision tree classifier:

I am splitting the data into Outlook, Temp, Humidity and windy as features, and Play Golf as target feature.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_excel('data.xlsx')
X = dataset.iloc[:, 0:4]
y = dataset.iloc[:, 4]

To test the DecisionTreeClassifier I need to transform categorical data into numeric:

from sklearn.preprocessing import LabelEncoder
lb = LabelEncoder()

X['Outlook'] = lb.fit_transform(X['Outlook'])
X['Temp'] = lb.fit_transform(X['Temp'])
X['Humidity'] = lb.fit_transform(X['Humidity'])
X['Windy'] = lb.fit_transform(X['Windy'])
y = lb.fit_transform(y)

The result is:

ix  Otk T   H   W
0   1   1   0   0
1   1   1   0   1
2   0   1   0   0
3   2   2   0   0
4   2   0   1   0
5   2   0   1   1
6   0   0   1   1
7   1   2   0   0
8   1   0   1   0
9   2   2   1   0
10  1   2   1   1
11  0   2   0   1
12  0   1   1   0
13  2   2   0   1

Should I use OneHotEncoder() class afterwards? Or 2 isn't that big to do so?


texts: Is there a way to overlay stem plot over line plot in python?

texts:
Date        Day_Perc_Change No. of Trades
2017-05-15  0.000000        40593
2017-05-16  -0.019184       22027
2017-05-17  -0.008550       15709
2017-05-18  -0.017699       47159
2017-05-19  -0.021748       85921

I visualized "Day_Perc_Change" in stem plot and "No. of Trades" in line plot. But now i want to compare these two plots by overlaying them on each other.. How do i visualize that?

x = df["Date"]
y = df["Day_Perc_Change"]

plt.stem(x, y, bottom=0)
plt.show()

df.plot(x="Date", y="No. of Trades", figsize=(10,8))

i Want to overlay these two on each other to compare them..is there a way to do that?

Part 2: post trying the answer mentioned below:
After trying the below code by @ElBurro
i get the following output

The expected output should look like this


texts: How to transform predicted results when doing cross-validation in sklearn?

texts:
I want to do cross-validation in sklearn like below, but the predicted result of X still need to be transformed to reduce the distance from y. How to do that with adding a custom function?

model = XGBRegressor(n_estimator = 500,
                     learn_rate = 0.05,
                     random_state = 0)

pipeline = Pipeline( steps = [('preprocessor', preprocessor),
                            ('model', model)
                            ])

scores = -1 * cross_val_score(pipeline, X, y,
                            cv = 3,
                            scoring = 'neg_mean_absolute_error',
                            verbose = 0)

There are only 0 and 1 in y, so I want to round and cut decimals off in the result of X, just after the step for predicting of X.

I maybe need other operations on y in the future. For example, fitting the correct result only includes the numbers like 0.5, 1.5, 2.5 ...

Example:

X - The input

ID Column_1 Column_2 Column_3
0    'A'       10     True
1    'A'       20     False
2    'B'       30     True

y - The correct result

ID Result
0  1
1  0
2  1

The current output

ID Result
0   0.899
1  -0.001
2   1.102

The expected output

ID Result
0    1
1    0
2    1

I have posted this question on Stack Overflow but also get no answers.


texts: How to change a Pandas dataframe into feature vector?

texts:
I have a Pandas dataframe with 10 columns, 9 of which are features to be used to predict the 10th column.

How is it ossible to convert this Pandas dataframe into X and y vectors to use in a linear regression problem?


texts: Why is the logistic regression decision boundary linear in X?

texts:
The logistic regression model,

\begin{equation}
\operatorname{p}(X) = \frac{\operatorname{e}^{\beta_0 + \beta_1 X}}{1 + \operatorname{e}^{\beta_0 + \beta_1 X}}
\end{equation}

is said to create a decision boundary linear in $X$.

As far as I understand, only the logit is linear in $X$. Is this the reason the decision boundary is linear in X? If so, why? And if this is not the case, what is the reason for this phenomenon?

I am confused about this because the decision boundary can be expressed as:
\begin{equation}
\operatorname{p}(X) = a, a \in [0,\, 1]
\end{equation}
And $\operatorname{p}(X)$ is not linear in $X$.


texts: Generate 3D "matrix" with Pandas, based on comparing two dataframes [Python]

texts:
Good morning everyone. I am working with Python and Pandas.
I have two DataFrames, of the following type:
df_C = pd.DataFrame(data=[[-3,-1,-1], [5,3,3], [3,3,1], [-1,-1,-3], [-3,-1,-1], [2,3,1], [1,1,1]], columns=['C1','C2','C3'])

   C1  C2  C3
0  -3  -1  -1
1   5   3   3
2   3   3   1
3  -1  -1  -3
4  -3  -1  -1
5   2   3   1
6   1   1   1

df_F = pd.DataFrame(data=[[-1,1,-1,-1,-1],[1,1,1,1,1],[1,1,1,-1,1],[1,-1,-1,-1,1],[-1,0,0,-1,-1],[1,1,1,-1,0],[1,1,-1,1,-1]], columns=['F1','F2','F3','F4','F5'])

   F1  F2  F3  F4  F5
0  -1   1  -1  -1  -1
1   1   1   1   1   1
2   1   1   1  -1   1
3   1  -1  -1  -1   1
4  -1   0   0  -1  -1
5   1   1   1  -1   0
6   1   1  -1   1  -1

I would like to be able to "cross" these two DataFrames, to generate or one in 3D, as follows:

The new data that is generated must compare the values of the df_F with the values of the df_C, taking into account the following:

If both values are positive, generate 1
If both values are negative, generate 1
If one value is positive and the other negative, it generates 0
If any of the values is zero, it generates None (NaN)

True table
Comparison of the data df_C vs df_F
df_C vs df_F = 3D
  +       +     1
  +       -     0
  +       0     None
  -       +     0
  -       -     1
  -       0     None
  0       +     None
  0       -     None
  0       0     None

You, who are experts in programming, could you please guide me, as I generate this matrix, I compare the values. I wish to do it with Pandas. I have done it with loops (for) and conditions (if), but it is visually unpleasant and I think that with Pandas it is more efficient and elegant.
Thank you.


texts: Outlier detection for unbalanced classes

texts:
I have to make a predictive model for predicting a boolean Won/Lost variable based on some other numeric data; and further find out the features of observations that have 'Won'.

However, the number of 'Won's' in my dataset is 0.05%. I've tried both oversampling and downsampling, but it hasn't worked. Even if I take an equal amount of 'Won's and 'Lost's', the model is not accurate for the rest of the 'Lost' values. I've also tried out weights, but it's not working well. Ideally I think I have to put a very high weight for 'Won'.

PS: Using RandomForestClassifier, with a confusion matrix to verify.

I'm not keen on trying out SMOTE, as I've heard it's tough in Python.

So now I'm trying to look at it in a different way, and do anomaly detection for the 'Won' case, as it natural for the data to have so few 'Won' cases.  So, two questions

Is this a correct approach?
How to go about it using Python?


texts: LIME is observing categorical features even though I am not passing any categorical features

texts:
Here is the code:

predict_fn = lambda x: xgb_model.predict_proba(x).astype(float)

feature_names = X_train.columns

for i in range(x_val.shape[0]):

    # Get the explanation for Logistic Regression
    val_point = x_val.values[i]
    print(val_point)
    print(val_point.shape)

    explainer = lime.lime_tabular.LimeTabularExplainer(training_data = Xs_train_array,
                                                      feature_names = feature_names,
                                                      training_labels = y_train,
                                                      mode = 'classification',
                                                      kernel_width=5)

    exp = explainer.explain_instance(val_point, predict_fn, num_features=10)
    exp.as_pyplot_figure()
    plt.tight_layout()

Few Notes:

Xs_train_array is of size (103,) and is type float.
There are no categorical variables.

Here is the error message I'm receiving:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-24-ee5cbeeb11ea> in <module>
     19     #print("Validation Prediction Probabilities: {}".format(xgb_model.predict_proba(val_point)))
     20
---> 21     exp = explainer.explain_instance(val_point, predict_fn, num_features=10)
     22     exp.as_pyplot_figure()
     23     plt.tight_layout()

~/opt/anaconda3/lib/python3.7/site-packages/lime/lime_tabular.py in explain_instance(self, data_row, predict_fn, labels, top_labels, num_features, num_samples, distance_metric, model_regressor)
    335             # Preventative code: if sparse, convert to csr format if not in csr format already
    336             data_row = data_row.tocsr()
--> 337         data, inverse = self.__data_inverse(data_row, num_samples)
    338         if sp.sparse.issparse(data):
    339             # Note in sparse case we don't subtract mean since data would become dense

~/opt/anaconda3/lib/python3.7/site-packages/lime/lime_tabular.py in __data_inverse(self, data_row, num_samples)
    534         inverse = data.copy()
    535         for column in categorical_features:
--> 536             values = self.feature_values[column]
    537             freqs = self.feature_frequencies[column]
    538             inverse_column = self.random_state.choice(values, size=num_samples,

KeyError: 87

This confuses me for a couple reasons:

categorical_features was not passed by me so I don't know where it is getting 87 from

Any help would be much appreciated. I used the same exact code on another dataset and am not running into any errors. I can't quite figure out what is causing this.


texts: Appearance-based hashing for similarity detection for picking the 100 most distinct images out 500 images

texts:
I would like to perform appearance-based hashing for similarity detection. I have 500 photos for each of my categories but I only want to maintain the 100 of them that are most distinct. How should I go about this? Are there already well-known baselines for this? I have a preference for Python and PyTorch or bash.
Also, what are the other known methods for this task? I was thinking I could also run the resnet50 on images, extract a 2048 dimension vector and do a cosine similarity but I wonder if there are deep neural network that can do the task I have in mind end-to-end or be more efficient than what I have in mind.


texts: Are Intersection over Union (IoU) scores preferring larger objects?

texts:
According to the fcn model I implemented and the PASCAL VOC benchmark (here)
I find that objects with larger sizes in an image receive better IoU or AP score in the test set. Why do the IoU scores have a bias towards larger scale objects? Can anyone give an explanation why this could happen？


texts: identifying input sequence using neural network

texts:
I fount simple neural scenario sayinh : network having 3 layers, 3 inputs and 2 outputs. It should be trained to recognize a simple pattern - if inputs are correspondingly 6.0,7.0 and 8.0 then outputs should be 3.0 and 4.0, otherwise outputs should be 23.0

What I understand is, hidden layer should Check input pattern in sequence with simple if else condition.

if in1 == 6 and in2 == 7 and in3 ==8:
    out1, out2 = 3, 4
else
    out = 5

Do I understand the problem correctly or something I am missing?


texts: Why might my validation loss flatten out while my training loss continues to decrease?

texts:
In my effort to learn a bit more about data science I scraped some labeled data from the web and am trying to classify examples into one of three classes. I am running into a problem that, regardless of what model I try, my validation loss flattens out while my training loss continues to decrease (see plot below).

A few potentially relevant notes:

I have about 15,000(3,000) training(validation) examples.
Labels are roughly evenly distributed and stratified for training and validation sets (class 1: 35%, class 2: 34% class 3: 31%).
I have 73 features that consist of: 10 numerical features, 8 categorical features that translate to 43 one-hot encoded features, and a 20-dimensional text embedding
I get similar results if I apply PCA to these 73 features (keeping 99% of the variance brings the number of features down to 22).
The plot shown here is using XGBoost.XGBClassifier using the metric 'mlogloss', with the following parameters after a RandomizedSearchCV: 'alpha': 7.13, 'lambda': 5.46, 'learning_rate': 0.11, 'max_depth': 7, 'n_estimators': 221
I get similar results using a basic Neural Network of Dense and Dropout layers.
Best model I've achieved only gets ~66% accuracy on my validation set when classifying examples (and 99% on my training examples).
Admittedly my text embedding might not be fantastic (using gensim's fasttext), but they are also the most important feature when I use Xxgboost's plot_importance function. Though, I was facing a similar problem even before I added the text embedding.


texts: Q learning - how to use experience replay, when playing against other agent?

texts:
I am currently trying to create a tic tac toe q learning neural network to introduce me to reinforcement learning, however it didn't work so I decided to try a simpler project requiring a network to train against static data rather than another neural network.
This lead to me following the guidelines from this website - http://outlace.com/rlpart3.html

however after programming this, the simple version works half the time, this is the version without experience replay. on some runs of the program the game will be played correctly, others it just moves back and forth wehn doing test runs.
When trying to implement experience replay to complete the harder version. the program will just constantly get itself into a loop of going back and forth when testing

i have a limit of 100 batches in which a batch is what the neural network is trained on. I am wondering whether this is an appropriate amount, or if there could be any common problems with implementing experience replay that i may have made.

My current perspective of what experience replay is:
1. run the program
2. after each turn, the data you used to train the network on, gets saved into a batch
3. when you have reaches x(100) amount of batches, pick one out and train on it.
4. overwrite the oldest batch with the new batches that come in.

If anyone could let me know where I have gone wrong, or if there is any feedback about the experience replay or the quality of the question please let me know and i would be very grateful.

EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself,m or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.


texts:
For below line of code

model.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same',
                 activation ='relu', input_shape = (28,28,1)))

Here, what does 'filers' and 'kernel_size' mean? or what is filter and kernel_size ?


texts: How can I group by elements in a column in pandas?

texts:
I have a dataset that looks like this:

I need to transform it so it looks like this:

Meaning I need to show the balance by balance groups and time gaps.
Now the time gaps are elements of a column. I tried to use the following;
b = a.pivot_table(values='GAP', index=a.index, columns='BALANCE_GROUP', aggfunc='first')

With different parameters but it outputs something really weird. I also tried to use the groupby function which works fine, but it won't allow me to set the elements in the gap column as seperate columns...
b = a.groupby(['BALANCE_GROUP','GAP']).sum()

and I got something like this:


texts: Chi-square as evaluation metrics for nonlinear machine learning regression models

texts:
I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are continuous variables. So an evaluation box plot looks like this:

I experiment with both linear (linear regression, linear SVMs) and nonlinear models (SVMs with RBF, Random forest, Gradient boosting machines ). The models are trained using cross-validation (~1600 samples), and 25% of dataset is used for testing (~540 samples). I am using R-squared and Root Mean Square Error (RSME) to evaluate the models on test samples.  I am interested in finding an evaluation measure to compare linear models to nonlinear.

This is done for a scientific research. I was pointed out that R-square might not be appropriate measure for nonlinear models, and that Chi-Square test would be a better measure for goodness of fit.

The problem is, I am not sure what is the best way to do it. When I browse Chi-square as goodness of fit, I only get examples where Chi-square test is used to see whether some categorical samples fit a theoretical expectation, such as here. So here are my considerations/questions:

One way I could think of is to categorize predicted (continuous) values into bins, and compare predicted distribution to the ground truth distribution using Chi-Square test. But that doesn't make much sense, i.e. we have a machine learning model that perfectly predicts ground truth values 2,3, and 4, and values 5 predicts as 1, and values 1 as 5 - Chi-Square test that I propose here would reject null hypothesis, although the model is mispredicting 2 out of 5 values.
As referred in a tutorial from USC I could use formula (1) to compute Chi-Square value, where experimentally measured quantities (xi) are my ground truth values, and hypothesized values (mui) are my predicted values. My question is, what is the variance? If we observe each value 1,2,3,4, and 5 as a distinct category, than the variance of ground truth within each category is equals to zero. Also, how one computes degree of freedom (N-r)?
Related to the statement I am interested in finding an evaluation measure to compare linear models to nonlinear is Chi-Square test the best (or even good) choice? What I've seen so far in machine learning competitions for regression tasks, either MSE or RSME are used for evaluation.


texts: How can I use multiple features in basic sentiment analysis in scikit-learn?

texts:
I've tried to reduce the problem to it's absolute basics. Assume I have data (csv) as such:
label,text-column,gender-column,day-column
1,"Sample positive text", female, 1
0,"Sample negative text", female, 3
1,"Another positive comment", male, 2
0,"Angry text sample", male, 7

And I have this code that trains on label by using BoW (in this case tf-idf) on the text-column. I do a 70/30 train test split and all works well.
vec = TfidfVectorizer()
clf = MultinomialNB()
training_data = pd.read_csv('trainset.csv', delimiter=',')
text_tfidf = vec.fit_transform(training_data['text-column'])
# gen_tfidf = vec.fit_transform(training_data['gender-column'])
X_train, X_test, y_train, y_test = train_test_split(text_tfidf, training_data['label'], test_size = 0.3)
clf.fit(X_train, y_train)

However, for the life of me I simply cannot figure out how to use more than one feature. E.g. I want to use, say, both text-column and gender-column to train the model and see how that impacts accuracy, but I don't understand how to do that!
Am I missing something conceptually important here? Thank you.


texts: Pandas: How can I update dataframe values?

texts:
I have two spreadsheets where one is updating the other.

How can I update this data using the pandas library?

Example, where 'b' updates 'a':

a = {'field': ['a', 'b', 'c'], 'value': ["", None, 1]}
b = {'field': ['a', 'b', 'd'], 'value': [1, 2, 1]}

Expected outcome:

c = {'field': ['a', 'b', 'c', 'd'], 'value': [1, 2, 1, 1]}


texts: What should I master better for professional data science in economics and finance?

texts:
First, excuse me for the noob and long question which is probably doesn’t even belong to here, I know there are several question been answered like this out there, but I think this is going to be up-to-date. Stack Overflow deleted my question and redirected me to here.

I study economics and finance on undergraduate level, and to be honest, I am not really into programming so far. However, I must admit it you can't doing really well nowadays without specific softwares and programming languages on economics/finance related fields.

According to my curriculum, I’ve encountered Matlab, some econometrics softwares, and of course MS Office, especially Excel with VBA. I have some shady framework in my mind, and please feel free to correct me if I am wrong. So as I experienced, for numerical calculations and doing the vast majority of math, Matlab, Octave and Mathematica exists. For econometrics, there are professional softwares like eViews, STATA, SPSS or the open source Gretl and Tableau for data visualization. And last, we can use Excel to manage databases.

Long story short, my basic question would be that, are these above the best tools for doing the job ? Or should I switch to more professional tools – like real programming languages - to being better in solving mathematical problems, numerical calculations, econometrics, data science and exquisite, high-quality data visualization? What are the most desirable skills in the data science industry nowadays in economic/financial areas?

I heard that R is a quite trending statistical programming language in these days, and getting better and better each day - I already wrote some functions and visualizations in Rstudio. I also heard that SQL is also a better option to manage really massive data sets instead of Excel, but is SQL able to do every kind of stuff with data what can be done in Excel ? It seems to me Python is generally the number one language for data analysis, it’s flexible and usable on a broad scale. I find Python libraries - such as matplotlib, numpy, pandas, bokeh - extremely attractive. What about Julia , is this going to be the next R in the future ? To be honest, I am also still confused a little bit by such terms like data science, data analysis, data mining, machine learning, big data – are there any serious difference between these phrases?

From above, which one is that I should really focus on and master it ? Keep practicing on popular softwares, or switch to R , Python, Julia, SQL ? Maybe both of them? Again, we are talking about only graduate and undergraduate level of economics and finance, and related jobs. I don’t want to develop serious and complex softwares/applications, just quantitatively analyze stock prices, corporate and economic data, like annual reports, employments, GDP and so on.

Experienced data analysts, please guide me through the confusing forest of data analysis tools. I appreciate every kind of comment.


texts: Time horizon T in policy gradients (actor-critic)

texts:
I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.

At the bottom of that slide, the gradient of the expected sum of rewards function is given by
$$
\nabla J(\theta) = \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \nabla_\theta \log{\pi_\theta(a_{i,t} \vert s_{i,t}) (Q(s_{i,t},a_{i,t}) - V(s_{i,t}))}
$$
The q-value function is defined as
$$Q(s_t,a_t) = \sum_{t'=t}^T \mathbb{E}_{\pi_\theta}[r(s_{t'},a_{t'})\vert s_t,a_t]$$
At first glance, this makes sense, because I compare the value of taking the chosen action $a_{i,t}$ to the average value in time step $t$ and can evaluate how good my action was.

My question is: a specific state $s_{spec}$ can occur in any timestep, for example, $s_1 = s_{spec} = s_{10}$. But isn't there a difference in value depending on whether I hit $s_{spec}$ at timestep 1 or 10 when $T$ is fixed? Does this mean that for every state there is a different q value for each possible $t \in \{0,\ldots,T\}$? I somehow doubt that this is the case, but I don't quite understand how the time horizon $T$ fits in.

Or is $T$ not fixed (perhaps it's defined as the time step in which the trajectory ends in a terminal state - but that'd mean that during trajectory sampling, each simulation would take a different number of timesteps)?


texts: Find the order of importance of random variables in their ability to explain a variance of Y

texts:
Problem statement:
Given a set of random variables Xs, give the order of importance of their ability to explain variance in target variable Y. Assumes all Xs are normalised to have 0 mean and std dev of 1.

Can treat this as regression problem(normal or ordinal)

Solution for 2 cases:
Case 1: When the calculated covariance matrix is diagonal, I know that random variables Xs are independent wrt each other. Then I can fit a linear regression model and order Xs by their weights.

Case 2: Or If I know that Xs are independent(https://en.wikipedia.org/wiki/Independence_(probability_theory)
even though their covariance in non zero, I can fit linear regression model and order Xs by weights.

How do I solve this problem when I know that random variable Xs are not independent and have non zero covariance?

Also how can I use a non linear model such as random forest to get variable importance? scikit provides feature_importances_ api for random forest.... Is there an gotcha with this api such as results are not meaningful when variables are dependent or covariant?

Found this question elsewhere: https://stats.stackexchange.com/questions/202221/for-linear-classifiers-do-larger-coefficients-imply-more-important-features/202846


texts: Are real analysis and measure theory essential to learn for data science?

texts: Some people say data scientists don't necessarily need to know real analysis and measure theory, but for others, real analysis and measure theory are very important for the undersdanding of kernel methods, stochastic processes etc. Should I learn real analysis and measure theory? I don't want to be too academic


texts: How to encode ordinal data before applying linear regression in STATA?

texts:
I have a data set that has student performance marks (continuous and dependent variable), Teacher Qualification (Ordinal and independent variable containing categories: Masters, Bachelors, High School). I want to apply the regression analysis to check the impact of teacher qualification on student's marks.
How can I encode ordinal data before applying linear regression?


texts: Pytorch Distributed Computing - Recomendations/Resources/Courses?

texts:
I would like to get into some distributed computing for processing Pytorch CNN models. I am completely fresh in this field and want to get some recommendations as to where I should start researching and learning techniques in distributed computing specifically for Deep Learning.

My motivation is that I have access to a lot of personal Windows 10 Desktops with great hardware, a few Ubuntu Linux machines of my own and then my personal desktop that is rigged with great hardware for a heavy workflow. Previously I've solely run separate scripts manually through other machines via RDP or SSH and let them compute for a few hours to a few days. But now I'm at the point it'd be a great idea to combine all these resources so when I have a large dataset to generate or train a Pytorch model on I can use 2-10x as many resources to cumulatively compute a script. (One heavy task I would have in mind for the future is something like Neural Architecture Search NAS, which seems simpler to parallelized between machines between each generation.)

My background is very Python and ML/DL focused through PyTorch. Then I also have a hobbyist level experience in networking, Linux VMs as well as dedicated servers, and computer hardware.

Other Ideas: Not familiar with, but not sure if any other buzzwords like grid computing, setting up a computer cluster, or etc would fit the scenario.

This is fresh for me but a long term interest. Point me in the right direction! Better forums, courses, articles, books, etc!

Edit 1: This could be a resource/recommendation for something PyTorch specific such as their distributed computing tutorial pages OR be something generalized for any python script distributed computing. Ex: Compute xx terabytes of spectrograms across 10 machines with a single call of a script.


texts: Can ROC/AUC help model training or just be used for model chosing?

texts: if can be used to help for training?how?


texts: Reinforcement learning: Discounting rewards in the REINFORCE algorithm

texts:
I am looking into the REINFORCE algorithm for reinforcement learning. I am having trouble understanding how rewards should be computed.

The algorithm from Sutton & Barto:

What does G, 'return from step t' mean here?

Return from step t to step T-1, i.e. R_t + R_(t+1) + ... + R_(T-1)?
Return from step 0 to step t?, i.e. R_0 + R_1 + ... + R_(t)?


texts:
As a total beginner I am trying to apply some "predictions" on top of a bunch of csv files which contains house transactions for the last 20 years divided per area. What I would like to predict is the trend of the transactions for lets say the next year for a specific area.

What general steps would you follow, to analyse those data and then predict?

I read different articles but what I am looking for it is a sort of "best general practice" for this sort of problem.


texts: Why is learning rate causing my neural network's weights to skyrocket?

texts:
I am using tensorflow to write simple neural networks for a bit of research and I have had many problems with 'nan' weights while training. I tried many different solutions like changing the optimizer, changing the loss, the data size, etc. but with no avail. Finally, I noticed that a change in the learning rate made an unbelievable difference in my weights.

Using a learning rate of .001 (which I thought was pretty conservative), the minimize function would actually exponentially raise the loss. After one epoch the loss could jump from a number in the thousands to a trillion and then to infinity ('nan'). When I lowered the learning rate to .0001, everything worked fine.

1) Why does a single order of magnitude have such an effect?

2) Why does the minimize function literally perform the opposite of its function and maximize the loss? It seems to me that that shouldn't occur, no matter the learning rate.


texts: Generating text using NLP based on parameters

texts:
I want to generate some text based on the value of certain parameters. For instance, let's say I want to generate descriptions of video games. So, besides real descriptions as training data, I would like that the model takes in account the following parameters (for example) about the game:

Violent: yes
Multiplatform: yes
Drugs: no

So that if the game has drugs content, the output text has some phrase referring to it.

Is this possible? If so, how could I do it in Python? I was going to use LSTM neural networks in Tensorflow.


texts: How google get the 'use over time for' data from the before internet years?

texts:
Recently I am reading a book and I found there are lots of words I rarely saw. While I was searching their meaning, I found that for a single vocabulary google has a use over time for: statistic -- from 1800 to 2010.

I may imagine how google could gather the frequency that a vocabulary to be used in Internet. But what about before the Internet? How they know the frequency of the people in 1800 using a vocabulary? Does they first convert the chartaceous material into digital data then computing for the result or something else?


texts: If a single element is missing more than 50% of its feature values, should you just remove it?

texts: I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 additional records that contain some missing data. Certainly, if a record is missing 90% of all of its feature values, it's best just to remove it - mean imputation would just be adding average records for all features to the training matrix and it can overfit. If a record is only missing 2% of its feature values, then imputation would be great and you can keep it. What do you believe is the correct threshold for deciding between keeping and tossing?


texts: Hyperparameter tunning for Random Forest- choose the best max depth

texts:
I'm trying to choose the best parameters for random forest model.
For that goal I hae run my model in loop with only one parameter and each time I have changed the number for the parameter max depth. I have created two charts: one for the model  score and one for the MAE.
this is how the charts look like:

My question is what is the right number to take for that parameter?
I fell likeI want to choose really low number when the training set and the test set are close to each other and high (the second point, when the accuracy is above 90) :

But I see that it gets stabalize much after and i'm afrid that it might lead to overfitting.
Is it true? should I choose bigger number from the stable zone or is ok to take low number as long as the training and test data are having the same accuracy and same error?