texts: Keras: How to normalize dataframe with continuous and categorical data?

texts:
I have a dataframe with about 50 columns. The columns are either categorical or continuous data. The continuous data can be between 0.000001-1.00000 or they can be between 500,000-5,000,000. The categorical data is usually a name, for example a store name.

How can I normalize this data so that I can feed it into a dense layer of a Sequential model?

The Y values are either 0 or 1, so it is a binary classification problem. I am currently normalizing all of the continuous data to be 0-1 and one-hot encoding all of the categorical data, so that if I have a column with 5 names it in, I will get a matrix with 5 columns filled with 0's and 1's. Then I join all of the continuous and categorical data and feed it into a Dense layer with init='uniform' and activation='relu'.

Is this the standard way of doing things?


texts: An average of averages problem, or two valid results?

texts:
From an event log I have derived the days between each user's logins. This leaves me with a dataframe like this:
I then took the average of column 'days_between_logins' which gave me a result of 15.9 days.
But then, if I calculate the average number of days between logins on an individual user basis and then average that across all users, the result is 34 days.
I think I have an average of averages problem? Then again, I think both results are valid.
I would really welcome any thoughts or supporting/opposing conclusions.


texts:
I have 5400 rows of data and 3211 columns of attributes.

The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity measures.

Huge dimensionality, I know, but I wanted to (as a first step) just see how this data clusters and finds similarity between all attributes.

I converted all attributes values to "0" if there was no value, and "1" if there was a value. I thought it'd be an easy first step to get me started with a clustering visual and similarity metric if I converted the values to binary.

Jaccard similarity seems to be a good measure for binary, but I'm stumped as to how to implement this (in Python) when I don't have any lists for comparison. Am I supposed to hard code each variable into the algorithm (3207 variables)?

I'm not sure where to start. Also, if there's a better way of doing this, I'm all ears. I've been researching how best to tackle this problem and there's so many similarity metrics that can be used, but I'm stuck on how to start since there are so many columns that need to be used.


texts:
I am new to deep learning as well as pytorch. I have got the following code (siamese network) working with pytorch:

https://github.com/KrisG04/Siamese-using-PyTorch/blob/master/Siamese-networks-medium.ipynb

Now I would like to replace the CNN with a pretrained network (trained on places dataset): https://github.com/CSAILVision/places365

How do I do this using pytorch?


texts: Selecting ML algorithm for music composition

texts:
I'm a composer and programmer. I'd like to use ML for composing music. There's already research on the general problem of composing by machine in known musical styles. I'm more interested in the specific problem of using a computer to assist myself in composing... to speed up certain repetitive tasks, or to delight me with unexpected ideas.

First, I want to design ML that can make a good/bad categorization on a single short passage. (Not looking to generate music at first.)

It's pretty easier to decompose such a passage into significant features. I can say things like

maybe there are particular "intervals" (harmony) present, and others absent
maybe notes are spread out evenly from low to high, or maybe not
maybe notes are clustered around a certain register rather than spread out
maybe the rhythm is particularly uneven
maybe a particular note occurs a lot

I can easily write algorithms that produce a list of such features for any given passage, and I can even be pretty sure that I'm including all of the relevant features. For an "art form," music is surprisingly amenable to analysis.

The trick is, first of all, there will be hundreds of observations, and second, that no one of these observations is good or bad in itself. Say we have three factors A, B, and C. A is not bad alone, but perhaps when A and B occur together, that's bad. But then maybe if A, B, and C all occur together, it becomes good again! And then maybe a fourth factor D, changes the context so that A, B, and C no longer matter much.

This seems like an ML problem, but I wonder what the best approach is?

I can construct or compose hundreds of examples of music labeled good and bad, and identify hundreds of "features" in each example. Some of the features are "present/not present" categories, while others are numbers that can be scaled from 0 to 1.

Is there a general and simple learning algorithm that even an ML novice such as myself (but I'm an experienced programmer) can implement?

I have to generate my own training examples and would be limited to around 300 to 1000, as I'm training it on simple passages in my style. This may actually be good problem to have... my job is not that complex compared to general musical intelligence.

I will contrast this task with image recognition. An image has millions of pixels and the features are spread out, occurring in endless  permutations. It makes my head hurt to try to figure out what's going on in a deep neural net.

However, in music, there will be at most a few hundred features. The patterns will not occur in endless permutations. In fact, in stripped-down situations my wild guess is that you could make an 80% accurate classifier by looking at 10 features (assuming you have the weights and combinations well-trained). I'm aiming somewhat higher than that (I need more features and a bit more accurate).

To give you some idea of my data set: I'm in the early stages of writing the software to extract the features, but I have a pretty good mental idea of what I'm aiming for (been composing by hand for years and years). To make this more concrete I created a web page showing what the data will look like. This is very simple and just totally made up.

http://theunexpectedpearl.com/dataset.html

This shows four rows. Each row is an example passage of music that I will compose or generate automatically. Each row is classified "good" or "bad" by my own ear. The rest of the row is a long list of features, each of which is either "yes/no" or a single number scaled from 0.0 to 1.0.

I don't think a Naive Bayes classifier will work for this (if I understand correctly) because features are neither good nor bad in and of themselves... it is only combinations of them (context) that matter. In fact, the very same feature can contribute to "goodness" in one context, and be an immediate strong indication of "bad" in another context.

The data could possibly be structured differently. In practice, this spreadsheet would be "sparse" and maybe features should be grouped in some way, or arranged in a tree.


texts: Do statistical test with orange between data

texts:
I'm learning orange and I'm looking to performe some statistical test on my data.
The test I usually do are z-test, t-test, chi-square test and Anova.
Can I use a widget to do these or should I do a python custom script ?

Thank in advance.


texts: can i make model that can take N shape input?

texts:
as far as i try all data that model take are fixed size or fixed shape, i had problem that i give data to model but it has no fixed shape and i can not cut part of it as all of it are important,so can i make model that have unshaped data as input then take unshaped data to predict ..
like have small problem that can be represented by N like famous problem travel sales man as in it N refer to number of cities


texts: Unbalanced training data for different classes

texts:
What precautions do I need to take while trying to develop a CNN for classification of images if there is much more training data for one label. For example:

label1 : 1000 images
label2 : 100 images
label3 : 100 images
label4 : 100 images

Numbers will become larger later but proportion is likely to stay the same.

Thanks for your insight.


texts: ZeroR as baseline for balanced / imbalanced datasets

texts:
I have a dataset containing thousands of text posts. I am building a binary classifier that will classify posts as safe (0) or  risky (1). I randomly picked some of them and manually labeled. Label 1 is minority.

Imbalanced data results in skewed performance. To overcome this, I undersample to obtain a 50%/50% distribution for both train and test set.

If I use stratified strategy (in scikit-learn) for ZeroR, it acts like random guesser. Is this a good baseline?
If I use "guess 1" all the time, then its recall is 100% all the time. However recall is the most important metric for me. So other algorithms' predictions look unsuccessful. Should I ignore recall when using ZeroR and only compare accuracy?
I also would like to apply 10-fold cross validation, should I apply undersampling before splitting as well?
Or instead should I go with F1-Score without undersampling? This time ZeroR has high F1 score due to 100% Recall.


texts: MNIST 2-layer NN failing to recognize certain digits

texts:
Recently I've been taking a look into some basic multilayer neural networks, and I decided to try tackling the MNIST database of handwritten digits using a 2 layer neural network with 100 neurons in the hidden layer.
My network has a hidden layer using ReLU, and the output layer uses the sigmoid function. The cost is calculated by MSE, and weights are updated using SGD. Below is the code written in MATLAB for training the network.
When training this network, I discovered that it has somehow failed to recognize every 8 and 9 in the data set. It has some problems recognizing other numbers, but not nearly as much as 8 or 9.
How can I fix this issue?
Edit: The results when training a network from a random set of weights are quite inconsistent and oftentimes have low accuracy. Is there something wrong with the backpropagation step or is the network structure not the best for digit recognition?
*dsigmoid and dReLU are the derivatives of their respective functions.
function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights1, weights2, inputs, outputs, alpha, shuffle)
% Trains a 2 layer perceptron using stochastic gradient descent and MSE
%
% function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights2, weights1, inputs, outputs, alpha, numHiddenLayers)
% Inputs:
%   weights1 = weights between input layer and hidden layer as a matrix
%              where row i and column j represents the weight between node
%              j of the input layer and node i of the hidden layer
%   weights2 = weights between hidden layer and output layer as a matrix
%              where row i and column j represents the weight between node j of the
%              input layer and node i of the hidden layer
%   inputs = inputs of the training data, where each set of inputs is
%            stored as a row vector
%   outputs = the correct outputs of the training data, where each set of
%             outputs is stored as a row vector
%   alpha = the learning rate, default is 0.005
%   shuffle = boolean value, determines whether the training cases will be
%             randomly ordered, default is false
%
% Outputs:
%   trainedWeights1 = the trained weight matrix for the weights between the
%                     input layer and hidden layer
%   trainedWeights2 = the trained weight matrix for the weights between the
%                     hidden layer and output layer

if ~exist('alpha','var') || isempty(alpha)
    alpha = 0.005;
end

if ~exist('shuffle','var') || isempty(shuffle)
    shuffle = false;
end

% Determining the size of the input data set
N = size(inputs,1);
fprintf('Data set size: %.0f\n', N);

% Shuffle input data
if shuffle
    fprintf('Shuffling data...\n')
    order = randperm(size(inputs, 1));
    inputs = inputs(order, :);
    outputs = outputs(order, :);
    fprintf('Shuffle complete\n')
end

% Train Model
fprintf('Starting training process...\n')
trainedWeights1 = weights1;
trainedWeights2 = weights2;

for dataSet = 1:N
    % Get inputs of current data set
    input = inputs(dataSet,:); % row vector

    % Calculate activations of hidden nodes and the output
    Z1 = trainedWeights1*input'; % column vector
    hiddenActivation = ReLU(Z1); % column vector
    Z2 = trainedWeights2*hiddenActivation; % column vector
    output = sigmoid(Z2)'; % row vector

    % Calculate the error/cost of the current weights
    correctOutput = outputs(dataSet,:); % row vector
    cost = sum((output - correctOutput).^2);

    % Update Weights
    trainedWeights2 = trainedWeights2 - alpha * 2*(output'-correctOutput') .* hiddenActivation' .* dsigmoid(Z2);
    S = sum(2*(output'-correctOutput') .* trainedWeights2 .* dsigmoid(Z2),1);
    trainedWeights1 = trainedWeights1 - alpha * S' .* input .* dReLU(Z1);

    % Display status
    if mod(dataSet, 1000) == 0
        fprintf('Data set %.0fk/%.0fk complete!\n',dataSet/1000,N/1000);
    end
end
end


texts: Is it possible to apply PCA on different subsets independently?

texts:
I need to apply PCA on a rather big set of data, but my machine is not able to handle the workload. So I was considering to split randomly my original set into 4 subsets, apply PCA independently on each subset and finally join the 4 subsets to have the original one with the PCA.

For my understanding PCA looks for correlated variables so they can be combined into one, which somehow will represent the values of the original variables. So I believe this operation happens on a row level. However, I guess the algorithm needs to analyse all the set as a whole to determine the correlation between features, since correlation among features row by row may differ, and some rows may even have NaN values.

So I would like to know if this approach with the subsets is correct, or if I may end up with one subset which after PAC combined features a and b and another subset which combined b and c.


texts: How to add jpg images information as a column in a data frame

texts:
I have jpg images stored in a folder. For ex: 11_lion_king.jpg,22_avengers.jpg etc.

I have a data frame as below:

data_movie.head()

movie_id  genre
11       ['action','comedy]
22       ['animation',comedy]
..........

I want to add a new column movie_image into the data_movie data frame with the jpg information mapped correctly with movie_id column as shown below:

movie_id  genre                movie_image
11       ['action','comedy]    11_lion_king.jpg
22       ['animation',comedy]  22_avengers.jpg
.........

Help will be appreciated.


texts: Is macro averaged the precision, recall and F1 score recommended for binary classification data? The calculation of precision is calculated for both class and averaging it makes it seems better. Or macro averaged is works better for multiclass classification only?


texts: Image and Video Formats with lossy-compression and fast subregion lookup?

texts:
I'm looking for image / video file formats that

support lossy compression to reduce disk size and
allow for fast subregion lookup. i.e. dont read the entire file when you only need to lookup a small space/time region.

The use case is for preprocessing large image and videos (i.e. we are keeping the raw data, which is why lossy compression is ok) and then being able to quickly lookup small regions for training ML algorithms.

So far I've found several candidates:

cloud-optimized-geotiff (CoG) format, which works very well for images. Unfortunately, to use it I need to use GDAL, which can be a pain to install, but it's not a huge deal. However, it lets me lookup arbitrary regions of large (20k x 20k) images very quickly, and I can either store the data with lossy jpeg compression or lossless compression. The main issue with the CoG format is that it doesn't support video.
numpy memmap - in terms of speed, this can't be beat. This can lookup arbitrary subregions of images or video in no time flat. However, the disk requirement is huge. I often see a 10x increase in disk usage with memmaps, which is not feasible when a single image/video data can be over 100MB. If mememaps had some sort of lossy block compression, then I'd probably be all set, but alas they do not.
hdf5 - this works for both image and video. Its quite a bit slower than memmaps (~100x), but it can load arbitrary subregions without loading the entire image. I've been using the h5py bindings for this, and unfortunately they don't seem to expose any lossy compression algorithms that provide any meaningful disk savings.

Generally, I'm wondering if there are any other formats or libraries out that that I'm unaware of.

Specifically, is there any similar alternative to the CoG format that doesn't require GDAL? Is there any analog of the CoG format for multi-frame video data?


texts: Can the spatial features(information of Input image) disappear in Fully Connected Layers?

texts: I also know the spatial features disappear at pooling layer. I just want to know the spatial features can disappear at FC layer (through the above procedure).


texts:
I have a feed-forward neural network with a customized cost function. Since my cost function has an exponential component,  I need to handle very large numbers. By default, tensorflow uses float32 datatype, which is not sufficient for my work. How can I have a tensorflow model with float64 datatype? I have tried to define all the tensors as float64 and hope the casting take care of the rest but it does not work.
Also, I have enabled the eager execution mode.


texts: Get the Polynomial Equation with Two Variables in Python

texts:
TL;DR
predict "price", given "length" and "wandRate"

I have some time-series data where the dependent variable is a polynomial result of 2 independent data points.
Here is a snippet:

This is past pricing data of Processed Rice Grains of a certain kind of rice.
Based on the variable "wandRate" (1st variable) which is the price for any "length" (2nd variable) over 8.2, prices of rice grains with lower lengths are calculated.
These prices are based on a long trial-and-error method of asking various experts on their "opinion" on how a certain grain of a certain length should be priced. There are other variables which can't be objectively measured, but length is the main indicator. I was wondering if it would be possible to create an objective model or find a polynomial equation in two variables to predict "price", given "length" and "wandRate"
I was led to thinking in terms of a polynomial when I plotted the data in google sheets and a sixth-degree-polynomial equation gave an intuitively correct looking trendline.

NOTE: I do not have a strong math background so simple google searches about "polynomial in 2 variables from data python equation" did not yield any implementable results.

I'm looking for some python code to accomplish this. ANY guidance on where to look would be appreciated.


texts:
Has anyone else encountered this error?  I am teaching a business analytics class with python this semester and Colab has been absolutely fantastic, yet one student is getting the following error.

Any thoughts or advice would be very helpful!


texts: Mass convert categorical columns in Pandas (not one-hot encoding)

texts:
I have pandas dataframe with tons of categorical columns, which I am planning to use in decision tree with scikit-learn. I need to convert them to numerical values (not one hot vectors). I can do it with LabelEncoder from scikit-learn. The problem is there are too many of them, and I do not want to convert them manually.

What would be an easy way to automate this process.


texts: Linear transformation from one sample to another

texts:
Generate a Sample $\underline{Z_1}$ $\underline{Z_2}\dots \underline{Z_{5000}}$ , while $\underline{Z_i} \sim N_2[(0,0)^T,I_2]$
generate new sample with size of $ n = 5000$ by applying linear transformation on $\underline{Z_i}$
$\underline{X_1}$ $\underline{X_2}\dots \underline{X_{5000}}$ , while $\underline{X_i} \sim N_2[(1,2)^T,\begin{pmatrix}2&1.5\\ 1.5&2\end{pmatrix}]$.
My attempt:
n1 <- 5000
mu <- c(0,0)
sigma <- diag(2)
y2 <- mvrnorm(n1,mu,sigma)

I have generated the first sample but aside from that , I have no idea how to continue..
Update
$X = A*Z + \mu$ while $\mu = (1,2)^T.$
To find $A$ we use this equation $\Sigma = A*A^T$.
ed = eigen(Sigma)
A = ed$vectors %*% diag(sqrt(ed$values))

but still I'm not getting the right value for $A$.
because $A*A^T\neq \Sigma$.


texts: What is best practice to feature engineer from prior event counts?

texts:
Say for example I am building a model to predict a customer churn event from Spotify, with my target being whether a customer churns in the next 90 days.
One feature I might expect could be predictive of this event is customers checking their billing statements online - so I might engineer features for each customer on each training date to encode the information of how many times they have checked their billing statements.
For example, I might create a feature CHECKBILL_CNT_0_10 which is a count of how many times this customer has checked their online bill in the last 10 days, with many of these such features across different time ranges.
I have seen two different styles of how data scientists do this:

CHECKBILL_0_10, CHECKBILL_0_30, CHECKBILL_0_90 ...
CHECKBILL_0_10, CHECKBILL_10_30, CHECKBILL_30_90 ...

Both technically encode the same information; however, I'm wondering if one of these options offers advantages over the other? I'm inclined to think that option 2 would be preferable since the features would be less correlated, & therefore the model might learn more easily, but this is speculative.


texts:
I have a video that consist of a lot of objects. I detect these objects correctly in every frame. For example you can see here 2 sequence frame detection of objects:

Frame 1:

Frame 2:

I want draw the path of all objects, but I don't know how.
the main problem is:

How can I find that the object in the next frame is the same object in the previous frame?


texts: Filtering outliers in Apache Spark based on calculations of previous values

texts:
I'm processing geospatial data using Spark 2.0 Dataframes with the following schema:

root
 |-- date: timestamp (nullable = true)
 |-- lat: double (nullable = true)
 |-- lon: double (nullable = true)
 |-- accuracy: double (nullable = true)
 |-- track_id: long (nullable = true)

I have seen that there are jumps of the location signal to a complete different place. The strange thing is, that the signal then remains for a certain time, say aound 25 seconds or 5 samples at the remote location and then jumps back to where I stand.

I'd like to remove these outliers by calculating the speed between the current and the "last valid record" by calculating the speed between the points.
If the speed is above a given threshold, the current record should be dropped and the "last valid record" remains the same.
If the speed is below the threshold the current record is added to the result data frame and becomes the new "last valid record".

I'm using Spark 2.0 with Dataframes.

Any suggestions of how to implement this strategy or any better strategy are highly appreciated. Thanks.

PS: I asked the same questions in stackoverflow, with a concrete implementation. But, since I'm not sure if this is the right approach, and do not want bias the answers to a certain Spark method, I ask here for any suggestions.
https://stackoverflow.com/questions/41002844/how-to-filter-outlier-rows-from-spark-dataframe-based-on-distance-to-previous-va


texts:
Let's say I've trained my model and made my predictions.

My question is... How can I append some new data to my pre-trained model without retrain the model from the beginning.


texts:
I'm currently training a CNN with Keras and I'm using the Adam optimizer. My plan is to gradually reduce the learning rate after each epoch. That's what I thought the decay parameter was for.  For me, the documentation does not clearly explain how it works:

  decay: float >= 0. Learning rate decay over each update.

However, when looking at the used learning rate in tensorboard, it stays the same as the initial learning rate. So, how does this decay parameter actually work?


texts:
I am working on using machine learning to correctly predict a binary classification using an input dataset that I receive about once a month.

The idea is that I train, test and validate the classifier on one month of data, and use it to classify another month. Whether I repeatedly add the next month of data to the training dataset and upset my classifier is open for debate.
I've spent a little while on this and have something that works well but it works better on some months than others. I am using either RandomForest or LinearSVC(penalty=’l1’) because I have a lot of features, around 400,000 observations and limited computing power.

I do not have much insight into the processes that produce the input dataset and based on previous analysis, I do not trust their quality control. What worries me is that one month I might receive a dataset that is very different to the data I have been training on and I wouldn’t really know unless the distribution of classes was quite different.

I am sure that I cannot be the first person to have this worry but I can’t find much information about it, so maybe I am misguided here. I’ve found the Population Stability Index, which is the right kind of idea, but I am unsure about it because it is dependent on bin size.

So my question(s):

How is a worry like this normally dealt with?
Can you point me to some ideas/case studies that will help me?
Is there anything I can do on the classification side that will make
my classifier more robust to population instability?


texts: I tried finding about exception lists in wordnet lemmatizers. "Morphy() uses inflectional ending rules and exception lists to handle different possibilities" which I read from http://www.nltk.org/howto/wordnet.html . Can you explain what is an exception list. Thank you.


texts: What is the typical accuracy of masked language models during BERT pretraining?

texts: I was reading the BERT paper but I didn't find any tables concerning the performance of the masked language models during pretraining. Does anyone know the accuracy of BERT's masked language model?


texts: Xgboost in R: Dealing with extreme class imbalance

texts:
My data has an extreme class imbalance - 99.7% is 0's and 0.2% is 1's and almost all the predictor variables (6 out of 7) are categorical. I trained an xgboost classifier after performing an upsampling (using ROSE in R). Evaluated the model on 20% test data and the following is the result,

Recall, as you can see, is not great. Additionally, if I run the model on a completely different test data (meaning not a chunk of the original data), then the recall value drops to exactly 50%. Is this happening because it is overfitting? Can someone please let me know how I can better this model, considering the extreme class imbalance?

I'm a newbie in the field. Any help would be much appreciated. Thanks!


texts: With Numpy, what’s the best way to compute the inner product of a vector of size 10 with each row in a matrix of size (5, 10)?


texts: How to do template matching without opencv?

texts:
How to do template matching without opencv ?

I have a order invoice of documents belonging to Amazon ,ebay,flipkart,snapdeal and i want to extract few information from the order invoice .
Since the fields like order number,customer name ,order details will be present at different position in these 4 templates ,

i need to first classify to which of these 4 templates the input image will belong to and after identifying the template i can do my next work of text extraction using tesseract and regex buy writing code for specific templates .

when i searched i found only openCV had this feature i was not able to find any convolution neural network model .Are there any such neural network model available for template matching and classification ?

i cannot use the standard model available for classifying dogs and cat because here im dealing with image of an invoice templates which has only text with rows and columns in some format .


texts: What are some good machine learning techniques for developing rules that govern changes in data?

texts:
Say I read a set of data in and it looks like so:

[ 1, 2, 3, 4, 5 ]

Now, there's an unknown rule set by my data's source that says if the 3rd element in my list is set to 5, the fifth element must be greater than 10. So, subsequent data reads look like this:

read #2:

[ 2, 2, 3, 4, 5 ]

read #3:

[ 3, 2, 5, 4, 11 ]

read #4:

[ 4, 2, 5, 4, 11 ]

If I'm only seeing the data, is there a way I can extrapolate that rule, so I end up with this:

if (field3 == 5 and field5 < 10)
    return false;
return true;

I know it would be hard for a machine to differentiate the rules without far more data than this example, but in reality, I have thousands of these data records with hundreds of rules that govern their state.

Is there an established method for doing this type of pattern recognition?


texts: How hidden layer is made binary in Restricted Boltzmann Machine (RBM)?

texts:
In RBM, in the positive phase for updating the hidden layer(which should also be binary), [Acually consider a node of h1 ∈ H(hidden layer vector)] to make h1 a binary number we compute the probability of turning on a hidden unit by operating activation function over total input (after the activation function operation, we would be getting values in the range between 0 and 1, since activation function I am using - sigmoid).
My doubt is that how do we make it binary by leveraging the probability computed. I don't think if P>=0.5, make it 1 else 0 is a proper method to work on.

By few literature reviews,
I found this document (by Hinton), in section 3.1: he has stated "the hidden unit turns on if this probability is greater than a random number uniformly distributed between 0 and 1". What does this actually mean? And also in this link, they say "Then the jth unit is on if upon choosing s uniformly distributed random number between 0 and 1 we find that its value is less than sig[j]. Otherwise it is off." I actually didn't get this. Whether the random number generated is same for all h ∈ H ? Another query is, what about the random in next sampling iteration?

I saw this video. Just watch the video from that point as per the link. How do you get that sampled number? Whether we have to just run rand() in Matlab and obtain it? Should it would be different for each h(i) (oh nooo! I don't think the machine will learn properly)? Whether the random number should be different for each iteration or the same random number can be used for all iteration to compare?


texts: What is advantage of using the theory of point processes instead of probability distribution functions?

texts:
I am interested in applications of point processes in machine learning, so I have been studying the theory of point process. However, I cannot see advantage of the theory of point processes compared with the classical probability theory.

For examples, I know the points of a Poisson point process scatters uniformly on a given domain. But, I'm confused what is the difference between Poisson point processes and uniform distribution functions (especially on a bounded domain).

I have similar questions for not only Poisson point processes but other point processes. Take sample points according to a point process. Can we obtain a similar (random) point set via using a probability distribution function?

What is advantage of using the theory of point processes instead of using probability distribution functions in statistics and machine learning?

Thank you so much in advance.


texts:
I am new using pandas.
I am asking about how to insert specific columns using to_sql.

This is my explanation.

I have a database with a table datasiswa with columns:

id: as int, with autoincrement, as primary key
name: string
age: string

And I have an excel file with header name and age.

How to use to_sql to insert in fields name and age only, without ids.


texts: I'm planning to record video now and then analyze it in the future once AI programs are more accurate, what resolution and FPS should I record at?

texts:
I'm going to record videos and want to future proof them for AI analysis as much as possible.
I have a decent amount of storage to work with, 2.7TB. I'm hoping to make it last for at least two years. I would be recording for about 10 - 15 minutes every day.
I'm trying to decide between 30FPS vs 60FP and 1080p vs 4K.
I'm leaning towards 1080p at 60FPS.
I would think recording at a higher resolution and FPS would be better because it would give me more data to work with and I could analyze the smaller changes. Is my thinking correct? I don't know if higher-resolution or higher framerate would be more useful in analysis. Thanks.


texts: how to get probability in classification problem

texts:
I am learning ML with this tutorial. https://pythonprogramming.net/using-trained-model-deep-learning-python-tensorflow-keras/
When I try to use the model to predict, I only get 0 or 1, but I thought that since the last layer is
model.add(Dense(1))
model.add(Activation('sigmoid'))

the final output would be in the range [0,1] and not the extreme value 0 or 1. How can I get the exact result?


texts: Statistical Significance for Pairwise Ranking Algorithm

texts:
I am using this library: https://github.com/lucasmaystre/choix/blob/master/notebooks/intro-pairwise.ipynb

to rank items that have been rated in a hot-or-not style. For example, two images are shown to users and the user picks the one that appeals most to him/her. The final output is a ranking: worst to best: image1 image5 image3 etc.

The algorithm is using a Bradley-Terry model on this data. choix provides several algorithms and uses a maximum-likelihood inference algorithm called I-LSR.

However, I do not know the statistical significance of the ranking. How do I figure out if this ranking is statistically significant?


texts: Reinforcement Learning: Policy Gradient derivation question

texts:
I have been reading this excellent post: https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 and following the RL-videos by David Silver, and I did not get this thing:

For $\pi_\theta(\tau) = \pi_\theta(s_1, a_1, ..., s_T, a_T) = p(s_1) \prod_{t=1}^T \pi_\theta(a_t | s_t)p(s_{t+1}|a_t, s_t)$ being the likelihood of a given trajectory in a cycle, the derivative of the value function becomes
$$\nabla_{\theta}J = E[\nabla_{\theta}log\pi_{\theta} \cdot r ]$$

which then immediately becomes

$$={{1} \over {N}} \sum_{i=1}^{N}(\sum_t^T \nabla_\theta log \pi_\theta(a_{i,t}, s_{i,t})) r$$

i.e. summed over all N paths for $\tau$, while I expected

$$=\sum_\tau \pi_\theta(\tau) \sum_t^T \nabla_\theta log \pi_\theta(\tau) r$$

What I do not get: Where does the probability for the trajectories $\pi_\theta(\tau)$ (left-most sum) go or why did it get replaced by the mean over all paths? Is it assumed that all trajectories are equally likely, given that you start from a known starting position?

(You can find the equations in the blog-post linked above, at the end of the chapter "Optimization", right before the chapter "Intuition".)


texts: How to decide which forecasting model to use?

texts:
So I have a sales forecasting problem where I have 3 years worth of data about weekly sales of a certain company.

There are 3 types of retail stores in that company, Type A, B, and C. There are a total of 45 stores which fall in either of the 3 category. There are a total of 99 departments under each store, but not all stores have all the 99 departments.

I also know which weeks were holiday weeks. I also have the average temperature, average fuel prices, average unemployment rate and the average consumer price index for each week of the region where a particular store is located. I also have the size of each store. Add to it I have the weekly sales for each store and whatever department is available. And I have a certain thing which are called markdowns for each week, which although I haven't been able to figure out what they mean.

Irrespective, after a comprehensive exploratory data analysis, I have come to the conclusion that:

Type A stores are larger in size than Type B stores which are larger in size than Type B stores. And Type A stores have higher average sales over the period of 3 years worth of data that I have
The weekly markdown data that I have aren't affecting the sales
The month of December records the highest sales
During the holiday weeks, the sales increase

So basically I need to have a model which works on one or all of the above mentioned factors. But I am not able to figure out which model should I be going with. Would be great if anybody could at least just give me a certain direction.

Thanks.


texts: why is more layers better in deep learning?

texts: i understand mathematically that deep learning has more than one hidden layer, whereas regular machine learning hs just one. is that right? if so, why and how is it better to have more than one layer, that give deep learning the edge over machine learning?


texts:
Let us say we are using a neural network with $4$ layers with $50,30,20,10$ neurons each. The problem of vanishing gradient would mean that the rate of change of parameters associated with the earlier layer (say first) would be significantly lower than the rate of change of the parameters associated with the later on layers (say 4th).
Now the cost function spans across $n$-dimensions where $n$ is the total number of the parameters and we try to find the minimum of this cost function by varying it across these dimensions.
My question is why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.
Here is the help I can provide on this: this is from neuralnetworksanddeeplearning.com

One response to vanishing (or unstable) gradients is to wonder if they're really such a problem. Momentarily stepping away from neural nets, imagine we were trying to numerically minimize a function $f(x)$ of a single variable. Wouldn't it be good news if the derivative $f′(x)$ was small? Wouldn't that mean we were already near an extremum? In a similar way, might the small gradient in early layers of a deep network mean that we don't need to do much adjustment of the weights and biases?
Of course, this isn't the case. Recall that we randomly initialized the weight and biases in the network. It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. Even if later layers have been extensively trained, they will still find it extremely difficult to identify the input image, simply because they don't have enough information. And so it can't possibly be the case that not much learning needs to be done in the first layer. If we're going to train deep networks, we need to figure out how to address the vanishing gradient problem.

Specific problems from the excerpt:

It is extremely unlikely our initial weights and biases will do a good job at whatever it is we want our network to do. Why? Why cant the cost function be nearly flat in the dimensions created by the earlier weights. To be precise why is the explanation not the the last 3 layers are enough to do such a bangup job that they always find weights corresponding to a very small proximity of whatever random weights we initialized the first layer to. Potato pothatoe. i am asking the same thing in different words to make sure i convey myself

To be concrete, consider the first layer of weights in a $[784,30,30,30,10]$ network for the MNIST problem. The random initialization means the first layer throws away most information about the input image. What does this mean? Kindly explain. from what i understand it is that we cant trace back the second layer inputs unambiguously to the first layer inputs. but, even if we are initializing the weights randomly, we still know what those weights are so how is it that we are throwing away the information?


texts: what can be done with the following data?

texts: I have scraped the rating that a customer gave in the following categories namely overall rating score, value for money, seat comfort,staff service, catering and entertainment from an airline forum. I would like to know what quality information I can deduce from such ratings about an airline. I have rating of nearly 300 customers who traveled in an airline.


texts:
I am working with 3D matrix in Python, for example, given matrix like this with size of 2x3x4:

[[[1 2 1 4]
  [3 2 1 1]
  [4 3 1 4]]

 [[2 1 3 3]
  [1 4 2 1]
  [3 2 3 3]]]

I have task to find the value of entropy in each row in each dimension matrix. For example, in row 1 of dimension 1 of the matrix above [1,2,1,4], the normalized value (as such the total sum is 1) is [0.125, 0.25, 0.125, 0.5] and the value of entropy is calculated by the formula -sum(i*log(i)) where i is the normalized value. The resulting matrix is a 2x3 matrix where in each dimension there are 3 values of entropy (because there are 3 rows).

Here is the working example of my code using random matrix each time:

from scipy.stats import entropy
import numpy as np

matrix = np.random.randint(low=1,high=5,size=(2,3,4)) #how if size is (200,50,1000)
entropy_matrix=np.zeros((matrix.shape[0],matrix.shape[1]))
for i in range(matrix.shape[0]):
    normalized = np.array([float(k)/np.sum(j) for j in matrix[i] for k in j]).reshape(matrix.shape[1],matrix.shape[2])
    entropy_matrix[i] = np.array([entropy(m) for m in normalized])

My question is how do I scale-up this program to work with very large 3D matrix (for example with size of 200x50x1000) ?

I am using Python in Windows 10 (with Anaconda distribution).
Using 3D matrix size of 200x50x1000, I got running time of 290 s on my computer.


texts: How to maintain CBOW dataset dimension and fit it in Neural Network?

texts:
I am new to neural network. I'm trying to train word embeddings without using word2vec package.

Using titles from reddit worldnews dataset I'm have done some CBOW representation.

For window size three here is some of my outputs:

0 Context : ['scores', 'killed', 'pakistan'] ---> Target: clashes
1 Context : ['japan', 'resumes', 'refuelling'] ---> Target: mission
2 Context : ['us', 'presses', 'egypt'] ---> Target: gaza
3 Context : ['presses', 'egypt', 'gaza'] ---> Target: border

For Vocab size = 513, I've collected 369 of target words against 369 of 3-grams context words. Each context word is one-hot codded of length 513.

Therefore, my dataset length becomes:
X.shape = (369, 3, 1, 513)
Y.shape = (369, 1, 513)

Now I'm having trouble in fitting the data in neural network. My Neural Network model is constructed with keras.

# create model
model = Sequential()
model.add(Dense(100, input_dim=1, init=369 'uniform' , activation= 'sigmoid' ))
model.add(Dense(1, init= 'uniform' , activation= 'sigmoid' ))
model.compile(loss= 'binary_crossentropy' , optimizer= 'sgd' , metrics=['accuracy'])

#train
history = model.fit(X, Y, nb_epoch=100)

Raised Error:

ValueError: Error when checking input: expected dense_9_input to have 2 dimensions, but got array with shape (369, 3, 1, 513)


texts:
I'm curious on how to interpret the following learning curves, which can be considered an actual good learning curve of the three learning curves. I'm not sure when to consider if there is a high variance or bias, likewise the presence of underfitting or overfitting respectively. Thank you in advanced.
1.

2.

3.


texts: I've trained an NER model with the use of Spacy, and I would like to test the accuracy on a test dataset. What would be the best way to perform this?


texts: How would you categorize email subjects to find similar emails?

texts:
I have a list of email subjects like:

<XYZ> commented on <ABC>
Weekly review for <Company>
Your account is ready

And I want to find patterns in them so I can group them.

Is there a well known algorithm I can use?

Preferably with wide language implementations or easy re-implementation.
The algorithm should be unsupervised.
The number of different emails is not known.

Update:

I think I can break this down into two problems:

Group subjects by the similar words they use, resulting in the following. Each group should be very distinct from the rest (they should be almost perfectly exclusive) and the algorithm should give relatively small number of groups with good length of the common words.

[commented, on]
[weekly, review]
[your, account, is, ready]

Once grouped, it should be easy to find a state automaton that accepts only the group's subject and thus eliminates variable
Then I can go back and check if there are any intersections and tweak the variables.

Having said that, is it better to use a completley different approach like neural nets maybe? I have zero experience with those, but if it makes more sense, I am open to learning.


texts: Compare Neural Network generalization results

texts:
I'm trying to develop my neural network with both early stopping and bayesian regularization (matlab implementation, lm algorithm is used for both).
Since in bayesian regularization I have not the validation set, how can I compare the generalization capability of the networks obtained with the two methodologies?
Thanks


texts: Dynamic learning rates in XGBoost cross-validation

texts:
XGBoost's xgb.train() method takes a learning_rates parameter, which can take a custom function to apply a dynamic learning rate, depending on the current training round.

I recently posted a paper explaining how I'm using it to both speed up training in the beginning, and making more precise towards the end.

However, there's a problem with this method: it tends to overfit on the eval set because there's no cross-validation.

I noticed that xgb.cv() method has no learning_rates parameter and therefore appears to not allow for a dynamic learning rate while doing CV.

So my questions:

Is there a possibility to use dynamic learning rates with xgb.cv()?
If not, is there a rationale behind this or is this just not implemented yet?


texts: Evaluating randomness in a model performance

texts:
I'm evaluating the variability in performance (AUC) in the test set of a machine learning model with an intrinsic random component (xgboost). How many sources of variation should I use?

Just replicate the analysis harnessing the model intrinsic randomness?
Bootstrap the train set?
Bootstrap the test set?
Bootstrap both train and test?

Thanks


texts: Does anybody know if there is a Mixed effect random forest model for Python Windows? The merf package https://anaconda.org/search?q=merf+ seems to only be available on a linux environment? thanks!


texts: I am looking for accuracy python code for kmeans clustering with no labels. Is there anyone who knows about it? it is ok that is not built-in function. Manually made is also ok


texts: Using Downsampling with RFE using pandas and sklearn

texts:
I have dataset which consist of 2 class with heavily unbalance distribution (600 for class A and 3000 for class B). When i do RFE to select our feature and test them using Decission tree, it gives a high accuracy but still out of our target. We found that the unbalanced data is one of the problem.
So we downsample the class with larger size and do RFE with CV. The result are still satisfied and better than before. But the problem is we only use CV based from balanced data so it will lost some important dimension from other record.
We want to know whether we can do a pipeline to do downsample inside sklearn RFECV as the CV data.
This is our code. We combine Scikit-Learn and Pandas to process our features:
CLASS_A = raw_data_shadow.loc[(raw_data['target'] == 1)].sample(frac=1)
CLASS_B = raw_data_shadow.loc[(raw_data['target'] == 2)].sample(frac=1)
len_CLASS_A = len(CLASS_A)
len_CLASS_B = len(CLASS_B)
print(len_lancar,len_macet)
CLASS_A_USED = CLASS_A.sample(n=len_CLASS_A,replace=True)
raw_data_shadow = CLASS_A_USED.append(CLASS_B)
Y = raw_data_shadow['target']
X = raw_data_shadow.drop(columns=['target'])
clf = clf = DecisionTreeClassifier()
rfe = RFECV(estimator=clf, cv=10, min_features_to_select=20, scoring='roc_auc')
#rfe = RFE(clf, n_features_to_select=10)
rfe = rfe.fit(X, Y)


texts: Is there any clustering algorithm to find longest continuous subsequences?

texts:
I have data which contains access duration of some items.

Example:
t0~t5 is the access time duration, 1 means the items was accessed in the time duration, 0 means it wasn't.

ID,t0,t1,t2,t3,t4
0,0,0,1,1,1
1,0,1,1,1,1
2,0,1,1,0,0
3,1,1,0,0,1
4,1,1,0,0,1

In the above example, groups ID=0,1 are what I want.

ID=3,4 aren't because their distance is short but they are not continuous.

I tried KMeans and DBSCAN, they all cluster ID=3,4 as one group and it makes sense. But it doesn't do what I want.

Is there any possible pre-processing of data to reach what I want ?

Or I should use other analytic tool?


texts: Categorical vs continuous feature selection/engineering

texts:
I'm working with a dataset with a number of potential predictors like :

Age : continuous

Number of children : discrete and numerical

Marital Situation : Categorical ( Married/Single/Divorced.. )

Id_User : Categorical ( an id of the user who conducted the first interview with this person )

I'm stopping at four potential predictors, there are more, but for the sake of shortness, these would be enough to ask my question.

Question : Continuous features are easy to deal with, normalize, and feed it to the model, what about categorical and independant ?

Note : I get that categorical features that follow a certain pattern can be encoded as integers and fed to the model, but what if those categorical features have no meaning as integers ( 1 for single, 2 for married , 3 for divorced ; for the model that treats it as a quantitative predictor it doesn't make sense to feed it to it like that)

Any ways to deal with these different types of features?


texts: Cross validation for unbalanced dataset using Orange data mining tool

texts:
I am using the Orange data mining tool to build and analyze models (decision tree, ANN, ...) predicting customer churn. As this is an imbalanced class problem (10% churn, 90% not churn), I need to oversample within the cross validation. However, I am not totally able to implement this by myself. Is there anyone with some Orange knowledge that could help me?
Thank you!


texts: What features are generally used from Parse trees in classification process in NLP?

texts:
I am exploring different types of parse tree structures. The two widely known parse tree structures are
a) Constituency based parse tree and
b) Dependency based parse tree structures.

I am able to use generate both types of parse tree structures using Stanford NLP package. However, I am not sure how to use these tree structures for my classification task.

For e.g If I want to do sentiment analysis and want to categorize text into positive and negative classes, what features can I derive from parse tree structures for my classification task?


texts: Good Libraries or Software for Temporal Bayesian Network Structure Learning?

texts: I would like to infer structures of BNs with edges within and between time slices by using data. I tried libpgm (http://pythonhosted.org/libpgm/) and found that 2TBN is ideal for my purpose, but libpgm does not support structure learning of 2TBN. BNFinder and ebdbNet cannot learn edges within time slices. Could you tell me any great libraries and/or software with which I can infer structures of 2TBN from data or something similar?


texts: How do I calculate for each person in the network how many people agree with their opinion?

texts:
I have a signed bipartite graph in which the nodes are (1) students and (2) topics. An edge is drawn between a student and topic node if the student mentions their opinion about the topic in a short answer (i.e., some students have an opinion about one topic but not another). The valence of the edge indicates whether the opinion is positive or negative.

My question is: how do I find out how many other students agree with a particular student? In terms of not just which topics they have an opinion on, but also what that opinion is (positive/negative).

EDIT: Based on a comment below

1) What exactly do you mean by agree? Should all existing opinions align or only those on a specific topic? What if one student has additional opinions about a topic?

All existing opinions (including topics and the valence) should align. If a student gives an same opinion about the same topics as another, but happens to talk about another topic as well, that would not count as full agreement. Perhaps there is a way to calculate partial agreement as well?

2) What exactly is your problem? Defining the characteristics is straightforward: Just count. Do you perhaps struggle with an algorithmic implementation?

Algorithmic implementation is perhaps what I am going for. Since I have 100 students, it would be difficult to hand-count the number of peers that agree with them. So if there is a way to calculate a value for each student, that would be helpful.


texts: How can I override the values of 1 column in a big dataframe using a small second dataframe?

texts:
I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are:

A recipient_id column indicating who received the donation
An office column indicating what legislative office the recipient holds.

Some of the values of the office column in donations_df are outdated, and I want to replace/override them.

So I have a second, much smaller dataframe (overrides_df). It contains the preferred office names I want to use. It has only a few rows (one for each recipient). For example:

| recipient_id | office             |
| ------------ | ------------------ |
| 3839314      | Governor           |
| 24811446     | Secretary of State |
| 12733609     | Auditor            |
| 6676482      | Attorney General   |

What is the proper way to join/merge/update this?

What I want to happen is that for each donation in donations_df, if that donation's recipient_id exists in overrides_df, then then the donation would pick up the new office value from overrides_df

What is the proper way to join/merge/update these two to achieve that result?


texts: test accuracy of text classification is too less

texts:
I have a data set of movies and their subtitles. My task is to classify them based on their ratings - [R, NR, PG, PG-13, G].

I have 13 examples for each class.
I preprocessed the subtitles in the following way:

I used word puns tokenizer to tokenize subtitles.
Removed stop words and punctuation.
Performed stemming.
Vectorized the subtitles using TF-IDF vectorizer.

The accuracy that I am getting using:

1) svm :
train accuracy is 1.0 and test accuracy is .17.

2) naive bayes:
training accuracy is 0.5 and test accuracy is .23.

I have the following questions:

1)Why is my accuracy so low and what can I do to improve the accuracy?

2)Will more training data help?

3)Should I perform feature selection?

4)What other classification algorithms can I use to improve the accuracy?


texts: How can I map category values to , e.g., increasing numbers (for anonymization purposes, say) in Orange?

texts: I have a dataset that contains data that needs to be preserved for e.g. clustering purposes, but whose concrete values do not matter. For example, order numbers or serial numbers with lots of accompanying measurement data. I would like to anonymize the serial numbers and order numbers, but still be able to discern measurement values that belong to the same order number or serial number. How can this be done in Orange3, preferrably w/o using Python? (In Knime Analytics there is a building block for just that.)


texts: How to compare performance between SVM and Keras models

texts:
I applied both SVM and CNN (using Keras) on a dataset. Now, I want to compare the performance of both models.

Keras model.evaluate function predicts the output for the given input and then computes the metrics function (A metric is a function that is used to judge the performance of a model) specified in the model.compile and based on y_true and y_pred and returns the computed metric value as the output. It is obvious that SVM's accuracy calculation is different than Keras' model.evaluate.

For my dataset, Keras model (accuracy calculated based on model.evaluate) gives better accuracy than the SVM. However, a comparison between y_true and y_predict (get the y_predict by model.predict) gives a similar accuracy (this accuracy is a little lower than the accuracy return from Keras evaluate) to the SVM.

I want to know, in this scenario, how can I compare the result between both SVM and Keras model? May I conclude that Keras' model is better as its model.evaluate is giving better accuracy than SVM. What is the standard approach in this scenario to compare performance between the two models?


texts: Are deep - learning toolkits targeted for certain areas or all-purpose tool kits?

texts:
Are any of the open deep learning toolkits targeted to certain areas, or a all toolkits all purpose toolkits, meaning it is a blackbox for deep learning.

My question comes in regards to Microsoft's CNTK which seem to contain examples of speech and text classification, where others usually just have MNIST or CIFAR...


texts: Should I remove features that occur very rarely to build a model?

texts:
I am trying ML techniques in language processing.
I have got 3000 short texts and I extract features(words and phrases) from all of them and build a vocabulary. I end up with 6000 od features and most of them occurs once or twice.
So for example from texts:

0: One text here
1: Another text there

I got
    One Another  Text  Here  There   Target
  0   1       0     1      1     0     True
  1   0       1     1      0     1    False

So if word "one" occured once I have got as a column and it is False for all other 2999 texts.
Should I drop these columns? Or should use different technique?
This amount of columns make me some problems becuase it takes a lot of time to build a classify.


texts: Data Snooping, Information Leakage When Performing Feature Normalization

texts:
Assume that we have a training data set (with both features and labels) and a test data set (with only features).

When we build a machine learning model that requires normalization of the features, the correct way of doing normalization (to prevent information leakage) is to only use the training data set. Namely, a wrong way of doing normalization is to stack the training (exclude the Y column) and test data set together, and perform normalization (i.e., using the mean and variance of the entire training + test data set). The intuition here is very clear: if you want to get an unbiased estimator of your model performance in production, then when you train your model, you shouldn't instill any information in the test data set that will be used to gauge the actual performance of your model.

My question is the following: When we have trained the model correctly and about to use this model to do prediction, how should we normalize the test data set? I believe the correct way to normalize the test data set is: using the mean and variance obtained from the training data set to do normalization on the test data set.

However, why not just normalize the test data using its own mean and variance? Or why not stack train and test data set together and using the overall mean and variance to normalize the test data set? In the prediction stage, the idea and intuition of data snooping and information leakage is not clear to me.


texts: how to know what my data are uninformative and that machine learning will not work with it?

texts:
following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I also tried Neural Networks and the results were worse than Random Forest.

So I thought maybe is something wrong with my data as I also read that data can have bad quality somehow(I'm not sure what does this mean). Since I'm not a data scientist and don't have background in Data Science in general (I'm an Electrical Engineer), I want to ask here and maybe someone can help me.

I read that correlation between features and target that I want to predict is important, now this is not my case, my features doesn't have a strong correlation with my features, does this mean that my data is bad and I can't predict that target from those features ?

I ll add here an image of the heatmap of my data:

my target that I want to predict is xdistance and ydistance. It have negative correlation with one another but I think it is not important since I'm predicting every target on its own, but what is interesting here is that there is no feature that correlate strongly with one of the two targets. So I fear that this is an impossible task to try to predict those targets, am I right?


texts:
For example I have the following data structure:

user: Chris
age: 32
income: 60.000
basket value: 45

I want predict the basket value, and my features are the age and income.

With a linear regression I get a regression function as the result of the fitting for example: $$y = 0.5x + 0.785$$

Now I can use the function for prediction.

What is the form of the result of the fitting by regression decision tree? Is it also a function?


texts:
Below is the dataset for which I am trying to implement Linear regression in
python.

   age     sex     bmi  children smoker     region      charges
0   19  female  27.900         0    yes  southwest  16884.92400
1   18    male  33.770         1     no  southeast   1725.55230
2   28    male  33.000         3     no  southeast   4449.46200
3   33    male  22.705         0     no  northwest  21984.47061
4   32    male  28.880         0     no  northwest   3866.85520

I am confused what to do with the columns children, smoker, sex as they are of type "object".

Data columns (total 7 columns):
age         1338 non-null int64
sex         1338 non-null object
bmi         1338 non-null float64
children    1338 non-null int64
smoker      1338 non-null object
region      1338 non-null object
charges     1338 non-null float64

Do I have to convert this to numeric before building my model ?
Please provide your suggestions.

Thank you.


texts:
I try to train a neural network for time series. I use some data from Covid, mainly the goal is knowing 14 days of number of people at hospital to predict the number at J+1.
I have use some early stopping to not over fit, but almost one time over two the learning stop at patience+1 and there is no decrease of loss and val_loss. I have tries to move hyperparameters like learning rate but the problem is always here. Any guess?
The main code is below and the whole code with data :https://github.com/paullaurain/prediction
import os
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
from keras.models import load_model
from keras.optimizers import Adam
from sklearn.preprocessing import MinMaxScaler

# fit a model
def model_fit(data, config):
# unpack config
    n_in,n_out, n_nodes, n_epochs, n_batch,p,pl = config
# prepare data
    DATA = series_to_supervised(data, n_in, n_out)
    X, Y = DATA[:, :-n_out], DATA[:, n_in:]
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.1)
# define model
    model = Sequential()
    model.add(Dense(4*n_nodes, activation= 'relu', input_dim=n_in))
    model.add(Dense(2*n_nodes, activation= 'relu'))
    model.add(Dense(n_nodes, activation= 'relu'))
    model.add(Dense(n_out, activation= 'relu'))
    model.compile(loss='mse' , optimizer='adam',metrics=['mse'])
# fit
    es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=p)
    file='best_modelDense.hdf5'
    mc = ModelCheckpoint(filepath=file, monitor='loss', mode='min', verbose=0, save_best_only=True)
    history=model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=n_epochs, verbose=0,batch_size=n_batch, callbacks=[es,mc])
    if pl:
        plt.plot(history.history['loss'])
        plt.plot(history.history['val_loss'])
        plt.title('model loss')
        plt.ylabel('loss')
        plt.xlabel('epoch')
        plt.legend(['train', 'test'], loc='upper left')
        plt.show()
    saved_model=load_model(file)
    os.remove(file)
    return history.history['val_loss'][-p], saved_model

# repeat evaluation of a config
def repeat_evaluate(data,n_test, config, n_repeat,plot):
    # rescale data
    scaler = MinMaxScaler(feature_range=(0, 1))
    scaler = scaler.fit(data)
    scaled_data=scaler.transform(data)
    scores=[]
    for _ in range(n_repeat):
        score, model= model_fit(scaled_data[:-n_test], config)
        scores.append(score)
    # plot the prediction id asked
        if plot:
            y=[]
            x=[]
            for i in range(n_test,0,-1):
                y.append(float(model.predict(scaled_data[-14-i:-i].reshape(1,14))))
                x.append(scaled_data[-i])
            X=scaler.inverse_transform(x)
            plt.plot(X)
            Y=scaler.inverse_transform(np.array([y]))
            plt.plot(Y.reshape(10,1))
            plt.title('result')
            plt.legend(['real', 'prdiction'], loc='upper left')

            plt.show()
    return scores

# summarize model performance
def summarize_scores(name, scores):
# print a summary
    scores_m, score_std = mean(scores), std(scores)
    print( '%s: %.3f RMSE (+/- %.3f)' % (name, scores_m, score_std))
    # box and whisker plot
    pyplot.boxplot(scores)
    pyplot.show()

#setting variable
n_in=14
n_out=1
n_repeat=5
n_test=10
# define config n_in, n_out, n_nodes, n_epochs, n_batch, pateince, draw loss
config = [n_in, n_out, 10, 2000, 50, 200,True]
# compute scores
scores = repeat_evaluate(data,n_test, config, n_repeat, True)
print(scores)
# summarize scores
summarize_scores('mlp ', scores)

Result:


texts: How to improve identification of outliers for removal

texts:
I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running).

Unfortunately, I don't have a measurement that clearly indicates that the process is running, so I have to make the determination from the measured value data itself. Visually, the outliers are obvious (see below).

I have had success with several different techniques

IQR based outlier identification
3 sigma rejection
simple cutoff
modified z-score

These all work some of the time, but I am looking for a better solution for my data.

If you look at the image below, the outliers are all of the points near zero. In this specific case a threshold of 0.01 easily eliminates all of the bad data. Unfortunately, the threshold required for a given dataset can vary significantly.

I would appreciate any thoughts.


texts: Practical limitations of machine learning

texts:
Working on some applied machine learning problems, I've started to encouter some practical difficulties. Those difficulties relate to - but are not limited to - convergence of the learning process, stability trough recalibration, explainability, stability of the explainability trough recalibration. Those problems are a bit difficult to handle, notably because:

They are not really dealt with in introductory theoretical books. (However, there are some advanced resources to deal with them individually.)

They are mostly specific to advanced ML models, compared to more standard approaches.

You only get aware of them once you stumble upon them by doing some very specific tests.

I would be interested in some serious sources on the main practical problems related to machine learning (not necessarily how to deal with them). Both as a checklist of pitfalls to avoid and as an authoritative argument if I have to formulate some concerns about any future ML solution. Can you help me with any sources?


texts: CNN - imbalanced classes, class weights vs data augmentation

texts: I have a set of data with a few strongly imbalanced classes, eg the smallest class is about 54 times smaller than the largest. Therefore, data augmentation in order to equalize the size of classes seems to me a bad idea (in the example above each image would have to be augmented 54 times on average). So I thought that I could do less augmentation of minority classes, and then use class weights in the loss function. Is this approach better than the mere augmentation or just the use of class weights ?


texts: Featuring scaling whole data set before spliting it.

texts:
I am wondering why do we use scaling on train and test set separately. I understand that transform () on test data  μ and σ as computed from fit_transform() on  Train. But why can we compute μ and σ from all given data (before split) and then apply them on future data.

Do we do this because we don't know how the size of our future data?


texts:
I want to extract various amounts and tenure of contracts from different contract documents that we have.
For example: Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as agreement fee.
Expected output : 3 Months, $3000
Please note that this is just an example but the sequence, format, currency and tenure is not fixed in the actual problem.


texts:
I am training an SVM model to predict the trend of stock prices (one-day ahead predictions. Classification task). It Had completely slipped from my mind that SVMs assume IID data until I had a conversation with a friend.
This made me rethink about my approach and I have a few questions.

Why does SVM assume IID data in the first place?

By IID, does it mean that all the features should be linearly uncorrelated or should there not be any non-linear relationships as well?

The features include basic stock data (Open, High, Low, Close, Volume) and technical indicators (derived from basic data). Of course, the technical indicators will have dependencies with basic data. So I removed the basic data from the feature space. However, this does mean that the technical indicators are not related. Also,

How does one make sure that the data is identically distributed? Does scaling (Normalization or Standardization) help in this case?

Lastly, I have read some papers wherein people have used NN and SVM (both assume IID data) for stock trend prediction, but nowhere did I see any mention of the IID assumption and the fact that stock data is actually not uncorrelated and to some extent exhibits some autocorrelation as well. So,

How does one justify the use of non IID data as input to such algorithms which assume that the data is IID?

Thank you :)


texts: Python 3: Please help.. Found input variables with inconsistent numbers of samples: [3292, 3326]

texts:
I am trying the below code:

X = np.array(df.drop(['label'], 1))
X = preprocessing.scale(X)
X = X[:-forecast_out]
df.dropna(inplace=True)
y = np.array(df['label'])
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2)
X = X.reshape(X.shape[1:])
X = X.tranpose()

and getting error:

  Found input variables with inconsistent numbers of samples: [3292, 3326]


texts:
I have set of newspaper articles. I want to identify important less frequent words in the set of newspaper articles. Currently I am using TF-IDF scores. However, it does not seem to be a good metric in my problem. Is there any better way of doing this?

Thank you in advance :)


texts: Linear Regression on data with bimodal outcome

texts:
I have a data set with 3,000 features and continuous dependent variables of time with 18,000 instances. The histogram of the dependent variables show that the they have a bimodal distribution. I am building linear regression models that forecast the time, but none of the models are able to make predictions; the $R^2$ values of all of the models are $0$. I plotted the residuals of the models and verified that they are normally distributed

I have used LassoCV/Lasso, ElasticNetCV/ElasticNet, XGBoost, and Kernel Ridge Regression in Scikit-Learn to make the models. I used the weights of the Lasso coefficients to filter my data frame so that I use the features that have the most magnitude in my predictions, but that did not work. I used the feature_importance_ method for XGBoost to do something similar, but that also did not help.

Kernel Ridge Regression ran with the parameters

KernelRidge(alpha=1, coef0=1, degree=3, gamma=0.001, kernel='polynomial',
      kernel_params=None)

but overfit the training data completely with $R^2 = 1$.

I have been looking into non-parametric methods like Kernel Density Estimate and Gaussian Mixture Models, but am uncertain in how to use them for my regression task. I have also been looking into Modal Regression and fitting of multi-functions, but am not familiar in how to implement them with Python.

I have also considered finding out what features are contributing to the modes and splitting the data set as two distributions and running the models on the two separate data sets, but am not sure on how to paint the picture for the entire data set.

In general, how should I approach building a regression model with a multimodal dependent variable?


texts: What is the best deep learning architecture for image classification with different intra-class variances?

texts:
I implemented a CNN image classifier with three classes. The samples belonging to the first class have a low variance (patterns and colors are very similar from image to image). The samples from the second class have a higher variance. Finally, the samples from the third class have been obtained by taking samples from the second class and adding a specific pattern in a random location.

What features should I add to a traditional CNN classifier in order to improve the performances, considering that I have a very large dataset (>15K images per class)? Could it be useful to pre-train the network with the two similar classes and then adding the other one?


texts: Polynomial regression vs. multilayer perceptron

texts: Polynomial regression and multilayer perceptrons have different structures and different learning procedures. What are these two algorithms pros and cons? Are there some situations where one should perform better than the other?


texts: Overfitting with sklearn pipeline - reasons why?

texts:
So....

I've been playing around with this for FAR TOOO LONG now and I really need some advice.

Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems to provide good results when scoring but i believe this to be data leakage and an incorrect process to perform in real world solutions.

I've kept apart 15% of data as a holdoutset and the rest for CV.

The issue I have is that if I put the scaler into a pipeline, the scale changes on each fold of the CV and gives me wildly different results to if I prescaled the whole dataset prior to hyper parameter tuning.

I'm even getting worse results once I fit to the whole training data (including the holdout set) which suggests its overfitting to the smaller data set.

Is there anything I can do to combat this?

Am I doing something wrong? (should I be putting a scaler into a pipeline?)

Or is this just what it is in the real world?

Any other help regarding overfitting etc would be amazing.


texts: Why does GridSearchCV (sklearn) change the value of n_samples?

texts:
I thought n_samples is the number of training examples. But when using GridSearchCV, n_samples becomes 32 rather than 50.

Error when using GridSearchCV:

  Expected n_neighbors <= n_samples,  but n_samples = 32, n_neighbors =
  50

Training examples:

print(X_train.shape[0]) => 50

print(len(y_train)) => 50

This works:

neigh = KNeighborsClassifier(n_neighbors=50)
neigh.fit(X_train, y_train)
result = neigh.predict(X_test)

This fails:

from sklearn.model_selection import GridSearchCV

grid_params = {
    "n_neighbors" : [50]
}

g = GridSearchCV(KNeighborsClassifier(), grid_params)
g.fit(X_train, y_train)

I'm confused why n_samples becomes 32 when using GridSearchCV.


texts: How to feed output of predict value back into the input using LSTM in python

texts:
The inputs here are the 3. The output here (LSTM) is the probabilities that the next x1 input ought to be.
Means here I have x1x2 and x3 input values. 1st three inputs LSTM output1 and then next if x1 value = 0  then Lstm output1 is back into as input and predict next output2. If this output 2 value is equal to next x1 value then back into as input and predict output 3. If not equal not to take output 2 as x1 input and take it as mentioned x1 input. The output (Yt) at timestep t depends on the input X1t and on the previous output Yt-1.
as a example

x1   x2    x3    predict (output)
100  30    40     120
 0   20    10     130
140  15    30     160

here second value of x1 column is 0, x1 value = 0 then take the value as output 1

x1 = output1

here x1 column 3rd value measured and output 2 value is not equal to x1 3rd value.
 then take input as measured value.
So this is the method that I want to do. But I don't know how to write it. Can anyone helps me to do it?
my LSTM code:

fit1 = Sequential ()
fit1.add(LSTM(32, return_sequences=True, activation='relu',input_shape=(3,1)))
fit1.add((LSTM(32, return_sequences=True)))
fit1.add(LSTM(32))
fit1.add(Dense(1))
batchsize = 3
fit1.compile(loss="mean_squared_error",optimizer="adam")
fit1.fit(x_train , y_train , batch_size = batchsize, nb_epoch =10,shuffle=True)
pred1=fit1.predict(x_test)
for i in range(len(x)):
    pred1.append(fit1.predict([x[i,None,:],pred1[i]]))
pred1 = np.asarray(pred1)
pred1 = pred1.reshape(pred1.shape[0],pred1.shape[2])

But this is not working. not having relationship inbetween input and output.

error:

Another example:
 here I have three inputs data x1,x2,x3 data with time series. I want to predict value in every one hour. (at t+1)
I have a y column that I got value in every one hour(t+1). But sometime i measured value after two hours. So in between past time and now time there is a value not measured at t+1. So I will predict value that I didn't measured at t+1. Here x1 value is depending on the output value(y) at t+1 . When I predict the value at that time period take it as y1(t+1), that value should have to read it as x1 value as x1(t) to predict the next output value at t+1 (y2). If I measured that value at t+1 ,if my prediction value == measured value then read it that value as x1(t) to predict the next value at t+1. This is the process that I want to write it as a code.
here example in :

time     x1(t)     x2(t)     x3(t)     y(t+1)
6:00:00  120        0         0         110 (I measured it at t+1)

when I predict it using LSTM, if that value is == measured value(t+1) read it as a second input value of x1(t) column. if not equal read the measured value as second input value of x1(t). SO

7:00:00  110       40         10         0  (not measured value at t+1)

Then I predict the value at t+1 =y2 , assume it came 70 then that y2(t+1) value will be the third input of x1 column as x1(t). So

8:00:00  70        0          30         200 (I measured value at t+1)

this is the process that I want to run it using LSTM.


texts: how can i compare two classification algorithms?

texts:
all,
i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. clearly they are very close! how can i assess if they are statistically different?
i have used mcnemars test:
from mlxtend.evaluate import mcnemar_table
from mlxtend.evaluate import mcnemar
lgbm_pred = second_best.predict(x_test)
xg_pred = chosen_model.predict(x_test)

tb = mcnemar_table(y_target=y_test,
                   y_model1=lgbm_pred,
                   y_model2=xg_pred)

chi2, p = mcnemar(ary=tb, corrected=True)
print('chi-squared:', chi2)
print('p-value:', p)

output is: chi-squared: 2.25 p-value: 0.13361440253771584, so i would not reject null that models peformance are equal. (hopefully i am using this correctly so pls let me know if i am not.)
i have seen some threads on using 'permuation tests' etc.. but i am unsure how to interpret these and also i thought these tests were only if you have small sample size which i don't.
https://stackoverflow.com/questions/52373318/how-to-compare-roc-auc-scores-of-different-binary-classifiers-and-assess-statist
basically how can i assess which classifier being better? when in an ideal world i would want a classifer which is able to predict has cancer. can i compare precisions of model to model? what is best approach - does anyone also know how i can use permutation tests?


texts: How to convert a SQLContext Dataframe to RDD of vectors in Python?

texts:
I have a SQLContext data frame derived from pandas data frame consisting of several numerical columns. I want to perform multivariate statistical analysis using the pyspark.mllib.stats package. The statistics function expects a RDD of vectors. I could not convert this data frame into RDD of vectors. Is there a way to convert the data frame?

Code:

 rdd = sqlCtx.createDataFrame(df_new)
 summary = Statistics.colStats(rdd)

I am getting df_new from

 df_new = df.applymap(lambda s: dic.get(s) if s in dic else s) #df is a pandas dataframe

I am getting a PY4JJava error at the summary line. The issue is with the format of rdd.


texts: Plotting estimates of Tail Dependence Coefficient in R

texts:
I want to plot the following estimate  of the tail dependence coefficient:

$\hat{\lambda}_n=\frac{1}{k}\sum_{j=1}^nI_{\{X_j>X_{(n-k)},Y_j>X_{(n-k)}\}}$
where $I_{(.)}$ is an indicator function and $X_{(1)}>...>X_{(n)}$ are order statistics.
If I consider this example

n=1000;
alpha=4;
U=runif(n);
phi=0.8;
sigma=0.1;
X=(1-U)^{(-1/alpha)} ;
Z=rnorm(n)      ;
Y=phi*X+sigma*abs(Z) ;

then $\lambda=(.8)^4=04.09$. When I plot $\hat{\lambda}_n$ vs $k$ then the graph seems to stabilize around $0.50$ which is not true. See below the graph.

n=length(X);
lambda_hat=c();

lambda_hat=sapply(1:(n-1), FUN = function(i,minXY=pmin(X,Y),Xsort=sort(X)) length(minXY[minXY>Xsort[floor(n-i)]])/i);
k=1:(n-1)

plot(k,lambda_hat,type = "l",lwd=2,col="blue",
     main="Tdc of X and Y",ylab="Estimates",xlab="Order statistics")


texts:
I'm looking at pybrain for taking server monitor alarms and determining the root cause of a problem. I'm happy with training it using supervised learning and curating the training data sets. The data is structured something like this:

Server Type A #1

Alarm type 1
Alarm type 2

Server Type A #2

Alarm type 1
Alarm type 2

Server Type B #1

Alarm type 99
Alarm type 2

So there are n servers, with x alarms that can be UP or DOWN. Both n and x are variable.

If Server A1 has alarm 1 & 2 as DOWN, then we can say that service a is down on that server and is the cause of the problem.

If alarm 1 is down on all servers, then we can say that service a is the cause.

There can potentially be multiple options for the cause, so straight classification doesn't seem appropriate.

I would also like to tie later sources of data to the net. Such as just scripts that ping some external service.

All the appropriate alarms may not be triggered at once, due to serial service checks, so it can start with one server down and then another server down 5 minutes later.

I'm trying to do some basic stuff at first:

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet
from pybrain.supervised.trainers import BackpropTrainer

INPUTS = 2
OUTPUTS = 1

# Build network

# 2 inputs, 3 hidden, 1 output neurons
net = buildNetwork(INPUTS, 3, OUTPUTS)

# Build dataset

# Dataset with 2 inputs and 1 output
ds = SupervisedDataSet(INPUTS, OUTPUTS)

# Add one sample, iterable of inputs and iterable of outputs
ds.addSample((0, 0), (0,))

# Train the network with the dataset
trainer = BackpropTrainer(net, ds)

# Train 1000 epochs
for x in xrange(10):
    trainer.train()

# Train infinite epochs until the error rate is low
trainer.trainUntilConvergence()

# Run an input over the network
result = net.activate([2, 1])

But I[m having a hard time mapping variable numbers of alarms to static numbers of inputs. For example, if we add an alarm to a server, or add a server, the whole net needs to be rebuilt. If that is something that needs to be done, I can do it, but want to know if there's a better way.

Another option I'm trying to think of, is have a different net for each type of server, but I don't see how I can draw an environment-wide conclusion, since it will just make evaluations on a single host, instead of all hosts at once.

Which type of algorithm should I use and how do I map the dataset to draw environment-wide conclusions as a whole with variable inputs?


texts: Giving more weight to a particular feature in scikit-learn decision trees

texts:
I have a model that I train on same data, but i want a feature to have a stronger weight.

Say I have three features:

Car manufacturer's name
Price
Top speed

and I want to classify my cars as "Sport", "Luxury", "Family", "Compact" using these indicators. However, say that I realise the manufacturer's name is the strongest indicator, so I want that to weigh more. For example, if I buy a Ferrari, that is a sport car, even though the price tag may make it look like a luxury car. So, while I want to use all features, I want one of them to have a stronger weight (this is just a silly example, my actual problem is different, but this gives an idea). If a manufacturer makes all 4 kinds, speed and price are important, otherwise the name should have a stronger weight than the price and top speed. Basically, my tree should really look at the manufacturer's name and then split depending on the other features.

Can I set up sklearn in python to do this? How?


texts: So for each user in a file I have their average power usage value every hour for 40 consecutive days. I need to predict their power usage during next 10 consecutive days. I am new to RNNs (I have experience with convolutional and regular NNs) and I am asking for some advice. How can I represent these timevalues and pass them to RNN? Because passing them as numbers from 1 to 40*24 wouldn't work. Stuff like day of the week would be bad as well (there might be cyclic patterns).


texts: Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)

texts:
I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest to either combine or seperate regularization methods.
In my case I have implemented/integrated both approaches (combined and separate). For which I have seen promising results when actually combining as it has helped me not to always overfit my models entirely while generally improving the r2 score of my model.
Question:
Is it preferable to combine L1/L2 Regularization with Dropout Layer, or is it better to use them separately?
Example Code:
def model_build(x_train):
    # Define Inputs for ANN
    input_layer = Input(shape = (x_train.shape[1],), name = "Input")
    #Create Hidden ANN Layers
    dense_layer = BatchNormalization(name = "Normalization")(input_layer)
    dense_layer = Dense(128, name = "First_Layer", activation = 'relu', kernel_regularizer=regularizers.l1(0.01))(dense_layer)
    #dense_layer = Dropout(0.08)(dense_layer)
    dense_layer = Dense(128, name = "Second_Layer", activation = 'relu',  kernel_regularizer=regularizers.l1(0.00))(dense_layer)
    #dense_layer = Dropout(0.05)(dense_layer)

    #Apply Output Layers
    output = Dense(1, name = "Output")(dense_layer)

    # Create an Interpretation Model (Accepts the inputs from branch and has single output)
    model = Model(inputs = input_layer, outputs = output)

    # Compile the Model
    model.compile(loss='mse', optimizer = Adam(lr = 0.01), metrics = ['mse'])
    #model.compile(loss='mse', optimizer=AdaBound(lr=0.001, final_lr=0.1), metrics = ['mse'])


texts: Decision Trees - how does split for categorical features happen?

texts:
A decision tree, while performing recursive binary splitting, selects an independent variable (say $X_j$) and a threshold (say $t$) such that the predictor space is split into regions {$X|X_j < t$} and {$X|X_j >= t$}, and which leads to greatest reduction in cost function.

Now let us suppose that we have a variable with categorical values in {$X$}. Suppose we have label-encoded it and its values are in the range 0 to 9 (10 categories).

If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits?
Should it rather perform the split based on == and != for this variable? But then, how will the algorithm know that it is a categorical feature?
Also, will one-hot encoded values make more sense in this case?


texts: Effect of weights on the Louvain communities detected

texts: The Louvain method for community detection aims to optimize modularity and hence detect communities in the given graph. In case of a weighted graph would it be valid to assume that an edge with a higher weight would be prioritized to be part of a community as against an edge with a lower weight while optimizing modularity ?


texts: Proof of the connection between V and Q in Reinforcement Learning

texts:
I've been studying some basics RL in the SpinningUp materials.
Is there any mathematical proof that $V^\pi(s) = E_{a \sim \pi} [Q^\pi(s, a)|s_0 = s]$ ?


texts: How to interpret continuous variables in a decision tree model?

texts:
After fitting a decision tree with some continuous variable, how do I interpret the effect that variable has on the target?

For example I'm predicting target Y. From sklearn random forest or Xgboost I can find out that the feature X is important. How do I determine if feature X's correlation to Y is positive or negative?


texts: Is there a way to implement nested features in unsupervised models?

texts:
Our project has built an unsupervised model that uses data about a number of companies. Some of these companies are public and some are private. The ones that are public have much higher financial reporting requirements than the private companies which means I have a lot more information about the public companies than I do the private ones.
Because the overall data we are working with has a large number of features, we don't build our model from direct data values, but rather we compute a set of principal components and then use those components as features. I would like to use the available financial data to inform my unsupervised model, but have so far had to exclude it because of the "missing" (i.e. unavailable) data from the private companies.
What I think I need is a way to incorporate a "nested" feature such that if the company is public (1), its financial data is utilized but if it is private (0) no financial data is utilized. This would create a reduced dimensional space just for the private companies instead of the entire dataset.
Are there any references to papers or examples of how to do something like this?


texts: approach for multi label text classification

texts:
I want to make a classifier that will label each text in a corpus with the correct label(s).
I can go straight to ML using sklearn multi-label text classification, or even to DL using LSTM.
But is it not better to start simple, and first use a rule-based system. That will help me understand the problem, and also set a benchmark accuracy score. Then I can make my algorithm more sophisticated (ML, DL) gradually in ways that only help the precision and recall.


texts:
Let's say I want to train a neural network to predict whether some state is 'good' or 'bad'. We can think of it as a value function for a board game. So a natural choice would be to place one neuron in an output layer with hyperbolic tangent function activation. Then I need to construct a dataset where for each 'good' position the target value equals  1 and -1 otherwise and train with mean squared error loss function.

So I chose target values in dataset that match my activation function's image. From what I've read this is a common approach. But what if I assign some states greater or lower values than 1 and -1 respectively? Does it make any sense? If so, can we say anything theoretically how such change would affect training?


texts: Why I get a very low accuracy with LSTM and pretrained word2vec?

texts:
I'm working on a reviews classification model with only two categories 0 (negative) and 1 (positive). I'm using pre-trained word2vec from google with LSTM. The problem is I get an accuracy of around 50% where it should be around 83% according to this paper. I tried many different hyperparameters combination and still gets a horrible accuracy. I also tried to change the data preprocessing techniques and tried stemming but it hasn't resolved the problem

here's my code

X, y = read_data()
X = np.array(clean_text(X)) #apply data preprocessing
tokenizer = Tokenizer()
tokenizer.fit_on_texts(X)

#converts text to sequence and add padding zeros
sequence = tokenizer.texts_to_sequences(X)
X_data = pad_sequences(sequence, maxlen = length, padding = 'post')

X_train, X_val, y_train, y_val = train_test_split(X_data, y, test_size = 0.2)

#Load the word2vec model
word2vec = KeyedVectors.load_word2vec_format(EMBEDDING_FILE, binary=True)

word_index = tokenizer.word_index
nb_words = min(MAX_NB_WORDS, len(word_index))+1

embedding_matrix = np.zeros((nb_words, EMBEDDING_DIM))
null_words = []
for word, i in word_index.items():
    if word in word2vec.wv.vocab:
        embedding_matrix[i] = word2vec.word_vec(word)
    else:
        null_words.append(word)

embedding_layer = Embedding(embedding_matrix.shape[0], # or len(word_index) + 1
                            embedding_matrix.shape[1], # or EMBEDDING_DIM,
                            weights=[embedding_matrix],
                            input_length=701,
                            trainable=False)

model = Sequential()
model.add(embedding_layer)
model.add(LSTM(100))
model.add(Dropout(0.4))
model.add(Dense(2, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

I also tried other optimizers like AdaMax and MSLE loss function. I'm just confused if the problem isn't with the model and preprocessing where could it be? Thanks


texts: Grouping similar rows to detect duplicates

texts:
I have a dataset containing all the real estate ads for sale in the process of publishing a city:

    ID  URL CRAWL_SOURCE    PROPERTY_TYPE   NEW_BUILD   DESCRIPTION IMAGES  SURFACE LAND_SURFACE    BALCONY_SURFACE ... DEALER_NAME DEALER_TYPE CITY_ID CITY    ZIP_CODE    DEPT_CODE   PUBLICATION_START_DATE  PUBLICATION_END_DATE    LAST_CRAWL_DATE LAST_PRICE_DECREASE_DATE
0   22c05930-0eb5-11e7-b53d-bbead8ba43fe    http://www.avendrealouer.fr/location/levallois...   A_VENDRE_A_LOUER    APARTMENT   False   Au rez de chaussÃ©e d'un bel immeuble rÃ©cent,...   ["https://cf-medias.avendrealouer.fr/image/_87...   72.0    NaN NaN ... Lamirand Et Associes    AGENCY  54178039    Levallois-Perret    92300.0 92  2017-03-22T04:07:56.095 NaN 2017-04-21T18:52:35.733 NaN
1   8d092fa0-bb99-11e8-a7c9-852783b5a69d    https://www.bienici.com/annonce/ag440414-16547...   BIEN_ICI    APARTMENT   False   Je vous propose un appartement dans la rue Col...   ["http://photos.ubiflow.net/440414/165474561/p...   48.0    NaN NaN ... Proprietes Privees  MANDATARY   54178039    Levallois-Perret    92300.0 92  2018-09-18T11:04:44.461 NaN 2019-06-06T10:08:10.89  2018-09-25

I would like to group real estate ads for the same property. Indeed, several professionals (see individual) publish ads on several real estate portals for the same property.

ID  URL CRAWL_SOURCE    PROPERTY_TYPE   NEW_BUILD   DESCRIPTION IMAGES  SURFACE LAND_SURFACE    BALCONY_SURFACE ... DEALER_NAME DEALER_TYPE CITY_ID CITY    ZIP_CODE    DEPT_CODE   PUBLICATION_START_DATE  PUBLICATION_END_DATE    LAST_CRAWL_DATE LAST_PRICE_DECREASE_DATE

What methodology could I use to detect duplicates are the rows aren't ? So far I thought eitehr of checking if the description was exactly the same (but I would need to remove special characters before) or if the images on the url are the same (but I am not a star in image processing)

To my mind, as far as people that post the same houses does it on different website these won't be duplicates on the same website. Probably people post the same DESCRIPTION or the same IMAGES. The houses surely have the same surface however we may have several dealer for a same house. So I did:

# Let's add a new boolean column to our DataFrame that will identify a duplicated order line item (False=Not a duplicate; True=Duplicate)
df['is_duplicated'] = df.duplicated(['DESCRIPTION'])

And did the sum:

# We can sum on a boolean column to get a count of duplicate order line items
df['is_duplicated'].sum()

Which returned 249. I don't know how to compare the images yet.

Is there a better strategy?


texts: How to construct a graph using Neural Structured Learning framework?

texts:
My dataset (both features and label) consists of continuous variables. Dimension of features is (12,). The number of samples are of 7th order of magnitude (about 11 million).
So in terms of a simple feed-forward neural network input_shape is (12,) and number of units of output layer is 1 using a ReLU activation.
I am currently using Adversarial Regularization in Neural Structured Learning framework to train my network which simply works out of the box if you don't have an explicit relation between your variables.
However what I would like to do is to use Graph Regularization. I'm trying to follow the given instructions from here (https://www.tensorflow.org/neural_structured_learning/tutorials/graph_keras_lstm_imdb) although the given problem is different, while mine is a regression one. Therefore, I would need to construct a graph. In order to do this, as far as I understand, I would need to create sample embeddings and then compare them in order to create the graph.
All information I have found up to now refer to word embeddings and, generally, classification problems. Converting my regression to classification problem by e.g. binning my data is not an option since every single value is important.
My main question is how to create these embeddings in my regression problem?
Thank you