texts: Backprop: backward pass way faster than forward pass

texts:
I started to work with my own implementation of backpropagation algorithm, that I made five years ago. For each training sample (input-output pair), I make a forward pass (to compute outputs of each neuron), backward pass (to compute "Deltas" for each neuron) and then, I update the weights.

When there are 3 layers of neurons: input layer, hidden layer, output layer, there are "two layers of weights". In a forward pass, I need to go through all weights (both layers). But in a backward pass, I omit the first layer of weights (that goes between the input layer and the first inner layer).

Is seems to be wrong, as I thought a backward pass requires going through all weights. But it seems to work quite well. So is it true, that not all weights are used in a backward pass?

When I use it for 20x20px images (layers 400 : 60 : 10), a backward pass is 40x faster than a forward pass, as it processes only 60x10 weights, omitting 60x400 weights.


texts: Keras.engine.topolgy import Node equivalent in tensorflow 2.X

texts:
I am trying to convert an old code written in Python 2 and tensorflow 1.x into the new tensorflow 2.X and python 3.7.9. There is a line of code
from tensorflow.keras.engine.topology import Node

which is giving an error
ModuleNotFoundError: No module named 'tensorflow.keras.engine

Any suggestions how to fix this would be appreciated.


texts:
I wondered if someone could help with a re-labelling 'fill' variables ' and the x-axis on a stacked bar chart please?
My ambition is to:

Express the fill variable(s) as of six domains in the health survey, titled ' MR, Mob, SC, Act, Pain, Anx')
And have the x-variable representing the 'health states' (0,1, 2, 3,4, 5), rather than X0, X1....X5

library(ggplot2)
library(dplyr)
library(reshape2)
library(viridis)
library(hrbthemes)
library(readxl)

data <- read.table(text = "0    1   2   3   4   5
MR  155 211 64  14  1   1
Mob 0   393 51  2   0   0
SC  0   427 12  7   0   0
Act 0   386 45  15  0   0
Pain    0   379 62  5   0   0
Anx 0   355 73  18  0   0", header = TRUE)

data$row <- seq_len(nrow(data))
data2 <- melt(data, id.vars ="row")

ggplot(data2, aes(x = variable, y = value, fill = row)) +
  geom_bar(stat = "identity") +
  xlab("\nHealthState") +
  ylab("Count\n") +
  theme_bw()

Would appreciate all feedback.


texts: Reducing the dependency between variables

texts:
I am trying to perform a multi linear regression model:

$$y_i = β_0 + β_1x_{i1} + β_2x_{i2} +... + β_px_{ip} + ε_i$$

where $$x_{i1}, x_{i2}, ..., x_{ip}$$ are highly correlated with each other (VIFs can be as low as 5 and high as 10).

I am just wondering if there exists a procedure with the following properties:

1) reduces the collinearity of the variables (e.g. VIFs should be lower than 5 after the procedure)

2) the variables after the procedure should maintain the original meanings/interpretations.. (so PCA and FA are out).

3) not dropping any of the variables. I should have all p original varaibles.. (So LASSO and RIDGE are out)


texts: Affect of multi-Col linearity in Machine Learning Algorithms

texts:
This is my general understanding on the affect of Multi-Col linearity on Linear and Logistic regression. However I am still not able to understand whether multi-col linearity is a problem in decision tree (Classification and regression).

Linear Regression - It has a huge impact as the co-efficient might vary drastically if there are 2 variables with high correlation. And regularisation would overcome this issue.
Logistic Regression - Same as Linear regression.

Can anybody please give me some insight on how it affects in decision tree classifier and decision tree regressor. And how to overcome this problem if it affects.


texts: Including repeating features into rnn input vector

texts:
My question is - will recurrent network such as lstm or recursive network learn anything from features that are being repeated over multiple timesteps.

So for example I have a charachter based model for text generation, and besides the actual letter I want to include the meta information about the word that this character belongs to - like part of speech role. If would to concatenate a spared vector representing this information to each letter vector of each word.

Thank you.


texts: Feed-forward neural network not training with Keras function generators deep_learning data_science machine_learning python

texts:
I am using feed-forward neural network for a classification task. My data is 1 million examples for 9 classes (imbalanced). Due to memory constraint in Keras, I used function generator to generate batches automatically with a batch size of 200. I trained a simple model with 3 hidden layers with ReLU activation function. Input layer is 39 dimensional MFCCs and output is 9 classes. This model worked fine when I used a subset of this huge data (Sure!) but now, while using function generator i.e model.fit_generator, I see that training accuracy is just wandering around and validation accuracy is just too low. Looks like the model is not learning at all. What might be the possible reasons for this behaviour?

about data

Data : Speech

subset of huge data mentioned was totally clean but I generated 1 million examples from 1300 examples using data augmentation techniques like equalization, time stretch, time compression, noises, reverb etc


texts: Computing number of batches in one epoch

texts:
I have been reading through Stanford's code examples for their Deep Learning course, and I see that they have computed num_steps = (params.train_size + params.batch_size - 1) // params.batch_size [github link].

Why isn't it num_steps = params.train_size // params.batch_size instead?


texts:
I wish to better understand the difference between lazy and eager learning. I am having difficulty conceptualising what the "abstraction" refers to between the two.

According to the text book I am reading it says, "The distinction between easy learners and lazy learners is based on when the algorithm abstracts from the data."

A lazy learner delays abstracting from the data until it is asked to make a prediction while an eager learner abstracts away from the data during training and uses this abstraction to make predictions rather than directly compare queries with instances in the dataset.

I understand that KNN algorithm loads all the data into memory so depending on the size of the dataset, the computing requirements could be enormous as well as the fact it's a non-parametric algorithm so there is no training to find parameters, the classification is done on a per query requirement??


texts: Purpose of gamma multiplier in gradient boosting

texts:
looking through the mathematics of gradient boosting on the relevant wikipedia page, intuitively what is the purpose of the multiplier $\gamma_i$?

This term does not appear in the following definition of $F_m(x)$

But confusingly, then appers in the subsequent definition:

Any advice appreciated, thanks.


texts: How can we decompose generalization gap as done in the paper "Generalization in Deep Learning"?

texts:
I am reading a recent paper "Generalization in Deep Learning" and I am unable to understand a step. In this step, they first take neural network as a direct acyclic graph(DAG) and described output of neural network as follows,

and then they decomposed generalization gap as follows,

without any explanation. Here z is,

Can someone explain this or point me towards the right direction.


texts: Learning Football Player Stats like FIFA's by only the game result

texts:
It is a general question on how to learning representation of one entity but the dataset is mixed with a lot of other entities, which their statis are always waiting to be learnt.

The question is best be explained by an example. Let's say, the entity is a football player. How can we assign player stats to them by just learning from the match result.

Dataset: Football match data with startup line of the two teams and the final result score.

Player stats, something like this https://www.fifaindex.com/player/158023/lionel-messi/ but much simpler. We can define a player stats is just 3D vector v = (x , y , z) where x , y , z is between 0 to 1.0.  We don't need to know what actually x mean shooting power or anything.

So after learning, It output each player a v vector assigned.

Any idea or some material that you can suggest?


texts: Which machine learning algorithms are hard real time capable?

texts: Are there any machine learning algorithms or algorithms categories which have predictable execution times (guaranteed minimal/maximal execution time e.g. when the algorithm is run on a single processor which is not loaded with other processing tasks) for predictions given any possible input making them suitable for the use in hard real time systems?


texts:
I'm making a ad-sales predictor and try to make it fit by using the S curve's below.

$ ()= \frac{}{1+} (1−^{−}) $

$f(x)$ is an ad-sales
$x$ is an advertise cost
both $a$ and $b$ are fitting parameters.

It looks like logistic function curve. I actually don't know a name of it though I have to use it for our client order. Do you know a name of the function or any text referring it?


texts: How to analyze predictions which seem to be in 0-255 range

texts:
I'm using this TFLite model on my Raspberry 4. I was using the normal version on my laptop but after having to deploy it on Raspberry I figured out the Lite version suited my project better.
When on laptop using the normal version of the model it returned me an array of probabilities whose sum was equal to 1. But when using the lite version, it returns an array of integers which make not much sense to me at first sight, except that they seem to be in [0, 255] range which is the input range (rgb images).
See for yourself (format: best value / sum of all values):
image1 -> 129/215
image2 -> 154/235
image3 -> 234/243
image4 -> 123/226
image5 -> 212/237
image6 -> 133/211
image7 -> 247/252
image8 -> 133/196
image9 -> 105/210
When computing those probabilies, I get somewhat identical probabilities of those I had on my laptop for the same set of images. When I divide the best value by 255 I almost get the same values as I did on my laptop using the normal version. But I'd like to understand what happens behind the scenes instead of just stupidly divide by 255.


texts:
I would like to run one-way ANOVA test on my data. I saw that one of several assumptions for one-way ANOVA  is that there needs to be homogeneity of variances. I have run the test for different data-sets. I find sometimes my p-values are larger than 0.05 and for some datasets it is smaller.
As I understand, if the p-value is smaller than 0.05, then I can reject the null hypothesis and tell that the variances are not equal (and then it makes sense to use ANOVA?)
The problem is that I see in different sources diferent conclusions: in some they run ANOVA if  they reject the null hypothesis, for some they run ANOVA if they cannot reject the null hypotheis. For example,
here it says that if is smaller we can run the ANOVA:

But here it recommended that p-value>0.05
To me, intuitively, it sounds like p-value should be greater than 0.05 but I would like to get clear answer regarding  that.


texts: How to start the process of coming up with the predicted math score?

texts:
I am working on a problem(non competition) from hacker rank https://www.hackerrank.com/challenges/predict-missing-grade

Basically you're given test data of a bunch of students of their scores in other subjects but math and you are to predict their score in math based off all their other test scores. Say you were passed data of

{"SerialNumber":1,"English":1,"Physics":2,"Chemistry":3,"ComputerScience":2}

How would you go about generating that student's score in mathematics or coming up with a prediction engine to generate the math score? I know that's the whole point of this question but can someone give me a hint or a resource to go to so I can have a chance of figuring this out and actually get started? I really want to learn.


texts:
I’m facing some (RAM) memory issues to train a neural network.

I have an input array consisting of grayscale images encoded as the type numpy.uint8 (therefore the whole range 0-255 can be covered). When feeding the data to the network, I’m supposed to normalize the values into the 0-1 range, making them now floating points (numpy.float64 or 32). However, this conversion makes my data 8x(or 4x) bigger, which my RAM memory can’t handle.

Would there be a way I could overcome this issue?

Thanks !


texts:
I've got a regression problem where a model is required to predict a value in the range [0, 1].

I've tried to look at the distribution of the data and and it seems that there are more examples with a low value label ([0, 0.2]) than higher value labels ([0.2, 1]).

When I try to train the model using the MAE metric, the model converges to a state where it has a very low loss, but it seems that the model has converged to a state in which it predicts a low value on many of the high value label examples.

So my assumption was that the data is imbalanced and I should try to weight the loss of the examples depending on their label.

Question: what is the best way to weight the loss in this configuration?

Should I weight each example by the value of its label using some function f(x) , where f(x) is low when x is low and high when x is high?

Or should I split the label values into bins ([0, 0.1), [0.1, 0.2) ... [0.9, 1]) and weight each bin (similarly to categorical loss weight)?


texts: Automating scoring of answers for a given question

texts:
There is a topic 'X' on which students are asked to write an English passage. Given a question, I have multiple solutions with their scorings out of 10.
For a given answer by a new candidate, one needs to read the passage and score the answer. The main focus of scoring is vocablury and not grammer.
How can I automate this process? What should be the steps?
Should I like train a simple model with answers as input and their scores as output. Then for a random answer I can check what is the score given my the model trained. But I feel like this will be very basic and it's accuracy will be very less.
I am new to machine learning. Any reading material on this..etc anything would be great help. Thanks.


texts: Contrasting logistic regression vs decision tree performance in specific example

texts:
I have a set of 10,000 integers, and another set of 100.
The integers in the first set are mapped to integers in the second set according to some rules (not mathematical rules, think of these values as codes for naming certain items, it is some categorical mapping).
The mapping is not necessarily 100 to 1, in some case I may have just 30 or so integers from the first set mapped to an integer in the second set, in other cases 300, but on average of course it is 100 to 1.
Using sklearn, I created a decision tree that was able to get over 99% accuracy, as I would expect.
When I tried logistic regression, though, accuracy was just 45%. The training sample is about 100,000 example, so, it should be enough to learn.
What is going on?
Is there something inherently different in the logistic regression method that I am missing?


texts: Generalise a CNN model for Question vs Spam . Techniques to generalise the model apart from more data

texts:
For my problem, every image which is not a picture of a question on a paper (either from text book or handwritten), is a spam. It means that each and every image in this world is a spam for my case except the pictures of textbook/handwritten questions. I have used ResNet50 with 47000 questions and 43000 spams images. For the spam questions, I have used Coco 2017 test set with 42K images. My model gave me an awesome val and train metrices prec,rec,acc, f1 of all greater than 0.99. But on the new images, it performed so poorly that it just gave me 0.6 F-1 score.
What should I do make a generalised model apart from using data augmentation and more data. And if I have to use more data, how much would be sufficient? I resized my images to 224,224. Do I need to use large images? Data collection is not a problem as I have millions of images and for the spams, I can use the Google Public  dataset of more than 70 Millions images. But it'll cost me a huge computational power and load.
What are the other methods that I can try out?


texts: How to properly implement a CausalImpact on a time series

texts:
I have a time series with the percentage of the subscriber based on the total users in my platform. Let's say I did an event that could, potentially, affected the figures. I could run a CausalImpact from Google Github and check the exact effect.

However, my question is. Should I apply the algorithm to the raw data of the time series? Or it would be better if I do some kind of decomposition before?


texts: Does it make sense that datetime encodes one-hot-vector like one-hot-encoding or something else like

texts:
I'm new to machine learning and deep learning. I've wanted to solve time series problem, which has data every single second. Plus, I've been doing research on word2vector and time series data lately. And one day, I've come up with an idea transforming sequence data like date time into one-hot-encoding?

    time
2017-11-01 00:00:01
2017-11-01 00:00:02
2017-11-01 00:00:03
2017-11-01 00:00:04
.
.
.

My idea has some limitations like below,

too high dimensions for learning (1day = 60* 60 * 24 = 86400(s))
unlimited time - time will be generated every moment even right now as well
difference between seconds is too small to learn

I want you to determine what I'm saying above about limitations. Plus, I'd like you to give me some idea to develop time series data into one-hot-vector for machine learning and deep learning? + what do you think about this idea?


texts: External GPU vs. internal GPU for machine learning

texts: What are the pros/cons of using external GPUs (e.g., connected through thunderbolt) vs. internal GPUs for machine learning?


texts: Which correlation test can be used on low count contigency table?

texts:
I need to find the correlation between the ratings given by two parties.

Let's say we have 2 coaches, namely A and B.

Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "Weak".
Coach B will rate the same 12 players.

If Coach A rated all the players as "Strong", and Coach B rated 10 players as "Strong" and 2 players as "Weak".

I have tried putting the results into a contingency table form and ran Somers D test. However the Somers D shows 0% which does not truly representing the correlation of the ratings. What test could we carry out to find the correlation between the ratings of the coaches?


texts: Analysis of passed/not passed students by enrolled year

texts:
I am analysing some data from my University about successful students through years. The dataset looks like as follows
Student     Passed        Enrolled_Year
    A             0                2016
    B             1                2008
    C             1                2008
    D             0                2012
    E             1                2007
    F             1                2019
    G             1                2006
    H             1                2006
    I             0                2012
    L             1                2019
    M             0                2018
    N             1                2008

I already plotted the frequency of passed/not passed through years, to see when students, who passed exams.
I would like to know if it makes sense to look at a possible correlation between Passed and Enrolled_Year, when the number of Passed students is small compared to those who not passed.
I would like to get some insights from my dataset:
https://drive.google.com/file/d/1tUgHoFdzq2vSLHaPzAiNQC5Xkrk2SmKX/view?usp=sharing and one of this was looking at the possible correlation.
I would appreciated it if you could let me know if it makes sense a such analysis with big difference among frequency.


texts: Why an eigenvector might be reasonable notion of centrality

texts:
Using adjacency matrix to represent a connection between node i and node j, 1 means connected and 0 means not connected.

Using eigenvector to represent the centrality means the more nodes with high centrality values a node is connected to, the higher that node's centrality value is.

Eigenvector itself is just a vector after applying a transformation, the result vector has the same direction or totally opposite direction as the original one.

I really can't see the relationship between those two properties.


texts: Prepare data for SVM, Is it valid to normalise the data before and after PCA dimension reduction

texts:
Is it valid to normalise a dataset, reduce dimensionality with PCA and then to normalise the reduced dimension data.    Assuming this is performed on training data, should the same PCA coefficients be used to reduce the dimension of the test data.  Should the same max and min normalisation values be used for the test and training data. I have included a simplified example of the code I am using which may describe I said better. Thanks in advance.

%% Prepare Training Data

% Normalise training data
mindata=min(TRAINDATA); maxdata=max(TRAINDATA);
TRAINDATA = ((TRAINDATA-repmat(mindata,[size(TRAINDATA,1),1]))./(repmat(maxdata,[size(TRAINDATA,1),1])-repmat(mindata,[size(TRAINDATA,1),1])) - 0.5 ) *2;

% Perform PCA
mTRAINDATA = mean(mean(TRAINDATA));
TRAINDATA = TRAINDATA - mTRAINDATA;
[Cpca,~,~,~,~]=princomp(TRAINDATA,'econ');
EigenRange = 1:2;
Cpca = Cpca(:,EigenRange);
TRAINDATA = TRAINDATA*Cpca;
TRAINDATA = TRAINDATA + mTRAINDATA;

% Normalise training data second time
mindata2=min(TRAINDATA); maxdata2=max(TRAINDATA);
TRAINDATA = ((TRAINDATA-repmat(mindata2,[size(TRAINDATA,1),1]))./(repmat(maxdata2,[size(TRAINDATA,1),1])-repmat(mindata2,[size(TRAINDATA,1),1])) - 0.5 ) *2;

%% Prepare Test Data

% Normalise using first normalisation values from training data
TESTDATA = ((TESTDATA-repmat(mindata,[size(TESTDATA,1),1]))./(repmat(maxdata,[size(TESTDATA,1),1])-repmat(mindata,[size(TESTDATA,1),1])) - 0.5 ) *2;

% Perform PCA
mTESTDATA = mean(mean(TESTDATA));
TESTDATA = TESTDATA - mTESTDATA;
TESTDATA = TESTDATA*Cpca;
TESTDATA = TESTDATA + mTRAINDATA;

% Normalise using second normalisation values from training data
TESTDATA = ((TESTDATA-repmat(mindata2,[size(TESTDATA,1),1]))./(repmat(maxdata2,[size(TESTDATA,1),1])-repmat(mindata2,[size(TESTDATA,1),1])) - 0.5 ) *2;


texts: Preparing for a Machine Learning Design Interview

texts:
I am not sure if this is a relevant post here but:

I made it to the final round for the Machine Learing Engineer position at Facebook. The final round interview is virtual (thanks to Corona) and will consist of:

2 - General Algorithmic Coding questions
1 - Machine Learning Design
1 - General Systems Design
1 - Career Background

Most of these I know what I need to do to prepare but the content given to me for the Machine Learning Design is scarce to say the least but typical example questions for these are:

Design a newsfeed ranking algorithm
Design local search ranking
Design if an old post would be good for "On this Day" memories feature

This is the key highlights of what is expected for the Machine Learning Design interview:

You should be able to describe the components of an end-to-end ML system, including but not
limited to; model development, evaluation, and deployment.

You should be able to use existing toolsets to model the problem and break down its
components.

Be ready to analyze your approach while understanding the tradeoffs between your design
decisions.

We expect you to have a good understanding of common ML tools/techniques, but we do not
expect you to know and memorize every ML algorithm out there.

Now, although I know Machine learning decently well (I would like to think so) and I do mainly do Machine/Deep learning at my curreny job. I have never designed a Machine learning system. Thus I am not sure how to prepare for this particular portion of the interview and I have not really found much material online in terms of designing machine learning system.

Any advice would be greatly appreciated. My interview is set in the beginning week of April.


texts: accessing individual layers from a saved model

texts:
I am in the process of writing a model where I have the layers of the model in the "def _init_(   )" section of a class
I want to save and load the model in the class' variables where i ahve stored the layers .
and inside this class are two models where model 1 is using the layers of model 2 to compute a loss function .
how will this system work if i were to save the model ?
(this is my first attempt at a custom model)
as far as I can tell the ways to save the model seems to restore the weights and biases in the moded in the variable that was holding it and i dont see how it will reflect in the class I have written .
https://youtu.be/dCKbRCUyop8?t=1061
trying to replicate something like what is said here int his video at 17:41
class mymodel(Model):
    def __init__(self,chandim=-1):
        #just an example
        super(mymodel, self).__init__()
        self.gdn1 = Dense(128 * 16 * 16, activation='relu')
        self.gbn1 = BatchNormalization(momentum=0.9)
        self.glr1 = LeakyReLU(alpha=0.1)
        self.grs1 = Reshape((16, 16, 128))

        self.gcn2 = Conv2D(128, kernel_size=5, strides=1,padding='same')
        self.gbn2 = BatchNormalization(momentum=0.9)
        #self.gdp2 = Dropout(0.5)
        self.glr2 = LeakyReLU(alpha=0.1)

        self.gcn3 = Conv2DTranspose(128, 4, strides=2, padding='same')
        self.gbn3 = BatchNormalization(momentum=0.9)
        self.glr3 = LeakyReLU(alpha=0.1)

    def get_model1(self,input_layer):
        hid = self.gdn1(input_layer)
        hid = self.gbn1(hid)
        hid = self.glr1(hid)
        hid = self.grs1(hid)

        hid = self.gcn2(hid)
        hid = self.gbn2(hid)
        hid = self.glr2(hid)
        out = Activation("tanh")(hid)

        model = Model(input_layer, out)
        model.summary()
        return model, out

    def get_model2(self,input_layer):
        hid = self.gcn3(input_layer)
        hid = self.gbn3(hid)
        hid = self.glr3(hid)

        out = Activation("tanh")(hid)

        model = Model(input_layer, out)
        model.summary()
        return model

    def lossFn_model2(self,X,Y)
        bx0 = self.gdn1(X)
        bx1 = self.gbn1(hid)
        bx2 = self.glr1(hid)
        bx3 = self.grs1(hid)

        #Note Shared Layers

        by0 = self.gdn1(Y)
        by1 = self.gbn1(hid)
        by2 = self.glr1(hid)
        by3 = self.grs1(hid)

        return tf.math.square(bx3-by3)
```


texts: Orange Import and process spectra time dependent/kinetics

texts: I would like to import FTIR spectra that have been collected over time then use Orange to preprocess these as a group. I currently have the spectra as individual SPC or CSV files. How do I import the spectra then plot them as a stack? I have tried using the "Multi-file' widget but that does not work.


texts: How do you compare term counts between two different periods, with different underlying corpus sizes, without bias?

texts:
I'll set the question up with an example. You are analysing news coverage text data from 2014, and find that a term appears less often in the third quarter of 2014 than the final quarter (let's imagine it's the term "Christmas"). Unfortunately, there are also far less news articles in the third quarter than in the second (due to the lack of news in the summer). So how do we accurately compare the counts in each quarter? We assume that there will be a greater number of occurrences in the fourth quarter, but how much does the magnitude of this difference depend on the change in size of the underlying text?

Heap's law shows the relationship between text size and number of unique terms. It's non-linearity implies that the rate of new, unique words introduced by the text decreases as you increase the size of the text, and the proportion of the text taken up by each existing word therefore increases. This applies given documents taken from the same 'distribution' of text, in other words the underlying zipfian distribution of word ranks is identical (see wiki).

In my example above this is obviously not the case, since the underlying topics, and resultant term distribution, will change between summer and winter, especially with regards to the term "Christmas". But take the same term count but over the whole of 2013 and 2014; you would reasonably expect the general underlying term distribution to be the same in each period, so Heap's law applies, but what if the volume of text has changed? Simply normalising by the size of the text, or the number of documents, does not, as far as I can tell, account for the relative change in expected value of the term count.

I have a hunch that this might be a simple application of Heap's or Zipf's laws, but I can't see how to apply them to this particular question. Appreciate any insight.


texts:
Very new to Machine learning process.

I am curious as to how researchers, companies, academics get training data for their research.

Do they pay some users to train data. If yes, how do they guarantee accuracy of it.

Do graduate students (on PhD track) do it for their professor. (This is meant to be a joke)


texts: How Many Features Are Used in Random Forest When max_features is not an integer?

texts:
In the RandomForestClassifier function in sklearn.ensemble, one of the parameters is max_features, which controls the size of the random subset of features that are considered at each node splitting. I would think this needs to be an integer, since of course the size of a subset needs to be an integer.
But one of the options here is "log2", which sets max_features to log_2(n_features).
I was wondering how that works, considering log_2(n_features) will not be an integer unless n_features is a power of 2.
Thanks.


texts: What are some of the best practices for sharing data and models with colleagues?

texts: As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central server or location where all of us can access (which means unix permissions etc). For models I also tend to send a weights file over to my colleague and share my github. Both I've found pretty cumbersome. What have some of you done?


texts: Decision tree, how to understand or calculate the probability/confidence of prediction result

texts:
For example, a drug prediction problem using a decision tree. I trained the decision tree model and would like to predict using new data.

For example:

patient, Attr1, Attr2, Attr3, .., Label
002      90.0   8.0    98.0 ...   ? ===> predict drug A

How can I calculate the confidence or probability of the prediction result of drug A?


texts: Is correlation needed when building a model?

texts: Some papers report the correlation between features when building a model and some don't. Is there a need to check the correlation between features and target feature? It won't be easy if the number of features is high.


texts: Algorithm to determine a single output value based on multiple input values

texts:
The main challenge is the lack of data.
Input values come from tests results of patients. A patient takes a breath test at an interval during a timespan. The result values can range from 0 to ~200, and can be plotted for diagnose by a doctor based on the curve shape.
I am looking for an algorithm that takes the values at every interval and comes up with a single output value from 0 to 1 that indicates a fuzzy boolean to determine a preliminary diagnosis.
As mentioned the data is scarce, the current data available is a set of around 10 typical values of positive and negative test results.
I am looking for an algorithm or model that can rely on the available data.
I looked into Fuzzy Logic which could be an option despite I am not convinced.
What are your thoughts?


texts: When I visualize data using matplotlib it displays very well, but when using Plotly, the data display very bad.


texts: image classification with training set with 4 classes and test set with 3 classes

texts:
I have to do image classificaion with a CNN, and for doing this I have been given a  training set with 4 classes and a test set with 3 classes. I am really confused because I don't know if this is going to influence my prediction. It never happened to me.

How can I deal with this? Thanks in advance.


texts: cv2.error: OpenCV(3.4.3) (-215:Assertion failed) !empty() in function

texts:
Would anyone have a tip on how to fix this empty function error in OpenCV? I am attempting to follow the guides on OpenCV.org

The script will detect a face in an image and draw boxes around detections.

import numpy as np
import cv2 as cv

face_cascade = cv.CascadeClassifier('haarcascade_frontalface_alt.xml')

img = cv.imread('images/1.png')
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(gray, 1.3, 5)
for (x,y,w,h) in faces:
    cv.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)
    roi_gray = gray[y:y+h, x:x+w]
    roi_color = img[y:y+h, x:x+w]

cv.imshow('img',img)
cv.waitKey(0)
cv.destroyAllWindows()

I get an error:

>>>
=== RESTART: C:/Users/Desktop/distance-to-camera/selectHaar2.py ===
Traceback (most recent call last):
  File "C:/Users/Desktop/distance-to-camera/selectHaar2.py", line 11, in <module>
    faces = face_cascade.detectMultiScale(gray, 1.3, 5)
cv2.error: OpenCV(3.4.3) C:\projects\opencv-python\opencv\modules\objdetect\src\cascadedetect.cpp:1698: error: (-215:Assertion failed) !empty() in function 'cv::CascadeClassifier::detectMultiScale'


texts: Why does the votingClassifier in sklearn.ensemble gives higher accuracy than each of the classifiers that make it? and should i depend on it?

texts:
I am running an ML classifier on my data.
I used SVM, RF and KNN. I used GScv for each of them and then used votingclassifier.The accuracy i got in each classifier independently was low, but from the hard and soft vote of the voting classifier is much higher! Why is that?
Here is my code
Fitting Kernel SVM to training set
from sklearn.svm import SVC
clf = SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma=0.005, kernel='rbf',
  max_iter=-1, probability=True, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

Checking accurancy, Best score of GV
print('Checking accurancy, Best score of GV')
best_accuracy =  grid_search.best_score_
print (best_accuracy)

[out] Checking accurancy, Best score of GV
0.5169491525423728

Applying RandomForest (RF) Classification
from sklearn.ensemble import RandomForestClassifier

clf_rf = RandomForestClassifier(random_state=42)
clf_rf.fit(X_train, y_train)
clf_rf.score(X_test, y_test)
print ("Mean accuracy is", (clf_rf.score(X_test, y_test)))

y_pred2 = clf_rf.predict (X_test)

from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred2)

[out] accuracy 0.21

Applying KNN
from sklearn.neighbors import KNeighborsClassifier

neigh = KNeighborsClassifier(n_neighbors=10)
neigh.fit(X_train, y_train)

print (neigh.score(X_train, y_train))

[out] accuracy 0.58

Predicting the test set results
y_pred3 = neigh.predict (X_test)

from sklearn.ensemble import VotingClassifier

eclf1 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'hard')

eclf1 = eclf1.fit(X_train, y_train)

print 'Hard vote accuracy =', eclf1.score(X_train, y_train)

eclf2 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'soft')

eclf2 = eclf2.fit(X_train, y_train)

print 'Soft vote accuracy =', eclf2.score(X_train, y_train)

[out]
Hard vote accuracy = 0.9830508474576272
Soft vote accuracy = 0.7711864406779662


texts:
Which is more important - stable training results or good test results?

For instance, is obtaining an unstable training accuracy in different epochs, but good test accuracy better? Or is obtaining a stable training accuracy in different epochs, but test accuracy is not good.

Which one is the best choice?


texts: How to generate higher quality jpg plots on R

texts: I have tried generating higher quality of data visualization plots from RStats. I tried increasing frame dimensions but it's still max at about ~350kb. How do I generate higher quality images from RStats?


texts:
Huber loss formula is

$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ \delta (|a| - \frac{1}{2} \delta) && |a| > \delta\end{cases}$ where $a = y - f(x)$

As I read on Wikipedia, the motivation of Huber loss is to reduce the effects of outliers by exploiting the median-unbiased property of absolute loss function $L(a) = |a|$ while keeping the mean-unbiased property of squared loss function $L(a) = a^2$ within the typical region (i.e. region without outliers)

I don't know why don't Huber loss has the form:

$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ c |a| && |a| > \delta\end{cases}$ where $c$ is some constant which controls the error magnitude of outliers

or similar forms ?


texts:
(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified)

Suppose I've got a dataset which has 5 predictors - v1 - v5.
Since my dataset contains NA's, I've been imputing these values during training, which is fine.

However, if I get a new item to classify, it may contain 1 or more NA's. My initial thought was to create a model for each predictor variable; so for example, create a model where v2 - v5 predict v1, in the case that v1 is missing.

This would result in 5 additional models. One problem is that if v1 and v2 are missing, I'd need to guess both values somehow. I'm not sure how to do this. It makes me think that this isn't the best approach.

I'm a bit stuck, and any insight into this would be much appreciated :)


texts: Should I keep common stop-words when preprocessing for word embedding?

texts:
If I want to construct a word embedding by predicting a target word given context words, is it better to remove stop words or keep them?

the quick brown fox jumped over the lazy dog

or

quick brown fox jumped lazy dog

As a human, I feel like keeping the stop words makes it easier to understand even though they are superfluous.
So what about for a Neural Network?


texts: XGBOOST - different result between train_test_split and manually splitting

texts:
I am trying to train XGBOOST model.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=43, stratify=y)

when I'm using train_test_split and pass the model X_train, Y_train and for eval_set X_test, Y_test, The model seems to be a very good one.
CM example:

But when I manually split the Dataset :
splitValidationIndex = round(dataset.shape[0]*0.6)
splitTestIndex = round(dataset.shape[0]*0.8)
X_train = X[:splitValidationIndex]
y_train = y[:splitValidationIndex]

Pass it to fit
X_val = X[splitValidationIndex:splitTestIndex]
y_val = y[splitValidationIndex:splitTestIndex]

Pass it to eval_set
X_test = X[splitTestIndex:]
y_test = y[splitTestIndex:]

Check the model prediction on that
that produced a much worse model
example:

What am I missing/doing wrong?


texts:
I am looking to use only one class, person (along with BG, background),  for the Mask RCNN object detection. I am using this link: https://github.com/matterport/Mask_RCNN to run the mask rcnn. Is there a specific way to complete this (editing specific files, creating an extra python file, or just by filtering selections from the class_names array)?
Any direction or solution will be highly appreciate. Thank you


texts: Data leakage and predictive models: should we use past predictions as a feature?

texts:
I want to develop a Random Forest Classifier model to predict whether or not a customer will convert 7 days from today. The model is re-trained once a week and makes predictions for the following week. The features I use were created using 2 years worth customer behavior data.

Since I'm running this model and generating predictions once a week, I have been storing the conversion predictions for every customer on a weekly basis. Furthermore, I will know if my previous predictions were correct or not.

So let's say I'm training the model again today for the coming week and I want to include following features:

last_wk_predictions = Probability of Purchase predicted last week for each customer (between 0 and 1)
did_convert_last_wk = did they convert last week? (0 or 1)

Is there a specific name for this approach? Is it considered data leakage to include past predictions?


texts: Showing a graph with PCA selected features in Random Forest Model

texts:
So I created a Random Forest Model like so:
incr_pca = IncrementalPCA(n_components=40)
Xtrain_pca = incr_pca.fit_transform(X_train)
Xtest_pca = incr_pca.transform(X_test)

rfc = RandomForestClassifier(class_weight='balanced')
rfc.fit(Xtrain_pca,y_train)
y_pred_default = rfc.predict(Xtest_pca)

Then I plotted the 20 most important features like so:
feat_importances = pd.Series(rfc.feature_importances_)
feat_importances.nlargest(20).plot(kind='barh')
feat_importances

This gives me only the column indexes of the features, how can I show their names?


texts: Details of the k-means++ algorithm that is used to seed k-means

texts:
Regard to K-Means++ algorithm, which is an algorithm for choosing the initial values (or "seeds") for the k-means clustering algorithm.

K-Means++ algorithm in Wikipedia

The exact algorithm is as follows:

Choose one center uniformly at random from among the data points.
For each data point x, compute D(x), the distance between x and the nearest center that has already been chosen.
Choose one new data point at random as a new center, using a weighted probability distribution where a point x is chosen with probability proportional to D(x)2.
Repeat Steps 2 and 3 until k centers have been chosen.
Now that the initial centers have been chosen, proceed using standard k-means clustering.

I dont understand step 3

"Choose one new data point at random as a new center, using a weighted probability distribution where a point x is chosen with probability proportional to D(x)^2."

What is probability proportional  ?

If I do not misunderstand  . . .

The next centroid x I choose must the distance

D(x) = D(x)^2 / summation of all distances from all data points square

Is that right ?

I still wonder about implementation. I try this in java but it does not work , the chance is very low and it make the selection distort.

public static double euclidean(Data a, Data b) {

    double accumValue = 0;
    double res;

    for (int i = 0; i < 72; i++) {

        res = a.features[i] - b.features[i];

        res = Math.pow(res, 2);

        accumValue += res;

    }

    double finalRes = Math.sqrt(accumValue);

    return finalRes;

}

public static double accumeratedSqrDistanceCal(ArrayList<Data> dataList, ArrayList<Data> centroids) {

    double[] squareDistanceCollection = new double[dataList.size()];

    for (int i = 0; i < dataList.size(); i++) {

        double minDistance = minDistanceFromClosetCentroidsCal(dataList.get(i), centroids);

        squareDistanceCollection[i] = Math.pow(minDistance, 2);

    }

    double accumerateDistance = 0;

    for (int i = 0; i < dataList.size(); i++) {

        accumerateDistance += squareDistanceCollection[i];

    }

    return accumerateDistance;

}

public static double minDistanceFromClosetCentroidsCal(Data d, ArrayList<Data> centroids) {

    double minDistance = 100000;

    for (int i = 0; i < centroids.size(); i++) {

        double distance = euclidean(d, centroids.get(i));

        if (distance < minDistance) {

            minDistance = distance;
        }

    }

    return minDistance;

}
public static void main(String[] args) {

for (int i = 0; i < CENTROIDS_SIZE; i++) {

            for (int j = 0; j < dataList.size(); j++) {

                double accumerateDistance = accumeratedSqrDistanceCal(dataList, centroids);

                double rand = Math.random();

                double distance = minDistanceFromClosetCentroidsCal(dataList.get(j), centroids);

                double distanceSquare = Math.pow(distance, 2);

                double chance = distanceSquare / accumerateDistance;

                if (chance > rand) {

                    centroids.add(dataList.get(j));
                    break;

                }

            }
        }

}


texts: Reinforcement Learning for classification of a CSV file

texts:
I would like to try a Reinforcement Learning approach for a multi-label or binary classification of a CSV file. I know that Supervised Learning is probably easier and I have already tried a couple of approaches.  I works pretty good, but this RL approach could be an interesting thing for a small chapter and the outlook of my Master thesis.
So i just want to try it even if the results might be much worse.
I found a couple of papers about image classification using RL and it seems to be possible. So i don't see a reason why it shouldn't work for a numerical Dataset. I could use the difference of the prediction/action of the RL agent and the actual label to train the model.
My problem is that i am not that good in coding. I have never done Reinforcement Learning and don't realy know where to start. Has anyone tried to use RL for classification or knows a good publication with a rather simple code attatched that i could take a look at?


texts: TensorFlow doesn't learn when input=output (or probably I am missing something)

texts:
I have used TensorFlow before for training a model to recognize images (something similar to this example: https://github.com/tensorflow/models/tree/master/inception). I was trying to do something similar with inception-v4 from scratch (using the model from https://github.com/tensorflow/models). But it was not working, it wasn't learning, so I decided to reduce my code and simplify everything. The model is a hidden layer with the same neurons as inputs and what the model has to learn is that input=output. Seems pretty simple and easy, but the results of my code are far from good.

I have tried different things: sigmoid cross entropy instead of softmax cross entropy, other optimizers, different learning rates, models more comples with more neuros and more layers, reduce_sum instead of reduce mean for the loss ... But the only times I got "good results" (>90%) it seems because the model falls into a local minimum and I am not able to replicate them. Usually I see 70% precision with a high loss (>=0.7).

My conclusion is that there is something wrong in my code that I am not able to spot and that is the reason the learning process is slow and get stuck at high losses. Otherwise the code with inception-v4 for images will at least learn something and don't get the loss stuck at the same value all the time.

import os
import random
import shutil

import numpy as np
import tensorflow as tf
import tensorflow.contrib.layers as layers

# parameters
max_steps = 10000
batch_size = 64
output_size = 10
summary_path = './summary'

with tf.Graph().as_default() as graph:
    shape = [batch_size, output_size]
    inputs = tf.placeholder(dtype=tf.float32, shape=shape, name='inputs')
    outputs = tf.placeholder(dtype=tf.float32, shape=shape, name='outputs')
    global_step = tf.Variable(0, name='global_step', trainable=False)
    logits = layers.fully_connected(inputs, output_size)
    learning_rate = tf.train.exponential_decay(0.1, global_step, max_steps / 3, 0.1,
                                               name='learning_rate')
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, outputs, name='loss'))
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    train_op = optimizer.minimize(loss, global_step=global_step)
    init_op = tf.global_variables_initializer()
    input_label = tf.argmax(inputs, axis=1) # vector with the input labels
    output_label = tf.argmax(logits, axis=1) # vector with the predicted labels
    tf.summary.scalar('loss', loss)
    tf.summary.scalar('learning_rate', learning_rate)
    summary_op = tf.summary.merge_all()

# prepare summary
if os.path.exists(summary_path):
    shutil.rmtree(summary_path)
os.makedirs(summary_path)
summary_writer = tf.train.SummaryWriter(summary_path, graph=graph)

with tf.Session(graph=graph).as_default() as session:
    session.run(init_op)
    to_run = {'step': global_step, 'loss': loss, 'logits': logits, 'train_op': train_op,
              'learning_rate': learning_rate, 'summary': summary_op,
              'in_label': input_label, 'out_label': output_label}
    step = 0
    while step < max_steps:
        # fake input
        data = np.zeros(shape, dtype=float)
        for i in range(batch_size):
            data[i, random.randint(0, output_size - 1)] = 1.0
        # run
        results = session.run(to_run, feed_dict={inputs: data, outputs: data})
        # print data
        step, in_label, out_label = results['step'],  results['in_label'], results['out_label']
        precision = (in_label == out_label).sum() * 100.0 / batch_size
        print('step: {0}/{1}  learning rate: {2:.4f}  loss: {3:.4f}  precision: {4:.1f}%'
              .format(step, max_steps, results['learning_rate'], results['loss'], precision))
        summary_writer.add_summary(results['summary'], step)
    # print last batch
    print(data)
    print(results['logits'])
    print(in_label)
    print(out_label)
    print('precision {}%'.format(precision))

summary_writer.flush()


texts: what should i do if my target variable is categorical when using decision tree? (many categorical variables)

texts:
all,

i'm trying to classify a set of features to belong to a particular company (my dependent variable). my independent variables are a mixture of continuous and categorical features.

my data-set i am training on is labelled data with the label being the company ( the dependent var). I am not sure how i should go about dealing with my dependent variable should i use one hot encoding on the the entire data-set and then split it into training and test?

i am uncertain whether to onehot encode before i feed into decsion tree.


texts: Search for hyperparameters whith different features using Random Forest

texts:
I have a dataset in which I would like to perform a classification model, so I have decided to use Random Forest. The number of features that I have is approximately 200 and I would like to test which set of features gives me a better result. For the sake of experimentation, I do not want to use any feature importance method.

So what I have done is to split my features, without replacement, in groups of 20, 40,60,80 and 100 features. I am using GridSearch for tuning the hyperparameters of my RF. The question that I have is if I want to compare the accuracy of each one of the models (the one with 20 features, with 40, and so on), it would be fair to apply GridSearch with each subset of the features. Or should I only perform Gridsearch once, let's say with the model of 20 features, and then use the same hyperparameters with the set of 40 features, 60 features and so on.

Any help?

Thanks


texts: Which neural network can be trained on text attributes to find table headers on unseen document?

texts:
Due to my lack of knowledge in deep learning and neural network I need help picking the best approach to tackle a challenge.

I have a dataset with list of words where each word has 4 attributes (x, y, fontSize, textWidth). All words in this dataset are extracted from headers and subheaders in a document.

After training a neural network on this dataset, I need it to find all headers and subheaders in an unseen document that matches the shape of a chart.

Question: Which neural network would work best for this problem?


texts: How to convert Hindi/Telugu/Marathi text to vector for text classification problem?

texts:
sentence = 'अच्छा होगा अगर इसमें और गहने ना हों'

Which method will work for this task?
Is any pretrained model available to convert this text to vectors?
Please help by giving the code.


texts:
I was starting to learn stemming with nltk and a few words were quite inappropriately stemmed.
For example:- very was stemmed to 'veri', important to 'import', once to 'onc', poorly to 'poorli' , etc.

I was just thinking that during data analysis, does it cause any error or these errors can be ignored?
Or is there a better option for the same purpose


texts: Why is my Python Altair chart printing a blank line?

texts:
Wondering if anyone can help me understand why python altair chart is not printing...

I’m only seeing this when I run the code:

It literally isn’t outputting anything.

This is what my chart looks like converted to_dict; recommended for troubleshooting to ensure you have data.

Any ideas about why it’s not displaying a graph/chart/etc, and only printing a blank line?

Adding the head of the csv in pandas form.


texts: How to use time-sequence data with "meta data" of single value per sample?

texts:
I'm, trying to estimate a fish weight by a time-sequence signal of the change in resistance when the fish goes through a gate with electrodes installed.
When the fish pass through the gate there's a spike in resistance, and the shape of it supposedly derives from the fish weight.
Here is an example of resistance readout of 340 gram fish experiment:

And here is an example of the extracted spikes from the same 340 gram fish experiment:

However, there are other parameters that governs the signal's shape such as:

Water Temperature
Water Salinity
Gate formation (the proximity of the electrodes)

I wonder how to incorporate this type of data, which is single-valued per sample (for example, for all of fish #12 samples, temperature=28.2 Celsius, salinity=410, gate formation=80mm) with the time-series data of the sample itself (which is ~80 data points series).
So far I tried mostly neural networks (fully connected and 1-D convolutional architectures) and linear regression.
I searched the internet quite a while and couldn't find a suggested architecture for such data.
Would adding a channel for each parameter with the fixed value work?

Or alternatively, adding these parameters as part of the spike vector?

Any ideas / reference to some research?
Many thanks!


texts: Image classification vs medical grading problems

texts:
For image classification problems like cat vs dogs, the output layer is 2.
Image classification problems like diabetic retinopathy seem to be more of a grading classifier. Although the targets range from 0 to 4, (signifying the severity of the condition), is it better to have 1 as the output layer, or 5, for these kinds of problems?
I have seen Kaggle kernels where both are utilized.


texts:
Say I have an imbalanced data set, and I decided to over/undersample it during model training. I would then like to predict on new records but using the original, true imbalance in the classes as an apriori for the model.

i.e my classes are distributed in a ratio of 1:100, I undersample to 1:2, but would like the model to know that class A is very rare so be careful in predicting it.

For logistic regression I am familiar with a method of intercept correction, detailed here:

My question is: what if I use deep leaning instead of logistic regression?
My reasoning is that since the last layer of the NN is basically the same a logistic regression, I can use the same method for the intercept of this layer.
Can you think of any objections? any reason this doesn't make sense? If so, how would you go about correcting the model?

Also, is there a generalized method for intercept correction for a case when I have more than 2 classes?


texts: An algorithm to extract the purpose of a document

texts:
I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use a semi-supervised learning algorithm. I was thinking of training a Q/A algorithm (using Bert for example..) but I want something to be specific to the purpose task..
Any idea or keywords that might help ?
Thanks!


texts: How does ML algorithms treat unseen data, a conceptual discussion

texts:
I want to predict the occurrence of certain events, but these events only occur say 5% of the time in my data, hence in 95% of the data there is nothing to learn.

In order to teach the ML algo something I have learned to single out the 5% and drop the rest of the data. Let us say that I want to predict if a picture is of a dog or a cat. In my data 2.5% of the pictures are of dogs and 2.5% of cats, the rest are just random pictures. So, I single out the cat and dog pictures and label them so that the ML algo can learn from that. Am I broadly right so far?

So, if I train my algo on only cat and dog pictures and get a satisfactory accuracy, what will then happen in live usage when 95% of the pictures are not of cats or dogs? I.e. I show my model a picture of a house, what does it predict? Will my algo always predict either cat or dog, or will it somehow tell me that it has no clue what this picture is?

Any thoughts?


texts: How is the property in eq 15 obtained for Xavier initialization

texts:
I am new in this field so please be gentle with terminology.
In the original paper; "Understanding the difficulty of training deep feedforward neural networks", I dont understand how equation 15 is obtained, it states that giving eq 1 :

$$
W_{ij} \sim U\left[−\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}\right]
$$

it  gives rise to variance with the following property:

$$
n*Var[W]=1/3
$$

where $n$ is the size of the layer.

How is this last equation(15) obtained?

Thanks!!


texts: Training data for multi-category classification algorithm

texts: I am putting together a multi-category classification algorithm. Since it's NLP, the training data is very simple with one column for labels and another column for text. However, because it's NLP, some training records can fit multiple categories. Should I iterate through & have multiple entries (same data, different labels) or should each piece of data only have 1 category assigned?


texts: find the difference between two columns in specific rows

texts:
I have the following columns from one data frame

 0         1585742136995
 1         1585742137014
 2         1585742137035
 3         1585742137058
 4         1585742137080
          ...
 177809    1585570661653
 177810    1585570661675
 177811    1585570661686
 177812    1585570661709
 177813    1585570661731

Name: acctimestamp, Length: 177814, dtype: int64>

and

0         1585742136982
1         1585742136996
2         1585742137015
3         1585742137036
4         1585742137058
          ...
177809    1585570661632
177810    1585570661653
177811    1585570661676
177812    1585570661698
177813    1585570661710
Name: gyrtimestamp, Length: 177814, dtype: int64>

I want to start from 299 row and by 200 rows to find and print the difference between two columns (acctimestamp, gyrtimestamp)

so let's say at 299+200=499 row what is the difference between the 2 columns. After that what is the difference in 699 row And I want to do this for all the len of the data frame.

I am facing difficulties with the for loop can you help me a little bit?
how a loop like this would be?

i write this df['diff'] = df['acctimestamp'] - df['gyrtimestamp'] but it only show me the difference line by line

how to start from 499 row print the difference and print every 200 row?

i try this code:

for i in range (len(df)):
   i=200
   df1=df.loc[299+i, "acctimestamp" ]-df.loc[299+i, "gyrtimestamp"]

but the counter print the difference in line 499 and do not continue all the way to the end of the data frame


texts: How to build a speech corpus for continuous speech?

texts:
I am developing a speech corpus from audiobooks. I was able to collect recordings with their transcriptions but without timing information. Therefore I am looking for recent techniques to automatically build segments adapted to the development of ASR systems for large vocabulary continuous speech.
Thanks in an advance.


texts:
I have a training set and a testing set of vectors. All the vectors are labeled.

For each labeled vector in the testing set, there are 3 vectors in the training set with the same label.

I'm using the cosine distance in order to calculate the similarity between the elements in the vectors. In the picture, we can see the results of applying the cosine distance similarity in a subset of 6 vectors from the testing set and 18 from the training set.

Now, I would like to create a similarity matrix of the labels. So, in this case, I'd need a matrix of 6x6 dimension, but I am not sure how to transform this matrix of scores to a similarity matrix.


texts: Dealing with NaN (missing) values for Logistic Regression- Best practices?

texts:
I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with several missing (NaN) values. I get errors due to these missing values, as the values of my cost-function and gradient vector become NaN, when I try to perform logistic regression using the following Matlab code (from Andrew Ng's Coursera Machine Learning class) :

[m, n] = size(X);
X = [ones(m, 1) X];
initial_theta = ones(n+1, 1);
[cost, grad] = costFunction(initial_theta, X, y);
options = optimset('GradObj', 'on', 'MaxIter', 400);

[theta, cost] = ...
    fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);

Note: sigmoid and costfunction are working functions I created for overall ease of use.

The calculations can be performed smoothly if I replace all NaN values with 1 or 0. However I am not sure if that is the best way to deal with this issue, and I was also wondering what replacement value I should pick (in general) to get the best results for performing logistic regression with missing data. Are there any benefits/drawbacks to using a particular number (0 or 1 or something else) for replacing the said missing values in my data?

Note: I have also normalized all feature values to be in the range of 0-1.

Any insight on this issue will be highly appreciated. Thank you


texts: What are the advantages of contrastive divergence vs the gradient of the quadratic difference between the original data and the reconstructed data?

texts:
In this example I have a RBM with a visible layer and a hidden layer. The original data is "data", the values of the hidden neurons is "hid", the values of the visible neurons calculated from "hid" is "vis", and then the value for the hidden neurons calculated from "vis"is "hid2".

I calculate the positive_association and the negative_association by:

pos_associations = T.dot(data.T, hid)
neg_associations = T.dot(vis.T, hid2)

where T.dot represents the usual matrix calculation. I then update the weights using contrastive divergence (CD)

w = w + (training_rate)*(pos_associations - neg_associations)

like described in this post

I do get a good reconstruction (vis looks like data) and I can see features representing the weights. However, it takes a while and many epochs.

If instead of using CD I simply calculate the quadratic differnce between data and vis

cost = T.sum((data - vis)**2).mean()

and I update the weights using the gradient of cost, I get a faster approximation of the data. So my question is, what is the advantage of using CD?


texts: Cross-validation: K-fold vs Repeated random sub-sampling

texts:
I wonder which type of model cross-validation to choose for classification problem: K-fold or random sub-sampling (bootstrap sampling)?

My best guess is to use 2/3 of the data set (which is ~1000 items) for training and 1/3 for validation.

In this case K-fold gives only three iterations(folds), which is not enough to see stable average error.

On the other hand I don't like random sub-sampling feature: that some items won't be ever selected for training/validation, and some will be used more than once.

Classification algorithms used: random forest & logistic regression.


texts:
I have data of a metric grouped date wise. I have plotted the data, now, how do I remove the values outside the range of the boxplot (outliers)?

All the ['AVG'] data is in a single column,
I need it for time series modelling.


texts: Multi-GPU, multi-machine computation with Torch

texts:
I know that Torch supports multi-GPU computation on the same machine (Example).

Is it possible to perform multi-GPU, multi-machine computations with Torch?


texts: Iterating over trained Tensorflow models

texts:
I have a loop that looks like this
import glob

models = glob.glob('./model/*')

preds_array = []

for model in tqdm(models):
    this_model = keras.models.load_model(model)
    preds = this_model.predict(X_pred)
    preds_array.append(preds)

that runs through my model/ directory and reads in an ensemble of already trained models to look at the statistics of the different models. There's about a thousand of them. When I run this I get a bunch of printouts of the warning:

WARNING:tensorflow:5 out of the last 9 calls to <function
Model.make_predict_function..predict_function at
0x7f9e2b76b3b0> triggered tf.function retracing. Tracing is expensive
and the excessive number of tracings could be due to (1) creating
@tf.function repeatedly in a loop, (2) passing tensors with different
shapes, (3) passing Python objects instead of tensors. For (1), please
define your @tf.function outside of the loop. For (2), @tf.function
has experimental_relax_shapes=True option that relaxes argument shapes
that can avoid unnecessary retracing. For (3), please refer to
https://www.tensorflow.org/guide/function#controlling_retracing and
https://www.tensorflow.org/api_docs/python/tf/function for  more
details.

Is there some more efficient way to loop over saved models than what I have, or am I just going to have to live with this warning?


texts: Advantages of Recurrent Neural Networks over basic Artificial Neural Networks

texts:
I have started reading Deep Learning Book, and I am having trouble understanding the advantages of RNN. This part of confuses me:

  The unfodling process thus introduces two major advantages:

  Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition
  from one state to another state, rather then specified in terms of
  variable-length history of states.
  It is possible to use same transition function f with same parameters at every time step.

  These two factors make it possible to learn a single model f that
  operates on all time steps and all sequence lengths, rather then
  needing to learn a separate model for all possible time steps.
  Learning a single shared model allows generalization to sequence
  lengths that did not appear in the training set, and enables the model
  to be estimated with far fewer training examples than would be
  required without parameter sharing.

I understand the 2nd advantage. Because the computations are recurrent, the input besides the current element in the sequence is the output of the previous hidden state which has the same structure as the current hidden state, thus the shared parameters.

But I do not understand the first advantage. I cannot visualize it or mathematically prove it, or at least I dont know how. Can anyone help me with this one? And if anyone has anything else to add on the difference ( advantages ) of RNN over ANN I would really appreciate it. Thanks in advance!


texts:
I'm looking for recommendations as to the best way forward for my current machine learning problem

The outline of the problem and what I've done is as follows:

I have 900+ trials of EEG data, where each trial is 1 second long. The ground truth is known for each and classifies state 0 and state 1 (40-60% split)
Each trial goes through preprocessing where I filter and extract power of certain frequency bands, and these make up a set of features (feature matrix: 913x32)
Then I use sklearn to train the model. cross_validation is used where I use a test size of 0.2. Classifier is set to SVC with rbf kernel, C = 1, gamma = 1 (I've tried a number of different values)

You can find a shortened version of the code here: http://pastebin.com/Xu13ciL4

My issues:

When I use the classifier to predict labels for my test set, every prediction is 0
train accuracy is 1, while test set accuracy is around 0.56
my learning curve plot looks like this:

Now, this seems like a classic case of overfitting here. However, overfitting here is unlikely to be caused by a disproportionate number of features to samples (32 features, 900 samples). I've tried a number of things to alleviate this problem:

I've tried using dimensionality reduction (PCA) in case it is because I have too many features for the number of samples, but accuracy scores and learning curve plot looks the same as above. Unless I set the number of components to below 10, at which point train accuracy begins to drop, but is this not somewhat expected given you're beginning to lose information?
I have tried normalizing and standardizing the data. Standardizing (SD = 1) does nothing to change train or accuracy scores. Normalizing (0-1) drops my training accuracy to 0.6.
I've tried a variety of C and gamma settings for SVC, but they don't change either score
Tried using other estimators like GaussianNB, even ensemble methods like adaboost. No change
Tried explcitly setting a regularization method using linearSVC but didn't improve the situation
I tried running the same features through a neural net using theano and my train accuracy is around 0.6, test is around 0.5

I'm happy to keep thinking about the problem but at this point I'm looking for a nudge in the right direction. Where might my problem be and what could I do to solve it?

It's entirely possible that my set of features just don't distinguish between the 2 categories, but I'd like to try some other options before jumping to this conclusion. Furthermore, if my features don't distinguish then that would explain the low test set scores, but how do you get a perfect training set score in that case? Is that possible?


texts: Gradient boosting, where did the constant go?

texts:
In the very early papers on gradient boosting, the ensemble would include a constant and a sum of base learners i.e.

$F(X) = a_0 + \sum\limits_{i} a_i f_i(X)$

The constant is fitted first (i.e. if the loss is squared error, the $a_0$ will be the unconditional mean of the target)

Somewhere along the way this got dropped, i.e. as far as I can see a number of the current "state of art" gbdt algorithms fit a constant (xgboost, catboost etc). I believe the R mboost package does though.

Why is this, does anyone have any reference where it is discussed (I'm performing regression so particularly interested in this aspect). In particular with a regression tree as a base-learner, can it learn the constant more efficiently than the original algorithms?

One of my motivations is that when I boost with no constant, for a problem where the target is always positive, it seems to converge from below so it almost always has a bias in the residuals unless you have enough data such that you can boost for long enough to fit the highest values of the targets without overfitting (specifically I fit the relationship between temperature and electricity demand).

Edit: Some references

[Prokhorenkova et. al.] "CatBoost: unbiased boosting with categorical features" - Algorithm 1 intialises M to 0. I've used catboost a lot, I can confirm that explicitly setting the starting value to the mean produces a differerent result.

[Chen et. al.] "XGBoost: A Scalable Tree Boosting System" - Equation 1 excludes the constant.

[Sigris] "Gradient and Newton Boosting for Classification and Regression" - there's no mention of the constant alough it's not explicitly excluded


texts: If a categorical feature only occurs a few times in a data set, should I drop it?

texts:
I have a data set of mostly categorical variables. When I one-hot encoded them some of the features occur less than 3% of the time.

For instance the Tech-support feature only occurs 928 times in a data set with 32561 samples ie. it only occurs 2.9% of the time.

Is there a general cutoff point for when I should scrap these variables? I'm cleaning up this data set for binary logistic regression and an SVM.

Thank you!


texts:
We have a production database. The load on the database varies at different times. I want to identify anomalies; for e.g, the number of database processes responding to user queries at 9 am is 100 for a given day. If the number is 200, then it's an anomaly and as the DBA, we need to check the DB immediately. The goal is to identify a pattern and alert when there's an event outside this pattern.

Day  time  processcount label
Mon  09:00 100          Normal
Mon  09:05 150          Normal
Tue  09:00 200          Abnormal

I'm using pandas to collect the data but I'm not sure how to identify the pattern and report anomalies. The closest i could get is this thread
How do I approach grouped anomaly detection?


texts:
I am trying to get some data out of R snippet to Azure Python 3 Jupyter notebook (hosting is available for free at http://notebooks.azure.com).

I tried the following in Python 3 notebook:

%load_ext rmagic

then tried to embed R:

%%R -o x
x <- 1

then

x

Here I get Python error: name 'x' is not defined – see the picture below. What is the right way to embed R into Python 3 and exchange data using Azure Notebooks?

Azure Notebooks http://nogin.info/R2Py.png


texts: How to treat the datasets with unreliable labels

texts:
I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the samples in class C is actually belong to class A and class B. For now, I need to run some supervised learning (logistic regression, decision tree and random forests) models. According to the confusion matrix, the classification between A and B is relatively accurate, but the classification between C and other two classes is not acceptable.
I wondered whether is any way to deal with this issue?

For now, I'm considering to use a clustering algorithm for the samples in class C before runnning the model. After dividing the samples in class C into 3 groups, try to find a relatively better group as the datasets for class C.


texts: When I include validation_data=(x_val, y_val) in model.fit_generator, should I create another test dataset for accuracy measures?

texts:
While modelling in keras, often I see the usage of validation_data=(x_val, y_val) in model.fit_generator where (x_val, y_val) generally forms 10% of the dataset. While training, is it that the model takes hint from the validation loss calculated on (x_val, y_val) and I need to create another test_data for measuring accuracies in the end? Or, I can use the same (x_val, y_val) in input of model.fit_generator and measuring the accuracy in the end.

The confusion arises from the fact that we are often advised to create training, validation, test datasets while modelling. If the validation dataset has been used to judge some parameters like when to stop(early stopping), etc; wouldn't it be unfair to use the validation_data to calculate the various measures of accuracy?


texts: How to extract most occuring words based on month & what tool to use?

texts:
Hi guys I'm very new to data science,
I have intermediate background on programming and have used Pentaho Data Integration tool once for DB migration & data cleansing.

Let's say I have this kind of data:

item_details, timestamp
Wooden chairs, 01-07-2017
Plastic chairs, 02-07-2017
Stainless table, 11-07-2017
Decorated window, 12-07-2017

and so on

I want to know based on monthly time frame  what are the top trending items in that month.
Let's say in January the top 3 item is:
1. Table
2. Chairs
3. Window
In February :
1. Door
2. Chair
3. Cupboard
.. And so on

How can I achive this and using what kind of tool? (preferably free or open source tools, can be GUI based or script library, having a visualization or  dashboard is a plus)

Thanks for the help. Sorry for noob questions


texts: I want to build an ml model, which can when given a text input, can predict predefined tags or labels for the text. I already built one such model, but the problem with that is that it only predicts correct labels when a particular keyword is present in the text. For example - TEXT - "How to crack an interview?". LABELS PREDICTED - "Interview, Fresher". BUT, If TEXT - "How to clear hr round of xyz company, what questions the can ask?". Either it will predict some irrelevant label or no label at all.  My model is very similar to this model


texts: I have been reading about probability distributions lately and saw that the Normal Distribution is of great importance. A couple of the articles stated that it is advised for the data to follow normal distribution. Why is that so? What upper hand do I have if my data follows normal distribution and not any other distribution.


texts: Keras exception: Error when checking input: expected dense_input to have shape (2,) but got array with shape (1,)

texts:
I have an understanding of this error, it means that the input that I'm passing to the model is of a different dimension that what was expected. The error also states that the input that I'm passing is of the dimension (1,) while it was expecting (2,)

I have tested the input value dimension by using x.shape and it prints out (2,) still the error exists. As a counter-intuitive move I picked one of the data that was in the training data and printed the shape of the zeroth element x1[0].shape also used that as an input, the error still exists.

model.fit works well, having error with model.predict (tried passing one of the training data hardcoded, still doesn't work)

CODE:

import tensorflow as tf
import numpy as np
from tensorflow import keras
import csv

x1, ys = [], []

with open('./house.csv') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    line = 0
    for row in csv_reader:
        if line > 0:
            x1.append([row[1], row[3]])
            ys.append(row[5])
        line += 1

model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[2])])
model.compile(optimizer='sgd', loss='mean_squared_error')
x1 = np.asarray(x1, dtype=float)
ys = np.asarray(ys, dtype=float)
model.fit(x1, ys, epochs=500)

print(x1[0].shape)
while True:
    house_size = float(input('Enter the house size: '))
    house_size = house_size/3000
    bhks = float(input('Enter the BHK: '))
    bhks = bhks/3
    x = np.array([house_size, bhks])
    try:
        value = model.predict(x)
    except Exception as e:
        print(e)
        print(x)
        print(x.shape)
    else:
        value = value[0][0] * 500
        print(value)


texts: In multinomial logistic regression, how to explain the softmax outputs properly?

texts:
I tried to solve multiclass problem ("cat", "dog", "horse") problem and figured out that the more words in test text, the more difference between classes.
I grouped the output by number of words and created bins, getting the mean values of probability scores for the classes:
                cat          dog      horse      n_words
(0, 5]      0.322369    0.319800    0.357831    3.392157
(5, 10]     0.293250    0.307544    0.399207    8.722892
(10, 15]    0.301972    0.306258    0.391769    13.076923
(15, 25]    0.254966    0.280201    0.464833    20.546961
(25, 35]    0.239433    0.261795    0.498772    30.387755
(35, 50]    0.176405    0.300194    0.523401    38.571429

Is there any mathematical explanation for this observation? Any sources to read are very welcome


texts:
Background:
While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as stated here. (What is the "dying ReLU" problem in neural networks?)

Question:
What im hoping to do is to implement a check in the code itself to check if the neurons are dead. After that, the code could refit the network if needed.

As such, what is a good citeria to check for dead neurons? Currently im thinking of checking for low variance in the prediction as a citeria.

If it helps, im using keras.


texts: What are the ways to partition a large file that does not fit into memory so it can later be fed as training data?

texts: Is there any other way to partition a large file that does not fit into memory so it can be fed as training data other than using spark? or hadoop?


texts: Understanding Logistic Regression Cost function

texts:
Linear Regression cost function:

$$J(\theta) = \frac{1}{2 m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$

where:

$$h_{\theta}(x) = \theta_0 + \theta_1 x_1$$

Logistic Regression cost function

$$J(\theta) = \frac{1}{m} \sum_{i=1}^m - (y^{(i)} \times log(h_{\theta}(x{(i)})) + (1-y{(i)}) \times log(1 - h_{\theta}(x{(i)})))$$

where:

$$h_{\theta}(x) = g(\theta_0 + \theta_1 x_1 + \theta_2 x_2)$$

Intuitively linear regression is easy to understand as it optimizes the average squared distance between hypothesis and training data. But in case of logistic regression, I failed to understand the cost function.

What does the logistic regression cost function represent?


texts: Document similarity matching between Doc2Vec documents

texts:
I am creating a Doc2Vec model out of hundreds of PDF documents.

I have 17 documents that are part of this Doc2Vec that I want to use to check similarity with other documents in the Doc2Vec model.

For instance I want to something like: model.similarity(tag5, tag30)

Can this be done?


texts: How to reduce dimensionality of encoder decoder output?

texts: I have an encoder decoder architecture where the output  $ \bar{\bf{y}}_t $ is a sequence of integers of maximum length $n$. Each integer in the sequence is representative of a category so the sequence $ {0 ,1 ,3 ,4 ,6} $ could mean $\text{Car , Train , Plane , House , Dog}$. There are $m$ possible categories. The current output of the network is an $n \times m$ matrix where the  entry $(i,j)$ is meant to represent the probability that the $i^\text{th}$ element of the output sequence belongs to category $j$. I was wondering is there a way to reduce the dimensionality of this problem by sharing weights among the rows of the output matrix. I was thinking there may be a way of predicting the outputs sequentially so there is weight sharing among the rows


texts: Linking LDA topics to the input documents

texts:
I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. I am aware that we need use the generated model to classify any new document. I would like to know the topics for each training document. Do I have to treat them as new documents to find their topics?

lda.print_topics(num_topics=22, num_words=8)


texts: Avoiding Overfitting with a large LSTM net on a small amount of data

texts:
I'm reposting this question from AI.SE here as I think it was maybe off-topic for AI.SE...
1. Context
I'm studying Health-Monitoring techniques, and I practice on the C-MAPSS dataset. The goal is to predict the Remaining Useful Life (RUL) of an engine given sensor measurements series. There's a wide litterature about the C-MAPSS dataset, including both classical (non-DL) ML techniques and DL-based approaches. A few years ago, LSTM-based networks showed promising results (see Long Short-Term Memory Network for Remaining Useful Life estimation, Zheng et al, 2017), and I'm trying to reproduce these results.
The C-MAPSS dataset contains a low amount of data. The FD001 subset has for instance only 100 run-to-failure series. When I pre process it to get fixed-length time series, I can get up to ~20 000 framed series. In the article mentioned above using LSTM, they use two hidden LSTM layers with 64 units each, and two fully-connected layers with 8 neurons each (~55 000 parameters).
2. Problem
LSTMs induce a great number of parameter, so overfitting may be encountered when training such a network. I can use L1 or L2 regularization, dropouts, the net will still be largely oversized regarding to the dataset. Keeping the same architecture, I can't reach the scores and RMSE in the paper in the validation set, and overfitting is always here.
However, one thing that works is reducing the number of units of the LSTM layers. Expectedly, with only 24 units instead of 64 per layer, the net has much less parameters (~9000), and it presents no overfitting. The scores and RMSE are a bit worse than the one in the paper, but it's the best I can get so far. Although these results are fine for me, I'm curious about how it was possible for the authors of the paper to avoid overfitting on their LSTM(64,64) net.
3. Question
LSTM are great, but they induce a lot of parameters that hinder a correct learning on small dataset : I wonder if there is any method to tackle this specific issue. Would you have any advice on how to avoid overfitting with a LSTM-based net on a small dataset ?
4. Infos
I provide here below more infos about my net and results :
Network architecture
model = keras.models.Sequential([
    keras.layers.LSTM(24, return_sequences=True, kernel_regularizer=keras.regularizers.l1(0.01),
                      input_shape=input_shape),
    keras.layers.Dropout(0.2),
    keras.layers.LSTM(24, return_sequences=False, kernel_regularizer=keras.regularizers.l1(0.01)),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(8, activation='relu', kernel_regularizer=keras.regularizers.l2()),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(8, activation='relu', kernel_regularizer=keras.regularizers.l2(), bias_regularizer=keras.regularizers.l2()),
    keras.layers.Dense(1, activation='relu')
])

Scores (Validation set)

Paper:  Score = 16.14 ; RMSE = 338
My LSTM(64, 64): Score = 26.47; RMSE = 3585 (overfits)
My LSTM(24, 24): Score = 16.82; RMSE = 515

Edit : Results for solution proposed by @hH1sG0n3

LSTM(64, 64) with recurrent_dropout=0.3 : Score = 16.36; RMSE = 545


texts: how to consider some miss classifications “half correct” using as base categorical_crossentropy - for a trading system

texts:
I have a trading system where the model receives 9 time-series and predict :

A - strong down
B - week down
C - neutral
D - week up
E - strong up

(these classes are generated from an histogram to have a balanced training dataset ... the histogram is separated in 20% parts of examples centered in 0... accuracy of 20% is the baseline)

For each class I activate a different parametrized trading mechanism...

My model is giving acceptable results. Here is the resulting confusion matrix for val data (28.99% acc):

[[32 20  3  6  8]
 [35 19  9  7 16]
 [30  9  6 14 24]
 [21 14  9 18 29]
 [ 9 14  3 14 45]]

My question starts here :

I.e. If the model predicts B-"week down" but in reality is A-"strong down" is a miss, but in reality, it will make money...

So in this confusion matrix we can see that it happens 20 times (cell[0,1]) ... also if it is B but the model says A it will make money in 35 trades (cell[1,0])...

And also the same for the UP cases..

All together (from the confusion matrix) :
32+20+35+19  +  18+29+14+45 = 212 winning trades
21+14+9+14  + 6+8+7+16 = 95 losing trades

Assuming negative trades cancel in equal (in reality will not be equal..) positive trades, the total is = 117 winning trades.

What I want is to create a loss function based on categorical_crossentropy
 but somehow consider:

pred A real B - half miss
pred B real A - half miss
pred D real E - half miss
pred E real D - half miss

Do not penalize too much this cases. I think this will increase a bit the total number of positive trades. It will guide the learning a litle bit better (maybe not for accuracy but for a better loss that generates a better confusion matrix for profit )...

I have created a custom loss function that reduces 3% the loss for these cases:

def my_loss(y_pred, y_true):

    y_pre_indexes = K.argmax(y_pred, axis=1)
    y_test_indexes= K.argmax(y_true, axis=1)

    TN = K.tf.logical_or( K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,0),K.equal(y_test_indexes,1)),
                          K.tf.logical_and(K.equal(y_pre_indexes,1),K.equal(y_test_indexes,0)))
                        ,
                          K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,3),K.equal(y_test_indexes,4)),
                          K.tf.logical_and(K.equal(y_pre_indexes,4),K.equal(y_test_indexes,3))))

    pos_neg = K.cast(TN, K.floatx()) *(-0.03) + 1

    return K.categorical_crossentropy(y_pred, y_true)*pos_neg

(in the code the classes are :  0-A 1-B 3-D 4-E.   2-C is predicting neutral - ignore..)

but fixing to a fixed number of 3% to reduce loss for these cases seems a little bit hard coded.... Something better inside the categorical_crossentropy math philosophy should be better.

Any suggestions?

Thanks in advance!


texts:
Can some one please explain Isolation Forests more clearly? Everywhere I search, I find the same explanation:

  Isolation Forest ‘isolates’ observations by randomly selecting a
  feature and then randomly selecting a split value between the maximum
  and minimum values of the selected feature.

Let's take an example to solve this:

x1 = [2, 1, 4, 6, 4, 2, 1, 2, 3, 4, 19]

How would I say that 19 is an outlier?


texts: Math Behind GOSS (Gradient-Based One Side Sampling)?

texts:
As per my understanding through books & Google Search,

GOSS (Gradient-Based One Side Sampling) is a novel sampling method that downsamples the instances on the basis of gradients. As we
know instances with small gradients are well trained (small training
error) and those with large gradients are undertrained. A naive
approach to downsample is to discard instances with small gradients by
solely focussing on instances with large gradients but this would
alter the data distribution. In a nutshell, GOSS retains instances
with large gradients while performing random sampling on instances
with small gradients. Source

LightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value while XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split.

Can someone please explain the math behind GOSS?


texts: What is the purpose of the 'train model' step in data mining?

texts:
My understanding is that training a model is something done in machine learning using training data so that the model can predict values when new data is given to it.
Data mining is the process to find patterns in the existing dataset. So what exactly is the purpose of the 'train model' step in data mining?


texts: Alternatives to linear activation function in regression tasks to limit the output

texts:
I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not inside the range while training. I've not seen any paper but I have two solutions for that right now but I don't know if there is any standard way or not.

My first suggested solution is to use an activation function like the following. It is linear in the specified range and has high slope out of the specified range. Although the gradient will oscillate if I use simple gradient descent, I guess it will perform well if I use second-order optimization algorithms like Adam. Any suggestions for this?

My second solution is inspired by L1/L2 regularization. I find the output of the model. If it exceeds the absolute value of the range, suppose it's symmetric, I'll add a constant big value which is far from the real output. Here We can replace it with a big constant or maybe multiply it by a constant. The second one has this property that it will be differentiable with a slope.

Does anyone have suggestions which are applicable or even these are ok or any other thing?

In regression tasks, it is customary to use linear activation function as the non-linearity of the final layer in order to estimate a function which outputs real value(s). The reason sigmoid function is used is that its output is limited to the range 0 to 1 which is a good range for specifying probability. Moreover, sigmoid is used more for classification tasks where the classes are mutually exclusive in the input o.w. softmax activation function is used. Tanh is used due to having zero mean which accelerates the training process. These are usually not applied in classification tasks these days because their differentiation saturates in their limits. Relu is a customary choice because its slope does not saturate.
Sigmoid-shape activation functions cannot be used as the last layer of the regression tasks. I want to use a kind of activation function like linear as the last layer's activation function in order to have outputs in a special range. While training I want to train the model in a way that it never tries to output a value which is outside a specified range. For doing so, I've put two suggestions that I have tried. I don't know whether there is a paper about that or not. Moreover, about the provided answer, sigmoid does not penalize the outputs, I mean does not increase them in a way that the error value increases, which are outside the range of the domain of the function. What it does is saturating the output to zero or one. What I want to do is to make a huge difference between the output of the network and the real output of the input in order to cause a high error. Consequently, the network will try to find weights which do not yield results out of the range.


texts:
I am trying to fit a RandomForest model for a binary classification dataset and I have some feature like, the sales for a particular store and yes/no information indicating whether a store subscribed for a special program offered by the company. So the first one is numerical and second is categorical (binary) variable. Now it has many null values. This missing values are not due to any systematic error like some faulty sensor or some missing fields in docs. The values are missing, because they were not shared. So suppose I have to create the model for a particular time period (say entire 2020) and I have the Sales data only from April 2020. So the sales value for Jan-March are missing. Moreover, for the stores I have, some of them do not have sales information at all for some month. This might happen because those stores were not part of business before April. Same applies for that binary variable. Now these are the possible strategies I can apply:

Replace the missing sales values with some central tendency statistics like median or mode. (But using this will assume that the stores which were not present before April were also part of business)
Replacing sales values with some special value (like -1). But this might misguide the model. And also, if any return sales are there(negative sales), this approach won't work.
For the binary variable, I can fill it with some special token like "NULL", but that will require to create one more column in case I need one hot encoding.

What should be the best approach. I am kind of confused and I tried these approaches and all of them give more or less same AUC. Sometimes it reduces a bit.


texts:
We normally have fairly large datasets to model on, just to give you an idea:

over 1M features (sparse, average population of features is around
12%);
over 60M rows.

A lot of modeling algorithms and tools don't scale to such wide datasets.

So we're looking for a dimensionality reduction implementation that runs distributely (i.e. in Spark/Hadoop/ etc). We think to bring number of features down to several thousand.

Since PCA operate on matrix multiplication, which don't distribute very well over a cluster of servers, we're looking at other algorithms or probably, at other implementations of distributed dimensionality reduction.

Anyone ran into similar issues? What do you do to solve this?

There is a Cornell/Stanford Abstract on "Generalized Low-Rank Models" http://web.stanford.edu/~boyd/papers/pdf/glrm.pdf that talks specifically into this:

page 8 "Parallelizing alternating minimization" tells how it can be
distributed;
also page 9 "Missing data and matrix completion" talks
how sparse/ missing data can be handled.

GLRM although seems to be what we are looking for, but we can't find good actual implementations of those ideas.

Somebody wrote a version for Spark (e.g.
https://github.com/rezazadeh/spark/tree/glrm/examples/src/main/scala/org/apache/spark/examples/glrm
but it looks more of a proof of a concept rather than a version that
we could use in production);
H2O has a version of GLRM
http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/glrm.html
but it actually doesn't scale to our dataset size (see above).

Update 7/15/2018: Another Abstract is Fast Randomized SVD from Facebook (read here http://tygert.com/spark.pdf ) and also idea to do low-rank matrix approximation using ALS - http://tygert.com/als.pdf . Although there is no clear way how to use them now - see discussion at https://github.com/facebook/fbpca/issues/6

Any other ideas how to tackle this? Other available GLRM or other distributed dimensionalaity reduction implementations?


texts: Variable importance of Numerical features in Classification Model - Random Forest Classifier

texts:
I have few numeric features in my model. Out of 25 features, I have 7-8 numeric features in my model.
One thing I observed is model gives more weightage to numerical feature compare to categorical features.
Is it always behave the same way or it is because of the features?

Should I convert those numeric features into  categorical?


texts: How do i calculate prediction probability of a class in Java Weka Api?

texts:
I am developing a prediction model using Java Weka api. I can predict class for the new instance using the following code:

double predictClass = classifer.classifyInstance(instance)

However, I need class probability instead of class value. Thanks in advance for your support.