texts: How do you use Tensorboard to visualise my Chatbot? What can I learn from it?

texts:
I am currently learning DeepLearning and wanted to ask a few questions in relation to my current project https://github.com/deepcollege/deeplearning/blob/master/030-chatbot/chatbot_simple.py

First of all, how do you generate an useful Tensorboard model graph? As you can see from the code, current session.graph isn't outputing anything helpful or comprehensible. My model is basically a chatbot model based on Seq2Seq, how do you generate more simplified and useful Tensorboard model graph?

My second question is, in NLP situation (more specifically a chatbot), what other things can you do to analyse the model in Tensorboard? And how do you enable them?

Thank you


texts: How to use random forest model to new data?

texts:
I am new to this Data Science field. I have a question to apply Random forest to new data.

I have this table.

Y prop_A prop_B
A   0.8    0.2
A   0.7    0.3
B   0.5    0.5
B   0.4    0.6
B   0.1    0.9

I assumed that if the proportion of the group is high, chances are high that it is in the group. I built a model using random forest and test it with validation set (8/2 splits).

I thought the above model can be used for new data. This is an example of the data. The data structure and variable meaning is same, but the number of variable is different.

Y prop_C prop_D prop_E prop_F
-   0.8    0.1   0.05   0.05
-   0.6    0.3   0.05   0.05
-   0.5    0.4   0.05   0.05
-   0.4    0.2   0.4     0
-   0.1    0.5   0.4    0.4

The new data is unlabeled so I would like to make a label using the Random forest I used with previous data. Is it right approach to label the new data?

In the model, it doesn't works (due to different independent variables).

How should I do to label the new data based on a model using labelled data, which is  different?


texts: Handling a combined dataset of numerical and categorical features for Regression

texts:
I have a dataset that has a large number of categorical features and a few numerical features, and I want to predict the probability that any given input is one of two types of a certain binary output feature. I don't want to solve a classifier problem for reasons that are well outlined in this link .

I have seen many tutorials about how to handle them independently but am less sure how to handle them together.


texts:
I'm a new programmer and this is my first ever neural network for real world application.

Here is the deal, I'm using a top-less pre-trained VGG-16 with some dense layers on top of it.(for image classification problem)

But no matter what hyper parameters I change I always get plots similar to these.

So my question is at what loss (val_loss) can we consider the model trained?

And is there a problem if the images that I feed the VGG-16 are RGB/255?

And if someone could explain how high validation loss (even if validation accuracy is high) could impact the model?

Thanks for taking time to answer me, and sorry if the questions seems stupid but I didn't find answers when searching else where.


texts:
I was wondering if there's a good way to use ensembling when I have two or more algoritims producing ranked lists.  That is, suppose I have the following datasets consisting of ordered lists (higher to the top means more relevant):

Method1_Rankings  Method2_Rankings GoldStandard_Rankings
item1             item2             item1
item3             item1             item3
item2             item10            item5
...

Is there a way to optimally combine methods 1 and 2 (e.g., give the rankings some weights or similar)?  Thank you.


texts: Is it a good idea to disable or strongly regularize in time series deep learning models?

texts:
I'm training a recurrent network on a stock price time series. As you can imagine, the price increases with time. I think the importance of the bias decreases as the stock increases, especially since the test set has an ever higher price than the end of the training set.

Is it a good idea to "discourage" the learning via bias by using heavy regularization, or disabling the bias entirely? Here's my architecture:

class ForecastModel(Model):
    def __init__(self):
        super(ForecastModel, self).__init__()
        self.layer0 = Dense(128, activation='relu', dtype=tf.float32)
        self.layer1 = LSTM(256, return_sequences=True, dtype=tf.float32,
                            bias_regularizer=l1(25e-2))
        self.layer2 = GRU(512, return_sequences=True, dtype=tf.float32,
                            bias_regularizer=l1(25e-2))
        self.layer3 = SimpleRNN(1024, dtype=tf.float32,
                            bias_regularizer=l1(25e-2))
        self.layer4 = Dense(2096, activation='relu', dtype=tf.float32)
        self.flat = Flatten()
        self.concat = Concatenate()
        self.layer5 = Dense(1, dtype=tf.float32)

    def __call__(self, inputs, training=None, **kwargs):
        a = self.layer0(inputs)
        b = self.layer1(inputs)
        b = self.layer2(b)
        b = self.layer3(b)
        a = self.flat(a)
        a = self.layer4(a)
        x = self.concat([a, b])
        x = self.layer5(x)
        return x


texts:
I try to write a program to crop background from an image.

This is a sample of my training data.
I have images with and without a background. (manually cropped)

The background is always similar (changes of light and so on. Sometimes people on random places.)

The object is always on the same place but not always the same (there are different models).

Can someone give me an advice how I can solve this Problem?
What is the right way to do something like this???


texts: Difference in model performance measures of train and test data sets

texts:
I am using CART classification technique by dividing a dataset into train and test sets. I have been using Mis-classification error, KS by rank ordering, AUC and Gini as MPMs(model performance measures). The problem I am facing is that the MPM values are quite far apart.

Dataset
Metadata

I have tried with minsplit equal to anywhere from 20 to 1400 and minbucket from 5 to 100 but couldn't get expected results. I have also tried oversampling/undersampling through ROSE package but without any improvement. Moreover, the mis-classification error increased a lot. Following code is through which I could get the best values, but they were not enough.

#Reading Data
pdata = read.csv("PL_XSELL.csv", header = TRUE)

#Converting ACC_OP_DATE from type factor to date
pdata$ACC_OP_DATE<-as.Date(pdata$ACC_OP_DATE, format = "%d-%m-%Y")

#Paritioning the data into training and test dataset
set.seed(2000)
n=nrow(pdata)
split= sample(c(TRUE, FALSE), n, replace=TRUE, prob=c(0.70, 0.30))
ptrain = pdata[split, ]
ptest = pdata[!split,]

#CART Model
#Taking the minsplit, minbucket values as low as possible, so that pruning
#can be done later. Higher values didn't allow any scope for pruning
r.ctrl = rpart.control(minsplit=20, minbucket = 5,  cp = 0, xval = 10)

#Calling the rpart function to build the tree
cartModel <- rpart(formula = TARGET ~ .,
    data = ptrain[,-1], method = "class",
    control = r.ctrl)

#Pruning Tree Code
cartModel<- prune(cartModel, cp= 0.00225  ,"CP")

#Predicting class and scores
ptrain$predict.class <- predict(cartModel, ptrain, type="class")
ptrain$predict.score <- predict(cartModel, ptrain, type="prob")

Results that I got-: Train data Mis-classification error-.103 AUC - 0.679 KS - 0.259 Gini - 0.313

Test data Mis-classification error-.113 AUC - 0.664 KS - 0.226 Gini - 0.307

Is it due to the dataset or am I doing something wrong. I am new to Data Analytics. It is a part of my academic project, so I need to use CART technique only. I will put separate questions for Random Forest and Neural Networks. Kindly help.


texts: Unbalanced training set from balanced data

texts:
I am looking to get an unbalanced training set with a given ratio of classA:classB from a dataset without regarding if it is balanced or not. The point is to analyze the influence of data imbalance on the accuracy.
I don't see any built-in function to go about this, so I was wondering what approach I should take. For now, I consider either undersampling or oversampling, but I am worried that it might skew the results too much, as I aim to get class ratios of 10/90, 20/80, ..., 80/20, 90/10.


texts: How much more data does my classifier need?

texts:
I have a dataset of ~100k news articles, and I'm trying to build a classifier based on the headlines and leads of every article. The dataset is not pre-labeled so I'm manually labelling a subset of the articles to train on. So far, I have 9 categories with ~600 labeled articles for each category. I'm using doc2vec to create document vectors and the SVC classifier from sklearn to make predictions.

My cross validation score (with 10 splits and shuffling of data) is hovering around 0.89.

When I plot a learning curve for the training set and test set, I interpret it as the classifier suffering from high variance, and that I need to collect more data. But is there any way I can approximate how much more data is needed to get a cross validation score of say, 0.95?

Here is my learning curve and the mean scores:

Training scores: 1.0, 0.99919679, 0.9991984 , 0.99791667, 0.99615385, 0.99550155, 0.99487179.

Testing scores: 0.40769912, 0.72283119, 0.78529511, 0.83461723, 0.85527486, 0.86151579, 0.86471912.

EDIT: I performed a grid search on the C and gamma parameters to the SVC model, and adjusted gamma to 0.015 which made the two lines converge a bit more. Adding plot and new scores.

New training scores: 0.97758621, 0.98703072, 0.98177172, 0.97540872, 0.96717739, 0.96479244, 0.96241702.

New testing scores: 0.45800254, 0.78917394, 0.83526468, 0.853326, 0.86849859, 0.87232875, 0.87508134.


texts: Python - Appending a new Dataframe column that is a function of two separate numerical columns

texts:
I have a dataset that gives me the geographic coordinates of residential properties.

My goal is to append a new column to this dataframe that shows the properties' distances to various prominent locations in the city. I plan on doing this by finding the difference in the latitude and longitude to give X and Y values, and using the pythagorean theorem to find the straight-line distance between the two locations.

How do I structure the code so that the new column is a pythagorean function of the differences between latitude and longitude?

Thank you.


texts: Finding sequence combinations that impact target variable the most

texts:
One can create a time series model to predict a target variable. What I need to do is find the input combinations and sequences that impact the target variable the most. In this case, the input data is a series of time steps, each of which has many features. The desired model would give clarity to not only the best features but the combinations and sequences of combinations that impact the target the most. This is desired because it models a system that I'd like to optimize based on this analysis.

Hope that makes sense. Ideas?


texts: Convert categorical data in numeric preserve euclidean distance

texts:
I m looking how to preserve Euclidean distance with categorical attribute.

Ad example, if I have a dataset with attribute of people, Age, weight etc..and i find a attribute "sex" where contain "female" ad "male" for gender, how can i do for analysis?

I seen that i can trasform in 0 and 1, but for me dont have more sense. Why i can't choose 10 and 20 like number for male and female?
I Wish that this value in my analysis take a sense.

Sameone have to suggest or explain a great tecnique?


texts: sklearn "balanced_accuracy_score" sample_weights not working

texts:
I wanted a metric where I could weigh each class as I wish while measuring "total accuracy". sklearn seems to have this with balanced_accuracy_score. Irrespective of the sample_weight, I am getting the same "balanced accuracy". Why? what was the point of sample_weights?
from sklearn.metrics import balanced_accuracy_score

sample_weight = np.array([1 if i == 0 else 1000 for i in y])
balanced_accuracy_score(y,m.predict(xs),sample_weight=sample_weight)

Here are the docs.


texts: when will the mlp give constant prediction?

texts:
I have a regression task.(to predict price for finanical market)
I build a mlp to do the regression.
I found mlp will stop at giving a constance prediction. which i think it's useless.
Does this mean, i cant do a better prediction than constant value? but i tried catboost, it give a meaningful prediction.
So, if the mlp give the constant value, is there any common reason? and how to improve that?


texts:
I've just read Geoff Hinton's paper on transforming autoencoders

  Hinton, Krizhevsky and Wang: Transforming Auto-encoders. In Artificial Neural Networks and Machine Learning, 2011.

and would quite like to play around with something like this. But having read it I couldn't get enough detail from the paper on how I might actually implement it.

Does anyone know how the mapping between input pixels to capsules should work?
What exactly should be happening in the recognition units?
How it should be trained? Is it just standard back prop between every connection?

Even better would be a link to some source code for this or something similar.


texts: Inverse Relationship Between Precision and Recall

texts:
I made some search to learn precision and recall and I saw some graphs represents inverse relationship between precision and recall and I started to think about it to clarify subject. I wonder the inverse relationship always hold? Suppose I have a binary classification problem and there are positive and negative labeled classes. After training some of the actual positive examples are predicted as true positives and some of them false negatives and some of the actual negative examples are predicted as true negatives and some of them false positives. To calculate precision and recall I use these formulas:
$$Precision = \frac{TP}{TP + FP}$$ and $$Recall = \frac{TP}{TP + FN}$$ If I decrease false negatives then true positives increases and in that case don't precision and recall both increase?


texts:
I have a dataset as training input with this shape:

First column: is a random word.
Second column: is the sum of its ASCII characters.
Third column: LeftLimit of the current Bin.
Fourth column: RightLimit of the current Bin.
Fifth column: is a bin that is used as a label for this word.

For example, bin $0$ is a label for every word that its sum of ASCII characters is between $117$ to $140$.
I want to create a neural network that after training to predict the label (bin) of every word of a huge file that I give.

What type of neural network do you propose me to use?


texts: What should be the labels for subword tokens in BERT for NER task?

texts:
For any NER task, we need a sequence of words and their corresponding labels.
To extract features for these words from BERT, they need to be tokenized into subwords.

For example, the word 'infrequent' (with label B-count) will be tokenized into ['in', '##fr', '##e', '##quent']. How will its label be represented?

According to the BERT paper, "We use the representation of the first sub-token as the input to the token-level classifier over the NER label set".

So I assume, for the subwords ['in', '##fr', '##e', '##quent'] , the label for the first subword will either be this ['B-count', 'B-count', 'B-count', 'B-count'] where we propagate the word label to all the subwords. Or should it be ['B-count', 'X', 'X', 'X'] where we leave the original label on the first token of the word, then use the label “X” for subwords of that word.

Any help will be appreciated.


texts: K-Fold cross validation and data leakage

texts:
I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the remaining k-1 folds as training set. Now I want to do feature scaling and data imputation on that training set and then apply the same transformation on that validation set. I want to do this for each step. I am trying to avoid data leakage as much as possible and at the same time rescale my validation sets to get better results.
How can I do this with a few lines of code?
Secondly, is it necessary to do this? Because I don't see many people do this for k-fold validation. I have seen many times, they do feature scaling and imputation on the entire dataset first and then do the k-fold cross validation. But doesn't this cause data leakage?


texts: Is there a known convolutional net architecture to calculate object masks for images?

texts:
I would like to train a convnet to do the following:

Input is a set of single channel (from black to tones of grey to white) pictures with a given object, let's say cars.
Target is, for every picture in the set, the same picture, however pixels are either black or white. The pixels corresponding to the car object are in white (i.e. intensity 255) and the pixels corresponding to the background are black (i.e. intensity 0).

After training I would like to feed the net with pictures of cars and I would like the prediction - the ideal prediction at least - to be a picture with pixels either black or white, where white corresponds to the object and black to the background.

I assume that the input layer is a 2D convolutional layer and the output layer is also a 2D convolutional layer, each one with as many neurons as pixels in the pictures.

Can anyone please explain what kind of network architecture would accomplish just that?

It could be either the architecture (in theory) or implemented in code.

I expect to tweak it, but it would be nice not to start from scratch.


texts: Are euclidian vectors and unit vectors same thing?

texts: Consider this statement : Let the field K be the set R of real numbers, and let the vector space V be the Euclidean space R3. Consider the vectors e1 = (1,0,0), e2 = (0,1,0) and e3 = (0,0,1). Then any vector in R3 is a linear combination of e1, e2 and e3.


texts: detect diagram region in research papers

texts: How can I detect diagram region and extract it from a research paper


texts: Why should I know C++ ,if I am a machine learning engineer?

texts:
I see there's a lot of machine learning job openings with skills requirements, python ,R, keras,tensorflow,pytorch,spark, etc.which are completely fine & reasonable, but why many of the recruiters include C++ ,like what is use of C++ in ML research, or even creating ML pipelines ?

How much C++ should I know if I'm good at rest of ML skills ?


texts: AlphaGo (and other game programs using reinforcement-learning) without human database

texts:
I am not a specialist of the subject, and my question is probably very naive. It stems from an essay to understand the powers and limitation of reinforcement learning as used in the AlphaGo program.

The program AlphaGo has been built using, among other things (Monte-Carlo exploration of trees, etc.), neural networks which are trained from a huge database of human-played go games, and which are then reinforced by letting play versions of the program against itself many times.

Now I wonder what would happen is we tried to build such a program without human database, i.e. starting with a basic program of Go just knowing rules and some method to explore trees, and letting play against itself to improve its neural network. Will we, after many games against itself, arrive at a program able to compete with or beat the best human players? And if so, how many games (in order of magnitude) would be needed for that? Or on the contrary, will such a program converge toward a much weaker player?

I assume that the experiment has not been made, since AlphaGo is so recent. But the answer may nevertheless be obvious to a specialist. Otherwise any educated guess will interest me.

One can also ask the same question for "simpler" games. If we use roughly the same reinforcement-learning technics used for AlphaGo, but with
no use of human database, for a Chess program, would we eventually get
a program able to beat the best human? And if so, how fast? Has this been tried? Or if not for Chess, what about Checkers, or even simpler games?

Thanks a lot.


texts: Monitor Model Training Progress over HPC Clusters

texts:
As a part of my research in Deep Learning, I have to frequently train models which require a lot of computing power. As such, I use my university's HPC environment to submit my jobs and to train my models.

However, I run into one major issue - MONITORING THE TRAINING PERFORMANCE & METRICS!

I generally build my models with Keras, and it is convenient to check the console from time to time to get to know about the model training/performance.

There's a tool - CometML, which I use when I train models on my own system. However, as the HPC does not allow socket connections, it's not possible to monitor.

Is there a way/tool which can be used to monitor the metrics?
For now, I take a dump of the logs from time-to-time and download them into my system and then check. But it's extremely time-consuming and inefficient.

If there is an efficient way/tool, please let me know.

Thanks.


texts: Classification model to classify large number of classifiers?

texts:
Hello I am  very new into the field of machine learning/deep learning , and I am finding it hard to select the right model for my research.

What I am trying to build is a model to classify which subway route a user have used based on travel time and transfer time given the origin station and destination station.

Here is a description of my data set.

   BSEC BSTN ASTN1 BSTN2 ASTN2 BSTN3 ASTN3 BSTN4 ASTN4 BSTN5 ASTN TFtime Ttime
 69551 1001  1703  1703     0     0     0     0     0     0 1003    399  2933
 69664 1001  1703  1703     0     0     0     0     0     0 1006    399  2284
 66606 1001  1703  1703     0     0     0     0     0     0 1701    118  1750
 66600 1001  1703  1703     0     0     0     0     0     0 1701    118  1750
 66601 1001  1703  1703     0     0     0     0     0     0 1701    118  1750
 69434 1001     0     0     0     0     0     0     0     0 1703      0  1005

ASTN1,BSTN2,ASTN2...BSTN5 refers to via stations
BSTN ASTN refers to boarding and arrival stations.

I have a another data set of route information labeled.

The problem starts here.

I am trying to build a model that can classify which route a user have used given BSTN, ASTN, and time info BSEC, TFtime, Ttime.
There are too many labels of routes because the routes all differs for each pair of Origin Station and Destination Station.

Below are number of routes per origin station and destination station

   BSTN  ASTN trips
  <dbl> <dbl> <int>
1   150   152     3
2   150   153     7
3   150   154     2
4   150   156     2
5   150   157     2
6   150   158     4

as described there are already 20 different routes for only 5 Origin Destination pair. There are total of 109,425 pairs of origin and destination
and 236,213 number of routes. I could not give label to every 236,213 routes for the model to classify.

i tried making random forest model for every pair of Origin Destination pair.
But I was not able to tune or interpret them because there are too many type of models.

What would be a proper model for my situation?
Would there be a way the model could interpret given OD pair, and then perform classification within the Origin Destinatnion pair assembly?

I would really appreciate some advice or help.


texts: Is it possible to change the input columns of a trained ML model while making predictions from it without affecting the accuracy?

texts:
Consider the following scenario.
I have trained a K-Means model on some input features, say, (A, B, C, D and E).
Now at the time of making predictions I want to make the model predict using only fewer number of features e.g. (A, D, E) as opposed to the number of features it was trained initially i.e. (A, B, C, D and E).
With the understanding of above facts I have some questions which are as follows :-

Is this approach a correct approach, or logical with respect to machine learning principles ?
Will it affect the model accuracy and if yes then how ?
If I have to provide features B and C, then can I populate them with zeros and then provide it to trained model for making predictions.
Will the action taken in step 3) affect the model accuracy, if yes and then why and how ?


texts: Orange3: Is it possible to get the total number of rows for a given column from inside 'Feature Constructor'?

texts:
I'm using the Orange3 gui and have been asked to generate basic plots where values relative to a "mean" or  "Percentile Rank" are colorized.

I looked in Feature Constructor, but as far as I could tell it lacks a way to calculate the total number of rows, or instances of incoming data.

I've looked elsewhere for widgets that might help and stumbled upon Box Plot, however, if one attaches a Data Table to the output of Box Plot, we can see that no new columns contain the values of the mean or standard deviation are generated.

Is there some widget that has a function that can output these values, or the total number of rows selected / in the data set so that I can write my own function?


texts: CNN combined with a competitive search algorithm

texts:
I'm reading some papers about Deep Neural Networks applied for board games, like for Go with AlphaGo, AlphaGo Zero and some other games, like Othello and Chess. Most of the works are using CNN's as a direct move predictor. I want to make some experiments using Tic Tac Toe to implement a CNN, but I want to combine Deep Learning with a competitive search algorithm as Minimax or Alpha-Beta Pruning (CNN will be a "heuristic function" or evaluation function to the search algorithm). In this way, each leaf node generated by the search algorithm will be a input to the CNN, which will output a numerical value between -1 and 1 (values next to -1 means bad position and values next to 1 means good position) and return to the search algorithm. So, such approach makes sense? Is there anybody who tried something like this?

Thanks!


texts: Classify pdf files - image approach vs. text approach

texts:
I'm about to start a project with the objective to classify PDF-documents. I'm wondering if there's a best practice approach to tackle this problem.

Concretely I'm wondering if one of the following two approaches performs usually better:

use an OCR reader to convert the files to text and train the classifier model on the text data
convert the files to images and train a CNN classifier model

I'm planning to classify different testimonials and certificates mostly.
Since most of these files share a similar layout and text within the classes the ideas should both work out. I'm wondering if anybody has already made experience with this and could tell me about some advantages/disadvantages when using a specific approach.

I'd highly appreciate any kind of help.


texts: How to use HMMs for continuous value prediction

texts:
I have some time-series data, which I need to use to predict a continuous value for a given time-stamp. I was initially doing it using a Multivariate Regression Model but I later figured that a time-series based problem could be better solved using Hidden Markov Models.

The dataset consists of a time-stamp label, around 30 features collected from IoT sensors and then there is one target class which is a continuous variable. The problem is how do I determine the structure of the HMM given these features and number of hidden states to represent the high dimensional data and once the above is done, how to predict the continuous target value using the set of transition probabilities and model parameters that get generated.

I have gone through references where the structure of the HMM is specified beforehand which makes it easy to work with, but how do we go about predicting a continuous value with a HMM?


texts:
I have a training set where each row is a series of user actions on a website (logged in, sent an invoice, etc.) and times deltas in ms between these actions. Each row has a label — a corresponding user class (10 classes):

action132, 2340, action21, 300, ...  3
action238, 1240, action22, 350, ...  6
...
action763, 1240, action42, 750, ...  2

Say, I have 500 unique action types and each row has a variable length. Max sequence length is 100k and there are 10k rows in the training set (10k users).

I need to predict labels for the test set.

If I had only the action sequence, it would be a more understandable task similar to DNA sequence classification of text classification which are normally solved with LSTM/CNN models or transformers. But in my problem these time deltas are good indicators of user behavior and I'd like to make use of these features. Also, the action order is important as well.

What would be an efficient model for this problem?


texts:
I was starting to look into area under curve(AUC) and am a little confused about its usefulness. When first explained to me, AUC seemed to be a great measure of performance but in my research I've found that some claim its advantage is mostly marginal in that it is best for catching 'lucky' models with high standard accuracy measurements and low AUC.

So should I avoid relying on AUC for validating models or would a combination be best? Thanks for all your help.


texts: How do I test and validate a Markov Model in RStudio?

texts:
I've created a Markov Model in RStudio using the seqHMM library. I'm new to R and would usually do this kind of thing in Python, but I wanted to use seqHMM. I can see the model has been created and it plots successfully. However, how do I run some test / validation data through it (some more sequence data)?
library(seqHMM)
library(TraMineR)
print("Sequence prediction")

house_seq <- read.csv(file = "train_test.csv", header = TRUE)
# Print the first 3 observations
head(house_seq, 3)
dim(house_seq)

# Define a color palette for the sequence data
attr(house_seq, "cpal") <- colorpalette[[6]]

# Estimate the Markov model
mm_house <- build_mm(observations = seqdef(house_seq$noun))
plot(mm_house)


texts: What are the differences between Convolutional1D, Convolutional2D, and Convolutional3D?

texts: I've been learning about Convolutional Neural Networks. When looking at Keras examples, I came across three different convolution methods. Namely, 1D, 2D & 3D. What are the differences between these three layers? What are their use cases? Are there some links or references to show their use cases?


texts: Predicting continuous value from Image with CNN (pytorch)

texts:
I'm having problem when predicting continous values from image.
As you can check on csv image below, I want to predict continuous value from .jpg image
such as like (0.84, 200,0.00001 )

but I'm having problem with make it as linear regression with 1 output dimension, the predicted value become only 0 as shown on image

here's my network architecture
and the way how I normalized the label data with MinMaxscaler
scaler = MinMaxScaler()
#the way how I normalized the label values (0.84,0.98,0.90.... etc)
all_labels_1 = csv_data.iloc[:, 3]
all_labels_1 = all_labels_1.values.reshape(-1,1)
all_labels_1 = scaler.fit_transform(all_labels_1)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.layer1 = self.conv_module(3, 16)
        self.layer2 = self.conv_module(16, 32)
        self.layer3 = self.conv_module(32, 64)
        self.layer4 = self.conv_module(64, 128)
        self.fc1 = nn.Linear(128*8*8, 4000)
        self.fc2 = nn.Linear(4000, 2000)
        self.fc3 = nn.Linear(2000, 1)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)    #the way how I normalized the label values (0.84,0.98,0.90.... etc)
all_labels_1 = csv_data.iloc[:, 3]
        all_labels_1 = all_labels_1.values.reshape(-1,1)
        all_labels_1 = scaler.fit_transform(all_labels_1)
        out = self.layer4(out)

        out = out.view(out.size(0),-1)

        out = self.fc1(out)
        out = self.fc2(out)
        out = self.fc3(out)

        return out

    def conv_module(self, in_num, out_num):
        return nn.Sequential(
            nn.Conv2d(in_num, out_num, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(out_num),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))

and this is how I trained
To check model works well, I just trained with label_1, which has 0.84,0.98.0.90.... values
custom_model = Net().to(device)

# Loss and optimizer
MSE = nn.MSELoss()
optimizer = torch.optim.Adam(custom_model.parameters(), lr=hyper_param_learning_rate)

for e in range(hyper_param_epoch):
    for i_batch, item in enumerate(train_loader):
        images = item['image'].to(device)
        labels_1 = item['label_1'].to(device)
        labels_2 = item['label_2'].to(device)
        labels_3 = item['label_3'].to(device)

        # Forward pass
        outputs = custom_model(images)
        loss = MSE(outputs, labels_1.float())

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i_batch + 1) % hyper_param_batch == 0:
            print('Epoch [{}/{}], Loss: {:.4f}'
                  .format(e + 1, hyper_param_epoch, loss.item()))

Is there anything that I'm missing? I can't find any example for predicting continuous values from image file with CNN


texts: Is there a representation of the separating hyperplane in t-sne?

texts:
I have used t-sne to visualize a set of images which I have used for training a binary classifier. Let us assume that the binary classifier is trained to detect cat(1) vs. no-cat(0). I have used the final features (last hidden layer) of 64 values to run the t-sne algorithm. Following image shows the output:

The color shows the probability (red showing towards 1, blue showing towards 0) of the trained model and the shape shows if the input image is cat or not (triangle showing cat, circle showing not cat). So, red circles and blue triangles are false detection and missed detection respectively.

Now can we visualize the hyperplane that separates the 64-dimensional space in the t-sne output space (i.e., 2-D space)? From the image it looks like the red and the blue points can be separated by a crooked open curve. Can we plot as well? If we cannot plot it, is it correct way to think that a crooked curve is the true representation of the hyperplane?


texts: Train, test and submission files - what am I supposed to do with all of them?

texts:
this might be very beginner's question.
I'm working on Kaggle's HomeCredit Default Risk problem which has among others dataset train, test and submission files as can be seen in the link provided. The test dataset does not contain TARGET feature and submission has ID and TARGET columns where TARGET has fixed value 0.5.

I'd normaly split the train dataset and used it to train and score a model and than created a submission file based on the test dataset.
Thus, I don't understand what I am supposed to do with the already existing submission file.

Futher, when I score a model based on splitted train data, I'm getting unexpected high values which can't be right. I did before OHE-encoding, standardization and PCA, it that might have some impact on the model accuracy result.

Insights would be apprecited.


texts: What are the best ways to use a time series data for binary classification

texts:
I have large number of csv files and each of them are timeseries based csv files sampled at Avery 5 seconds for 2-3 mins. I have 20k such files with 200-300 variables in each file.
I am aggregating the data by mean over the entire 2-3 mins window are using it for binary classification.

Currently I am using mean of each column in the .CSV file to represent that file, so basically I am summarizing the csv's using one scalar value per column.
 so each file is one sample represented by its respective mean value.
Could anyone suggest me some better ways to summarize the timeseries data.

Thanks for your time.


texts: Possible harm in standardizing one-hot encoded features

texts:
While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)?
Standardizing definition:  applying (x - mean) / std  to make the feature mean and std 0, 1 respectively)
I prefer applying standardization to my entire training dataset after one-hot encoding, rather than applying it only to the numerical features. I feel it would significantly simplify my pipeline.
For example, if I have a binary feature then the vector that will be provided to the model is [1,1,0,0,0,1,1].
If standardization is applied to this binary feature prior to fitting the model (subtract mean = ~0.57 and divide by std = ~ 0.49), the vector will become
[ 0.8660254 ,  0.8660254 , -1.15470054, -1.15470054, -1.15470054, 0.8660254 ,  0.8660254 ]


texts: How do I get a word frequency list? Can't export from Word Cloud

texts:
I have a list of survey responses and want to analyze the frequency of words used.

Using Orange, I've loaded the file into a Corpus, then Preprocess Text, then Word Cloud.

The Word Cloud does not show the cloud image, and I can't figure out why. Everything appears to be set up just like the Orange documentation shows online. Word Cloud does show a nice list of words and their frequencies, but I can't copy/paste that into Excel or export that list.

How can I export that list of word frequencies into a table where one column is the word, and the other column is the count? Is there an easier way to do this?


texts: Questions When Advancing from Vanilla Neural Network to Recurrent Neural Network

texts:
I've recently learned how a vanilla neural network would work, with given number of inputs, hidden nodes, and the same number of outputs as inputs.

I've been looking at various posts now related to recurrent neural network, and I understand the concept behind it, but I fail to understand certain parts of the logic behind the RNN.

Here are four main questions I have:

How does back-propagation work in recurrent neural network?
Are the weights that lead from the input to the hidden nodes the same for every other step? What about the weights from the hidden nodes to the outputs?
How do biases exactly work in a recurrent neural network?
Why is tanh function usually used instead of sigmoid function as the activtion function?

I realize some of these questions are very basic, but I guess the basics are exactly what I need right now.

Even links to related videos or posts will be extremely helpful, and so will google keywords that show the right results. These four questions are hindering me from understanding sample Python codes, so I really need some help.


texts:
I have made a TensorFlow model and got the training part working.
The model trains successfully and the prediction while training is accurate.
Unfortunately, when using the model to only predict, the answer is mostly gibberish and sometimes even blank.

This is the code:

def train_model():
    loss_track = []
    max_batches = 10001
    batches_in_epoch = 1000
    try:
        for batch in range(max_batches):
            fd = next_feed() # This is a wrapper that prepares the correct response (decoder_targets) and returns batch_method(query_list) (batch method returns encoder_inputs, encoder_input_lenghts)
            _, l_t_a = sess.run([train_op, loss], fd)
            loss_track.append(l_t_a)

            if batch == 0 or batch % batches_in_epoch == 0:
                #fd = random.choice(list(fd))
                #print(fd)
                print('batch {}'.format(batch))
                print('     minibatch loss: {}'.format(sess.run(loss, fd)))
                predict_ = sess.run(decoder_prediction, fd)

                for i, (inp, pred) in enumerate(zip(fd[encoder_inputs].T, predict_.T)):
                    index = random.randint(0, len(fd[encoder_inputs].T)-1)
                    print('     sample {}'.format(i + 1))
                    print('     Query                  > {}'.format(batch_to_words(fd[encoder_inputs].T[index])))
                    print('     Predicted Response     > {}'.format(batch_to_words(predict_.T[index])))
                    if i >= 2:
                        break
                print()
        saver.save(sess, MODEL_LOCATION)
    except KeyboardInterrupt:
        print('Training interrupted.')

def test_model(sentence):
    saver.restore(sess, MODEL_LOCATION)
    batch__ = []
    query = get_formatted_sentence(sentence) # This returns the array of the sentence. ex. ['hello','world']
    batch__.append(
      words_to_batch(query) # This is the inverse of batch_to_words(), it transforms the sentence array in an int array, based on the word vocabulary position
    )
    batch, batch_len = batch_method(batch__)
    prediction = sess.run(decoder_prediction, feed_dict={encoder_inputs: batch, encoder_inputs_length: batch_len})
    batch_response = []
    for i in prediction:
        batch_response.append(i[0])
    print(batch_to_words(batch_response))

What is wrong here? For the prediction part I insert the query inside another list as batch_method() requires a list.

Keep in mind that for some reason the prediction in the training looks like this: [12,5,3,5,1] (and that is why I am able to call batch_to_words() directly), while in the prediction only mode it returns something like this: [[3],[4],[64],[23],[1],[0],[1]] (Which is very weird)

I also call train_model() or test_model() separately, never both on the same run (that is why I save the model -- as always, I am not sure if I am doing it correctly).

I am not sure at this point if this is to be expected or if I messed up somewhere.

Any help is appreciated.


texts: Rank feature selection over multiple datasets

texts:
Through backward elimination I get a ranking of features over multiple datasets. For example in the dataset 1 I have the following ranking, the feature in the top being the most important:

feat. 1
feat. 2
feat. 3
feat. 4.

...

, whereas for dataset 2 I have for example the following ranking:

feat. 3
feat. 1
feat. 2
feat. 4.

I want to filter out those features which end up in the top of the ranking the most (incorporating that finishing in the top is better than finishing in the 3rd place). Which kind of ranking metric can I use for this problem?


texts: When the generator fit calls the "fit_generator" function that i've created

texts:
To understand better how the fit generation works I want to know if the "fit generation function" is called only once per epoch (And in each epoch is trining to minimize the gradient of a given dataset) or once per step (and in each step inside the epoch it generates a new dataset).
For what i understand, i think is the second aproach but if someone can confirm as i have done a huge research and I didn't found anything.
Thank you in advance,


texts: how do I predict the next's alarms ? (time series)

texts:
I'm trying to solve a time series forecasting problem, where the main goal is to read data with various alarm logs and make a prediction about what may happen in the future.
Specifically, my data is a stream of alert data, where at each time stamp, information such as the alert monitoring system, the location of the problem etc. are stored in the alert. These fields are all categorical variables.
If someone already had an identical problem, it could help me find the best technologies and which ML models are most used in these cases and some practical examples of models.
I need to predict multiple occurrencies of future alarms. In this case the response variable is problem name.

**I know that in a pre-processing phase I have to encode the variables.
EDIT:
my goal is to have a multi-step output of the "problem name" label. This label is not numeric or percentual, its a string (categorical variable). i guess i will have to feed a model with data resembling something like this (after noramalization):

I guess that for this problem I will have to use LSTM models but my problem is that all the examples I see are trying to predicte continuous values, such as temperatures or stock values.
Are any models (and special layers) targeting this kind of problems?
Any good examples available online?


texts: Keras LSTM model for binary classification with sequences

texts:
I'm currently working on a bigger project.
The goal is to automatically find split points in time series which splits the series into elementary patterns.

I have a lot of training data in form of time series with different lengths and split points manually recorded on useful positions.
Basically, I have the x, y, z positions and the distances between those points over a centered window of the lengths 2, 25 and 50 (dist2, dist25, dist50). I also divided the room into a 3D grid and figured out in which cell the position lies (cell_x, cell_y, cell_z). I think this could be useful because elementary operations will more likely be in one or two cells.

cell_x, cell_y, cell_z based on size 50. The vertical red lines are the split points of the training data.

Now I want to know at every point in the time series if it is a split point or not.

In my opinion, these are enough features to start with but I think my Keras model isn't correct because the result is always and only 1 at the 0's time series index. It's basically, a binary classification problem based on past and future values. This was the reason I've tried to solve it with an LSTM. It looks like this:

model = Sequential()

model.add(LSTM(20, input_shape = (None, input_dim), return_sequences = True))
model.add(Dropout(0.5))
model.add(LSTM(20))
model.add(Dropout(0.5))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(loss='binary_crossentropy',
            optimizer='adam',
            metrics=['accuracy'])

Because I'm at the very beginning of the whole Machine Learning topic and the mass of materials is a bit overwhelming I'm here to ask for some help. I know that there are a lot of other things (bad training data, wrong features, wrong parameters, ...) which can cause to this results but I want to know how I can build a Keras model for this problem correctly. I already found a lot of Keras examples but I'm not sure if they fit the problem.

Example of data: Features and label of one record: gist.github.com/chryb/2b3312b0ab640a1f28c8cb2d3dbbd86e | The column is_keyframe represents the labels for each index. Based on a Stackoverflow question I transformed the data with the following to the right LSTM input format: gist.github.com/chryb/3de34ba44488012bf79ff4c51a2a43ae


texts: Prediction with non-scalar output (label)

texts:
I have recently confronted with a (at least for me) new kind of ML problem, where the output of the model should be a vector/matrix (depending on the interpretation, but there is no difference actually), not a scalar as usual. This is totally unknown for me. What kind of approach should one apply here? Are the "usual" (scalar-based) models applicable on this problem?

(Just for the sake of completeness, the problem is an image segmentation task where the model should decide first: if there a given pattern on the picture?, second: if so, where is it? - In latter case, it should define the borders of the subset pixels).


texts: In Federated Learning what happens when mobile devie is not suitable available?

texts:
I've been doing some research about federated learning and came across two questions that I want to clarify. I've been reading and watching videos however no luck and also there aren't questions on stackExchange.
In federated learning each mobile devices gets model distributed if they are suitable available.
Device is said to be Available if:

Idle (not lot of thing going on)
plugged in
on wifi

and it is suitable Available if device has sufficiently large data needed for training.
My question is what happens when at the time of training my device is NOT suitable available?
Taking Gboard for example, when I am trying on Gboard it recommends query using federated learning. When I am trying on Gboard I need to get model from cloud and train it as I am typing however in case I am not suitable available what happens?


texts: Age range prediction according to browsing history

texts:
I need to predict age or age range of a user according to his/her browsing history from a news website. The only data I am able to get is USER-ID, URLs-visited, TAGs from those URLs, "which category news a person reads most" etc. My only option seems like creating this dataset somehow. But I do not know how do I decide the age range of a person. The problem is if a person is reading a politics news he could belong to an age range of 50 and above but then what if this is a high trending news and everybody is reading it. So basically how do I decide which age range people read which kind of news most?

I hope I am able to explain what my problem is. If somebody has worked on such a problem I would like to hear how you proceeded with it.
Thanks.


texts: Normalisation of features extracted from audio files

texts:
I am building CNN and SVM models which take in MFCC features as input. The MFCC matrices shape is (13, n). The 13 rows are coefficients and n columns represent n time frames. So each row in the matrix is a representation of the value of the particular coefficient over different time frames. I am not sure how to normalise this matrix. Should it be rowwise (normalize a single coefficient over all the time frames) or columnwise (normalize all the coefficients in a single time frame).
[
  [1.08,  8.97, 78.7, ........],
  [1.08,  8.97, 78.7, ........],
  [1.08,  8.97, 78.7, ........],
   .
   .
   .
  [19.8,  7.65, 76.5, ........]
]

Currently I am using Normalize from sklearn, but I am not sure if its the right thing to do.


texts:
I've been trying to extend the LDA and wanted some help, direction and insight.

Can Author-Topic LDA be used as a document "category" model? The premise of the Author-Topic model is that multiple authors influence the Topic from which every word is derived. Can I instead treat "authors" as categories to which documents belong?

I've been trying to slightly alter this:

http://yaowu.co/docs/sigir12.pdf

and this

http://papers.nips.cc/paper/2994-modeling-general-and-specific-aspects-of-documents-with-a-probabilistic-topic-model.pdf

with Author-Topic model.

What I'm trying to do is find out document-level word associations in the posting-data of a forum on a specialized site. For example, this site is about Candy - its members post about sweets, candy and toffee. Obviously there are categories of candies, then brands and then the products themselves. My reasoning is that if I use Author-Topic and treat the observed node "author" as observed-node "candy categories", then I can introduce that candy category feature into recommender system down the pipeline.

Like the papers above suggest, a "word stream", switched by a multinomial 'x' decides which stream a word comes from (background, document or 'candy category').

Do you think this makes sense?

Furthermore, is there a book or resource that, very thoroughly and step-by-step explains the Collapsed Gibbs derivation for LDA?


texts:
I have few thousand audio signals to label into 2 different classes and save them to numpy array for further training of models. MATLAB recently released Signal Labeler for their Signal Analyzer, that could help to label time series, but for certain reasons, I can't use it.

Is there any specific tool for analysis and labeling of time series for Python? It is not necessary to save data and labels into numpy arrays, .csv format or anything similar is suitable as well.

Will appreciate any help!


texts: Is decision tree regression comparable to locally weighted regression

texts: I am new to decision tree method. For decision tree regression model, does it just fit a piece wise step function over data? When and why would people prefer it over some traditional regression like locally weighted (LOESS) regression?


texts: I have historic error of time series.  I want to analyze error series to improve forecast series. Are there any methods to do this?


texts: Mahalanobis distance between two clusters

texts:
I want to calculate the Mahalanobis distance between cluster $a$ and cluster $b$, each consisting from a set of multidimensional points. Assuming no correlation, calculating the distance between a random point $p$ and cluster $b$ can be achieved using the normalized Euclidean distance formula:

$$ d(p,b)=\sqrt{ \sum_{i=1}^d \frac{(p_i-b_i)^2}{\sigma_i^2}}$$

where $d$ is the number of dimensions and $\sigma_i^2$ is the squared standard deviation for each dimension in cluster $b$.

Now, I want to estimate the Mahalanobis distance between clusters $a$ and $b$. Should I assume that cluster $a$ is a single point (i.e. $a$'s centroid)? Or should I normalize using the standard deviation of both clusters? In the latter case that would translate in the following formula:

$$ d(a,b)=\sqrt{ \sum_{i=1}^d \frac{(a_i-b_i)^2}{(\sigma_i^a)^2 \cdot (\sigma_i^b)^2}}$$

where $\sigma_i^b$ is the standard deviation for dimension $i$ in cluster $b$ and $\sigma_i^a$ is the standard deviation for dimension $i$ in cluster $a$.

Thank you in advance.


texts:
The typical pipeline in ML is

Find a data-related problem that you want to solve
Build a model or algorithm that feeds on data related to the problem to try to solve the problem
Check if the solution produced by the model or algorithm is satisfactory by testing the prediction against another piece of data

My question is: given the vast amount of models, algorithms, and data currently available to use, e.g. on repositories like github etc, are there any current efforts that follow an inverted pipeline as follows?

Choose an existing model or algorithm or create a new one
Test it against many different types of data sets (as long as input output formats are compatible
Find problems for which the model or algorithm is good at, even if it was originally not design for this domain, or designed with any purpose at all

In other words, systematically test existing algorithms, or a new algorithm, against existing datasets to find triples of (algorithm, question, data) that work well and have not been thought together before.
I would image this to be increasingly easy to automate using things like autoML and etc.
Just to clarify: I'm not so much looking for a list of reasons why this is a bad idea, or why it can, or cannot be done (although I will take advice if given and good). I just want some references of people or papers doing something similar to this, if any. A reference that partially follows the inverted pipeline is better than nothing.
Thank in advance


texts: Training the document page layout and classifying good/bad layouts

texts:
I have a use case where I am supposed to get the coordinates of each block element in a page (whether its paragraph, image, table) where I train a model to understand how they are placed in a given page where some documents are with good layout and other with bad ones and I want to train this and throw in some coordinates of a new document and try to understand whether it has a good layout or a bad layout, I want to understand how I can achieve this using some deep learning techniques ?

can someone suggest me an approach for solving this?

Was trying to workout with RNN but not sure if that's the correct approach.


texts: Select more or less features if results are almost the same

texts:
I am having a dataset of 3500 observations with 70 features each with binary labels/targets for classifications purposes.

My aim is to score more than 90% precision and the highest recall possible for this precision.

I have tested many algorithms and thus far Random Forest performs the best.

Specifically, with 70 features I am getting:

Auroc: 0.71
Precision: 0.94
Recall: 0.18

However, I had a look at the importance of my features by calling the built-in function feature_importances_ of the RandomForestClassifier of SkLearn and by also using the Boruta algorithm with the boruta_py module.

I removed the 20 least important features. The number 20 was quite arbitrary because even the next 10 least important features were relatively close in terms of importance to these 20 least important ones. Only the top 10 most important features had a significant difference in importance in comparison with the others.

Then the results with 50 features were the following:

Auroc: 0.7
Precision: 0.91
Recall: 0.17

Since I can get almost the same results with less features should I remove these features?

Please also into take into account that in the future my dataset will be expanded to 5000 observations.


texts:
I have a dataset with a message (string) and an associated mood. I am trying to use an ANN to predict one of the 6 moods using the encoded inputs.
This is how my X_train looks like:
array([list([1, 60, 2]),
   list([1, 6278, 14, 9137, 334, 9137, 8549, 1380, 7]),
   list([5, 107, 1, 2, 156]), ..., list([1, 2, 220, 41]),
   list([1, 2, 79, 137, 422, 877, 5, 230, 621, 18]),
   list([1, 11, 66, 1, 2, 9137, 175, 1, 6278, 5624, 1520])],
  dtype=object)

Since every array has a different length, it's not being accepted. What can I do about it?
PS: The encoded values were generated using keras.preprocessing.Tokenizer()


texts:
How can one automate the extraction of a relevant title from a given document (docx, pdf, etc..)?

Some thoughts:

Intuitively likely to be the first line in the text
Cannot be something like a date, "all rights reserved", "Harvard University - Faculty of Law" etc.
Might be in the footer
The human brain solves this immediately (by filtering out common names and numbers)

I'm using textract in python to extract the whole text, which limits the solution to be based on text only. Let me know if you are familiar with any other packages.


texts: I am new to Data Science and I am trying to understand the 'Word2Vec' approach from a long time. Can someone please explain it in simple terms? Moreover, what are the problems that can be solved through 'Word2Vec' approach?


texts: Time Series Forecasting with Neural Network (mixed data types)

texts:
I have a dataset with the following format:

TimeStamp  | Action  | UserId
2015-02-05 | Action1 | XXX
2015-02-06 | Action2 | YYY
2015-02-07 | Action2 | XXX
...

I try to forecast future Actions for specific users based on the Users history in the dataset. I started with ntstool in MATLAB, but it can't handle mixed data types or non-numerical values. Now I am looking for other methods to predict future Actions and find periodic patterns in the records.

Is it possible to convert the values for Actions to numeric values or is there a possibility to create a neural network with mixed datatypes on the input? Maybe in R?


texts: Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

texts:
I am running the Kaggle Video Games sales dataset through an XGboost algo.

I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5

when I use:
    data = pd.get_dummies(data=data, columns=['Game_Rating])

pandas adds a column to my dataframe for each unique observation in Game_Rating

Rating_E  Rating_E10+  Rating_EC  Rating_K-A  Rating_M  Rating_RP  Rating_T
1            0          0           0         0          0         0
0            0          0           0         0          0         0
1            0          0           0         0          0         0
1            0          0           0         0          0         0
0            0          0           0         0          0         0

My question is two-fold:
Does Pandas have the ability to replace categorical values of a dataframe inline, preserving the original column structure

if not:

I have a (slower) method that does the encoding inline and preserves the column structure of the dataframe.  I am trying to keep the dataframe as similiar to the original dataset as possible for future processing. What are the risks I may not be seeing of using my own (inline) encoding method?


texts: Contemporary alternatives to SIFT for image feature extraction?

texts:
I've been learning about SIFT and all the ways its descriptors can be used to do different tasks.

I am particularly interested in the way SIFT can be used for image classification. (e.g. A 2006 paper by Niester & Stewenius relies on SIFT descriptors to build a vocabulary tree).

However, as of 2017, 11 years later, Deep Learning has been replacing the classical approaches in many ways.

What are some alternatives to find feature descriptors for images that, as of 2017, have shown more promising results?


texts: Where to save image features to avoid repeating extraction?

texts:
I am implementing Bag-of-Visual-Words. Part of that is SIFT, it has a task of extracting features and outputs a vector. Afterwards, the vector are clustered and made into a histogram. Histograms are then used to train for a model.
I want to train and test 100 image. Can features (vector) be saved into a separate (and what) file so that extraction steps are not repeated always?


texts: How to interpret a decision tree correctly?

texts:
I'm trying to work out if I'm correctly interpreting a decision tree found online.

The dependent variable of this decision tree is Credit Rating which
has two classes, Bad or Good. The root of this tree contains all 2464
observations in this dataset.
The most influential attribute to determine how to classify a good or bad credit rating is the Income Level attribute.
The majority of the people (454 out of 553) in our sample that had a less than low income also had a bad credit rating. If I was to launch a premium credit card without a limit I should ignore these people.
If I were to use this decision tree for predictions to classify new observations, are the largest number of class in a leaf used as the prediction? E.g. Observation x has medium income, 7 credit cards and 34 years old. Would the predicted classification for credit rating = "Good"
Another new observation could be Observation Y, which has less than low income so their credit rating = "Bad"

Is this the correct way to interpret a decision tree or have I got this completely wrong?


texts: Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?)

texts:
I am currently collecting second by second data regarding buyer vs seller initiated trades for different financial instruments (securities mostly). If there are more buyer initiated trades in a given second, then that second's data point would contain a positive value in the pertinent feature. If there are more seller initiated trades, then there would be a negative value. And if either there is an equal amount of buy vs seller initiated trades OR if there are simply not any trades in a given second, there will be a 0 for the feature in that data point. Along with this feature, there are several other features that are based on what occurred in the preceding seconds (eg if the value discussed above was 12 for the data point immediately preceding the current point, then the second feature for the current data point would be 12 - please let me know if this is not clear) After much troubleshooting, I have concluded that if there are too many data points with too many 0's for features, the classifier simply wont work. When I print out the probabilities of evaluation data points falling into different classes, I simply get

0:NaN,1:NaN

for all model evaluation points I try to classify. (I am using logistic regression from apache-mahout. In total have 183 features, but over 40million data points. There are three categories to which the data point can be classified)

I have found that if I set the default value to 1, then I no longer encounter this error, e.g. if there are no trades, the value will be a 1, if there is one seller initiated trade, the value will be 0.

So with all this in mind, I have two related questions:

1) Has anyone else encountered this issue? e.g. if you have a vector with x features, and for a majority of the data points, a majority of the features contain 0's, is this know to give issues?

2) Is shifting all values up by a constant (such as 1) a valid fix to this issue? I assume that if this constant is applied to all values, then it shouldn't skew the data, but I figure it won't hurt to check with the experts.

Also, I'm new to this, so if you believe that my question could use more info please let me know, and if you could give me ideas of what information to include, it would be greatly appreciated.

thanks in advance


texts: When exactly should I use weighted loss?

texts:
Do I have to use it in any case when the class distribution is imbalance
(Train: class A:10%, B:90% and Test: class A:10%, B:90%)

or when it is different
(Train: class A:10%, B:90% and Test: class A:50%, B:50%)

Thanks!


texts: ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

texts:
Firstly I have a pandas series of recommended product  (recmd_prdt_list). In this series there is a possibility of presence of deleted products. So as to remove deleted products from the recommended products, I did the following :

recmd_prdt_list = user_lookup['Recommended items']
recmd_prdt_list

0      PLV08, PLPD04, PBC07, 555, PLF02, 963, PLF07, ...

1      123, 345, R922, Asus009, AIMAC, Th001, SAM S9,...

2      LGRFG, LG, 1025, COFMH, 8048, BY7, PLHL4, 569,...

3      COFMH, 5454, 8048, 1025, LG, len123, Th001, PL...

4      LGRFG, AIM-Pro, 569, Asus009, PLHL3, PL04, PLH...

5      PLV08, PLF09, PLF02, PBC04, PLF07, AIM-Pro, PL...

type(recmd_prdt_list)

pandas.core.series.Series

DataFrame of product status

product_status

ItemCode  Status DeletedStatus
AIMAC     2      True
AIM-Pro   2      True
SAM S9    2      True
SH MV     2      True
COFMH     2      True
LGRFG     2      True

type(product_status)

pandas.core.frame.DataFrame

first_row = user_lookup['Recommended items'][0]
first_row

'PLV08, PLPD04, PBC07, 555, PLF02, 963, PLF07, HG8, jealous21, 4'

type(first_row)

str

Converting the str to list

first_row_list = list(first_row .split(","))
first_row_list

['PLV08',
 ' PLPD04',
 ' PBC07',
 ' 555',
 ' PLF02',
 ' 963',
 ' PLF07',
 ' HG8',
 ' jealous21',
 ' 4']

From the first row i took first itemcode to check the deleted status :

product_details = product_status.loc[product_status['ItemCode'] == 'PLV08']
product_details

ItemCode   Status   DeletedStatus

PLV08       2            False

type(product_details)

pandas.core.frame.DataFrame

product_details['DeletedStatus']

693    False

Name: DeletedStatus, dtype: bool

So as to check the deleted status of each product in the respective row and save to a new list. I wrote the following code :

itemcode = 'PLV08'
activ_product = []
if itemcode in product_status['ItemCode'].values:
    print(itemcode)
    product_details = product_status.loc[product_status['ItemCode'] == itemcode]
    print(product_details)
    if product_details['Status'] == 2 & product_details['DeletedStatus'] == 'False':
        activ_product.append(itemcode)

Error :

PLV08
     ClientId ItemCode  Status  DeletedStatus
499      2213    PLV08       2          False
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-35-9507e1ada5f7> in <module>()
      5     product_details = product_status.loc[product_status['ItemCode'] == itemcode]
      6     print(product_details)
----> 7     if product_details['Status'] == 2 & product_details['DeletedStatus'] == 'False':
      8         activ_product.append(itemcode)

~/.virtualenvs/sysg_python3/lib/python3.5/site-packages/pandas/core/generic.py in __nonzero__(self)
    951         raise ValueError("The truth value of a {0} is ambiguous. "
    952                          "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
--> 953                          .format(self.__class__.__name__))
    954
    955     __bool__ = __nonzero__

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

How to get solve of this error?


texts:
I need some help to interpret the 2 classification reports of the same logistic regression. The only difference between them is the size of test_size. Even though my second classification report has a much higher f1-score, because of its low test_size (0.1), it is an inferior algorithm than my first classification report?
I am not wholly sure how to interpret the relationship between the f1-score and support values. But from what I understand, the support values should all be close to each other and f1-score be as close to 1 as possible. In that case does that make the algorithm I developed a bad one then?


texts:
[Note : There is some serious problem with logic used to get the best banner. I got it late. Directly read the answer to get general info, or you can also try to the find the mistake.]

Problem: Given a set of user features, select an ad with the highest probability to be clicked.

Dataset - https://www.kaggle.com/c/avazu-ctr-prediction/data (First 100000 tuples from training set and split that it 80:20 training:testing)

Tutorial followed - https://turi.com/learn/gallery/notebooks/click_through_rate_prediction_intro.html

C14 is my ad id.
Problem :- Given ('device_type', 'C1', 'C15') return ad id.

Training:-

I have taken 'click' as my target and ('device_type', 'C1', 'C15', 'C14') as my input features.

I used logistic regression classifier in graphlab library to train the model.

I am doing ad selection in the following way:-
Given a set of features ('device_type', 'C1', 'C15', X) Iterate X over all possible values of C14 and pass the features to predictor to get the probability that X ad will be clicked. Return the ad with maximum click probability.

MY PROBLEM IS LOGISTIC CLASSIFIER IS ONLY RETURNING ONE AD for every test tuple, though with different click probability, it means only one ad is getting the highest probability to be clicked.

Can anyone explain this observation?

When using boosted tree classifier instead of logistic classifier for the above prediction, I am able to get different ads as my prediction and hence getting better results.


texts:
What is the influence of changing the padding value with its borders, I might miss vocabulary because I can't find many papers about this alternative. Also I'd be interested in doing this in Keras, creating a custom layer seems the most reasonable approach but why not extending the image with the appropriate values?

Actual behavior of a SAME padding (stride=6, width=5):

             pad|                                        |pad
   inputs:      0 |1  2  3  4  5  6  7  8  9  10 11 12 13|0  0
               |________________|
                              |_________________|
                                             |________________|

What about, for instance, repeating the border for circular inputs (like 360 images)? Like so:

             pad|                                      |pad
   inputs:   13 |1  2  3  4  5  6  7  8  9  10 11 12 13| 1 2
             |________________|
                            |_________________|
                                           |________________|


texts: What's wrong with my implementation of the Adaline algorithm?

texts:
I'm working through the textbook called Learning From Data and one of the problems from the first chapter has the reader implement the Adaline algorithm from scratch and I chose to do so using Python. The issue I'm running into is that the weights for my $\textbf{w}$ immediately blow up to infinity before my algorithm converges. Is there something incorrect I am doing here? It looks like I am implementing it exactly as the text describes. Below I've provided the question and my Python code. Here $\textbf{y}$ takes on the values of -1 and 1. So it is a classification problem.

import numpy as np
import pandas as pd

#Generate w* vector, the true weights
dim=2
wstar=2000*np.random.rand(dim+1)-1000

#Generate the random sample of size 100
trainSize=100
train=pd.DataFrame(2000*np.random.rand(trainSize,dim)-1000)
train['intercept']=np.ones(trainSize)
cols=train.columns.tolist()
cols=cols[-1:]+cols[:-1]
train=train[cols]

#Classify the points
train['y']=np.sign(np.dot(train.iloc[:,0:3],wstar))

#Now we run the ADALINE algorithm on the training data
#Declare w vector
w=np.zeros(dim+1)

#Column of guesses
train['guess']=np.ones(trainSize)

#s column
train['s']=np.dot(train.iloc[:,0:3],w)

#Set eta
eta=5
iterations=0
while (all((train['y']*train['s'])>1)==False):
    if iterations>=1000:
        break
    #Picking a random point
    randInt=np.random.randint(len(train))
    #Temporary values for calculating new w
    temp_s=train['s'].iloc[randInt]
    temp_x=train.iloc[randInt,0:3]
    temp_y=train['y'].iloc[randInt]
    #Calculating new w
    if temp_y*temp_s<=1:
        w=w+eta*(temp_y-temp_s)*temp_x
        #Calculating new guesses and s values
        train['s']=np.dot(train.iloc[:,0:3],w)
        train['guess']=np.sign(train['s'])
    iterations+=1


texts:
I have studied Random Forest and RainForest papers, but they are a bit confusing! In summary, I understand following steps for these algorithms. Could you help me to find out if I am right or not?

I appreciate your help.

In Random forest first:

define number of trees
partition data by bootstrapping
on each partition construct trees (in each node a sub sample of features is selected)
label leaf nodes
for classifying a  new instance vote over all trees.

In RainForest:

Partition dataset
Build AVC-set of a partition
Build tree over the partition by computing a purity criterion (like gini-index) over AVC-sets


texts: I am trying to learn data modeling by working on a dataset from Kaggle competition. As the competition was closed 2 years back, I am asking my question here. The competition uses AUC-ROC as the evaluation metric. This is a classification problem with 5 labels. I am modeling it as 5 independent binary classification problems. Interestingly, the data is highly imbalanced across labels. In one case, there is an imbalance of 333:1. I did some research into interpreting the AUC-ROC metric. During my research, I found this and this. Both these articles basically say that AUC-ROC is not a good metric for an imbalanced data set. So, I am wondering why would they be using this metric to evaluate models in the competition? Is it even a reasonable metric in such a context? If yes, why?


texts: I have noticed that most of the deep learning developers use TensorFlow. So why choose TensorFlow? What is the advantage of TensorFlow over Theano and CNTK?


texts:
I usually see convolutions performed over all the channels of the input. For example a $3x3$ kernel is really a $3x3xN$ kernel for a an input with $N$ channels, thus resulting in a single output channel per filter.

What would happen, if you were to do a convolution on individual channels or pairs of channels instead, with independent filters per channel? I am wondering if when certain channels are not correlated could you prevent some information loss by keeping the channels separate for some layers?


texts: Backpropagation with step or threshold activation function

texts:
I understand that gradient descent is local and it deals only with the inputs to the neuron, what it outputs and what it should output. In all I've seen, gradient descent needs the activation function to be differentiable, so a threshold function cannot be used.

Yet, the biological neurons either fire or they don't. The input to the neuron, in my understanding, is the equivalent of the membrane potential. Once it passes a certain threshold, the neuron fires (one or multiple times) and the input is reset. Considering this, the step function appears to be enough to reproduce the behavior of biological neurons.

I'm thinking about the usual weights (integer or even float) paired with the step function, trained with the following form of backpropagation:

looking at one neuron and its connections and backpropagating, it has an output of 1 (it fired) when it should have been 0. The neuron has a threshold of 5 and its net input did sum up to be higher than 5.
to train it, subtract from the weights so that the net input decreases by an arbitrary learning rate, so that the threshold is not exceeded or is closer to not being exceeded. For example, the net input was 5.5, so the "error" is 0.5. Multiply this "error" by the learning rate of 0.1 and the result is 0.05. This means that the weights need to be adjusted by 5% (multiplied times 0.95). The weights are changed only a little for each training example.

In my understanding, this is similar to backpropagation with gradient descent, but it's applied to step activation functions and it works even if they're not differentiable.

My question is, where are the implementations of this very simple training method? Why wouldn't it work?


texts: Error while plotting Logistic Regression Classification

texts:
I was trying to plot by using the following code

# Visualising the Training set results
from matplotlib.colors import ListedColormap
X_set, y_set = X_train, Y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('red', 'green')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('Logistic Regression (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

I'm getting the following error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-26-af47df65cc77> in <module>
      2 from matplotlib.colors import ListedColormap
      3 X_set, y_set = X_train, Y_train
----> 4 X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
      5                      np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
      6 plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
   2978             if self.columns.nlevels > 1:
   2979                 return self._getitem_multilevel(key)
-> 2980             indexer = self.columns.get_loc(key)
   2981             if is_integer(indexer):
   2982                 indexer = [indexer]

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
   2895                 )
   2896             try:
-> 2897                 return self._engine.get_loc(key)
   2898             except KeyError:
   2899                 return self._engine.get_loc(self._maybe_cast_indexer(key))

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()

TypeError: '(slice(None, None, None), 0)' is an invalid key

Datasets:

X_train

     Pclass  Sex        Age     Fare  Embarked
0         3    1  22.000000   7.2500         2
1         1    0  38.000000  71.2833         0
2         3    0  26.000000   7.9250         2
3         1    0  35.000000  53.1000         2
4         3    1  35.000000   8.0500         2
..      ...  ...        ...      ...       ...
886       2    1  27.000000  13.0000         2
887       1    0  19.000000  30.0000         2
888       3    0  29.699118  23.4500         2
889       1    1  26.000000  30.0000         0
890       3    1  32.000000   7.7500         1

[891 rows x 5 columns]

Y_train

     Survived
0           0
1           1
2           1
3           1
4           0
..        ...
886         0
887         1
888         0
889         1
890         0

[891 rows x 1 columns]


texts: Why is scale parameter on batch normalization not needed on ReLU?

texts:
I now read a book titled "Hands-On Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, the author writes the following explanation on batch normalization:

  Note that by default batch_norm() only centers, normalizes, and shifts the inputs; it does not scale them (i.e., γ is fixed to 1). This makes sense for layers with no activation function or with the ReLU activation function, since the next layer’s weights can take care of scaling, but for any other activation function, you should add "scale": True to bn_params.

The batch_norm function is from TensorFlow, FYI. The author explains that the γ parameter should not be set on ReLU activation function.

However, I don't understand why on ReLU, the next layer's weights can take care of scaling... I understand that the next layer takes input from the ReLU output, which is max(0, x). Why does it take care of scaling and thus no need to set γ parameter on ReLU?


texts: How to get an aggregate confusion matrix from n different classifications

texts:
I want to test the accuracy of a methodology. I ran it ~400 times, and I got a different classification for each run. I also have the ground truth, i.e., the real classification to test against.

For each classification I computed a confusion matrix. Now I want to aggregate these results in order to get the overall confusion matrix. How can I achieve it?

May I sum all confusion matrices in order to obtain the overall one?


texts: How can I find anomalies in each row of data?

texts:
I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility.

-----------------------------------------------------------------------------
| Facility  | 2017  Jan Visitors | 2017 Feb Visitors | 2017  March Visitors |
-----------------------------------------------------------------------------
| Facility 1|         1234       |       1345        |  100345              |
----------------------------------------------------------------------------
| Facility 2|        56          |      567          | 34                   |
----------------------------------------------------------------------------

How can train this panda dataframe like this row-wise?

Planning to use one class SVM from sklearn. I want to get the anomalies in each facility e.g. in Facility 1 I'd mark 100345 as an anomaly. I have data spanning a couple of years. While we are here I am a super noob in ML and data science can I get a pointer to a condensed tutorial on unsupervised learning most of the ones I am coming across are for supervised.


texts: get the size of dataset after applying a filter from tf.data.Dataset

texts:
How can I get the size or the len of the dataset after applying a filter. Using tf.data.experimental.cardinality give -2, and this is not what I am looking for!! I want to know how many filtered samples exist in my dataset in order to be able to split it to training and validation datasets using take() and skip().

Example:

    dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5])
    dataset = dataset.filter(lambda x: x < 4)
    size = tf.data.experimental.cardinality(dataset).numpy()
    #size here is equal to -2 but I want to get the real size which is 3

My dataset contains images and their labels, this is just an illustrative example


texts: Linear Regression - Metrics to consider?

texts:
While evaluating the Linear regression model, which metric do we need to consider R square or Mean Squared Error (or which needs to be given more importance)?

Thank you.


texts: Piecing together an equivalent of Google's Data Science / Engineering AIY Computer Viz kit

texts:
Google has a DS / AI / ML / engineering themed DIY kit for voice recognition and computer vision.

The big manufacturers (Micro Center, Mouser, Seeed, etc) who partnered with Google are sold out and currently don't list any future availability dates.

These projects are kind of publicity, goodwill, and community-engagement efforts by Google, very much in the vein of Google Cardboard. They're meant to be cheap, accessible, and easily cloned for popular use.

All of that is great and it's an attractive hobby for a Data Scientist, but the problem is that Supply < Demand due to low manufacturer price points (part of an agreement with Google to limit quantity of supply whilst simultaneously restraining price, thus resulting in a shortage). As a result, we are now getting price gouged on eBay by resellers for >$250/each.

Unlike with Google Cardboard, most of the businesses that might clone and sell their versions of it have not yet gotten into the game (to my knowledge).

Is it possible to piece together a list of what you'd need to buy to make your own?

Here's what I have so far. Some of this (the Pi) was meant to be purchased separately anyway:

Raspberry Pi Zero W board
Raspberry Pi Camera 2
Raspberry Pi Zero W compatible power supply (SKU: 001917)
Micro SD card (available everywhere offline and online)
Cardboard (I have questions on this below)
Piezo buzzer

For the cardboard are there any CNC files or do you just transcript/print their design and manually cut your own? For now I'll start with a cereal box or something and cut it to approx. that size.

What about the button and status LED? Can we find the same or sufficiently similar sold separately such that one could follow along with Google AIY projects, or demo projects for AIY users to replicate?

Not yet located:

VisionBonnet accessory board (×1)
11mm plastic standoffs (×2)
24mm RGB arcade button and nut (×1)
Privacy LED (×1)
LED bezel (×1)
Tripod mounting nut (×1)
Lens, lens washer (1 spare), and lens magnet (×1)
Button harness (×1)
Pi0 camera flat flex cable (×1)
MIPI flat flex cable (×1) ... is this is?
External cardboard box (×1)
Internal cardboard frame (×1)


texts: P-value mining on large number of combinations of variables

texts:
I really don't know any machine learning, but have a problem that seems like one where I should use some ML algorithm.

I am analyzing a medical study with one age-related condition, age, a treatment, gender, and the abundance of two particular gut bacteria. Many researchers in the field also like to look at the ratio of these two bacteria.

Playing around with some regressions with one, two, or three explanatory variables, I found some unexpected combinations with very good p-values. For instance, controlling for age, bacteria-A seems to be strongly associated with the condition regardless of treatment. The other bacteria seems to be strongly associated with the treatment regardless of age. I would have had no way to expect this to be the case.

I feel there might be value in searching for more unexpected associations. I can make a list of all one, two, and three combination explanatory variables and perform regression of my six variables against these combinations and basically sort by p-value. But, 1) this sort of p-value mining is generally frowned upon, and 2) there are a bazillion possible regressions.

Seems like there is probably some sort of ML algorithm that would hunt down the unexpected associations in an objective and systematic way.

What would that be?


texts: Count the max number of consecutive 1 and 0 in Pandas Dataframe

texts:
Hey I have the following Dataset
import pandas as pd
df = pd.DataFrame({
'column1': [0,0,1,0,1,0,0,1,1,0,1,1,1]})

I want to be able to count the number of consecutive 1 and 0 and generate 2 columns as such:
consec0: 1,2,_,1,_,1,2,_,_,1,_,_,_
consec1: _,_,1,_,1,_,_,1,2_,1,2,3

I then want to take the max number of consecutive and create two lists:
max_consec0: 2,1,2,1
max_consec1: 1,1,2,3

My dataset in the end will be just max_consec0 and max_consec1


texts: Reporting false alarm rate and detection rate in shilling attack detection without given any labels

texts:
I have a dataset of ratings given to movies by users.
I've applied the method mentioned in this paper to detect fake votes(I've used $H(X)$ and $M(X)$ measures). But the dataset I'm using doesn't have any label for checking the false alarm rate and detection rate of my code. So is there any way to report these two criteria without having labels?


texts:
newbie here. I finished a couple of online courses and read Intro to Statistical Learning, thinking of working on a personal project and would appreciate it if you can clarify some issues:

What does "cleaning" the data consists of? How do you know if your data needs cleaning anyway?
What features of the data determines which algorithm I should use? Or is it mostly trial and error?
Are there any additional things that I should be doing other than trying different algorithms and testing how well they fit the data?

Thank you!


texts: Validation data for multi-series stateful LSTM

texts:
With stateful LSTM the network state is propagated to subsequent sequences and batches. I have multiple data files with data that I present to the network for training (making this multi-series). My question is how to create the validation data.

At the moment I take 20% of the data from either head or tail of each file and use that for the validation data, and the other 80% I submit to the NN.

However, because this is stateful, I wonder if that's not a good way to create my validation data. To elaborate; when training I potentially present several years or a decade of data, so when it comes to the final sequence the network, it has the state built up from previous years of data. If I present some validation data which is maybe just a week of data, then the network doesn't build up any prior state to use to generate the final output.  Am I right in thinking this isn't a good way to validate a stateful RNN/LSTM?

What I'm thinking instead is to put aside 20% of the data files I have, and use the entire data in those files to create the validation loss. Would that be better?


texts: I know which forums specific user read - how to cluster the data?

texts:
I have a dataset as follows. For each user, I have a separate row with the forum he reads.
There is up to 100 different forums.

I would like to cluster this data, so each user will be assigned to one of the groups (I do not know how much of groups there could be) based on the forums he read.
Do you know if there is a ready algorithm that I can use?
For now, I'm thinking if I can calculate the vector from the list of all the forums and use k-means on that.
Second thing would be to transform the data as follows:

I guess that if I would only use 1, if user read the forum, or 0 if not, that would not work with k-means. I can extract number of posts, reputation or upvotes instead of "1". Will it work with k-means?


texts: What do "compile", "fit", and "predict" do in Keras sequential models?

texts:
I am a little confused between these two parts of Keras sequential models functions. May someone explains what is exactly the job of each one? I mean compile doing forward pass and calculating cost function then pass it through fit to do backward pass and calculating derivatives and updating weights? Or what?

I have seen in some codes, they only used compile function for some of their LSTMs and fit for some other ones! So I need to know each of these functions do what part of the work(training a neural network).

It's also interesting for me to know what exactly do predict function as well.

Very thank you in advanced!


texts: Is there a difference between tf.nn.conv1d and tf.nn.convolution in Tensorflow?

texts: I want to know what is the difference between a these two. For me, they are the same function, so I do not see the reason of existance two same functions.


texts:
I am doing a project on finding famous people who are similar to each other.

For this, I am extracting a bunch of features and applying a distance function on them to evaluate who is closer to whom.

Is there a way to test this out quantitatively? Take the example of actors for easy reference.

Say, one model predicts Robert Downey Jr. to be most similar to Christian Bale. While after applying more weights to the 'heights' feature, George Clooney is found to be most similar to Christian Bale.

Now how do I quantitatively assert that one is better than the other?

One might say that Robert Downey is a better result because both Christian Bale and Robert Downey have starred in cult superhero movies (albeit different ones), while someone else might say that since George Clooney and Christian Bale were both batmans at one point, George Clooney is the better result.

Is there a way to test this other than intuition?


texts: Elements of Statistical Learning - question on p. 12

texts:
I am starting to work through Elements of Statistical Learning, and right off the bat I am coming across things that I don't understand. I would be grateful for any help from this community. Please let me know if this is not the appropriate forum to post these questions (in which case, if you're feeling extra-nice, please point me to the correct forum).

On page 12, the authors present the familiar expression for linear regression:

  Y-hat = X-transpose * β

The authors then state:

  Here we are modeling a single output, so Y-hat is a scalar; in general Y-hat
  can be a K–vector, in which case β would be a p×K matrix of
  coeﬃcients. In the (p + 1)-dimensional input–output space, (X, Y-hat)
  represents a hyperplane. If the constant is included in X, then the
  hyperplane includes the origin and is a subspace;

Questions:

Is the input-output space only p+1 dimensional assuming that Y is a scalar, rather than a K-vector? If Y is a K-vector, would the input-output space be p+K dimensional?
What does the statement that "(X, Y-hat) represents a hyperplane" mean? Assuming that X-transpose is a single column vector representing the fact that there is only one input variable, could you help me visualize what the hyperplane would look like?

Thank you in advance!


texts:
I created a model with machine learning. Using the library Accord.NET.
I would like to keep the pre-trained model in a file (not in the RAM memory) how this is possible?
Note: not necessary to create the model with Accord.NET could be with Python (TensorFlow, PyTorch, etc) but to use the pre-trained model from my desktop application in C#.


texts: Isolation Forest Prediction Mechanics: Does it compare value with every tree (and the original training subset)?

texts:
So I understand the general idea of how isolation forests works, but I'm having trouble understanding how the model makes predictions on new data.

Does it pass every new point (separately) through every tree in the trained model, and then runs the exact same splits with the exact same original subset data (plus this single new point), to determine the number of steps (anomaly score) until this new point gets isolated?
Then this score gets compared back to the anomaly score threshold that was set when the model was trained?


texts: how to remove columns in Transformer function in Pipeline

texts:
I already use a custom transformation function in a sklearn's pipeline. In this function I only add features to my data frame. It works greatly.
Below is a working example.

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_blobs

x, y = make_blobs(n_samples=300, n_features=2, centers=1)
x_train = pd.DataFrame(x[:150,:], columns=['x1','x2'])
x_test = pd.DataFrame(x[150:,:], columns=['x1','x2'])

class myTransformation(object) :
  def __init__(self, colname):
    self.colname = colname

  def transform(self, x) :
    dat = x.copy()
    squared = dat.loc[:,self.colname]**2
    squared.name = "%s_sqre"%self.colname
    dat.loc[:,squared.name] = squared
    dat.loc[:, self.colname+'_2'] = dat[self.colname]
    return dat

  def fit(self, dat, y=None) :
    return self

makePipe = Pipeline([('makeTransfo', myTransformation(colname="x2"))])
fittedPipe = makePipe.fit(x_train)
x_1 = fittedPipe.transform(x_train)
x_2 = fittedPipe.transform(x_test)

Now I want to be able to add the possibility to remove the equal columns in the data frames.
For now, I have the following function :

def delSameCols(df) :
  cols = []
  for i in range(df.shape[1]) :
    for j in range(i+1, df.shape[1]) :
      if (df.iloc[:,i].dtype!='O') | (df.iloc[:,j].dtype!='O') :
        if np.array_equal(df.iloc[:,i],df.iloc[:,j]) :
          cols.append(df.columns[j])
  cols = list(set(cols))
  print( u'      -%s features removed'%len(cols) )
  return df.drop(cols, axis=1), cols

I have no idea how to deal with this : add a new function in the pipeline or directly in the existing function ?
Does anyone have any idea please ?


texts:
I am a researcher working on traffic flow prediction. For my project I would need to obtain historical traffic flow raw data. Google Maps has a traffic flow overlay which is pretty much the kind of data I need. However, Google doesn't seem to provide access to historical data. There are several other companies that provide analytics based on traffic data to their clients but raw data is never provided. How would one go about obtaining this kind of data?

Edit: the kind of data I need would be spatio-temporal. Take a bunch of N sensor locations on an inner city road network and give me the driving speed of the cars passing through each one of them at T equally spaced time points during a day. So we would have N*T data points per day. The denser the locations the better.
If you look at Google Maps traffic overlay you can see traffic flow speeds for almost every small street. I think they take this data from GPS in driving cars as well as stationary sensors. Google doesn't provide historic data but maybe one could record this from the website directly?