texts: SVM is taking too long for hyperparameter tuning

texts:
I am running SVM,Logistic Rregression and Random Forest on the credit card dataset. My training dataset has the shape (454491, 30). I performed 5-fold cross validation(which took more than an hour) with 'scoring' set as 'f1_weighted' and got the below results:
Logistic Regression Cross Validation F1 score: 0.9501631748525725
Random Forest Cross Validation F1 score: 0.9999383944188953
Support Vector Cross Validation F1 score: 0.9989703035983751

I chose SVM as Random Forest is prone to overfitting and SVM scored better than Logistic Regression.
I wanted to add regularization by hyperparameter tuning. I initially used GridSearchCV but it was taking a long time so I changed it to RandomizedSearchCV but even this is taking a very long time (around 4-5+ hours). According to the data description, the features have been scaled and PCA has been performed to preserve the private information. I have also used RobustScaler() on Amount and Time columns as they were not scaled.
tuned_parameters={"kernel":['rbf','sigmoid'],
                 'gamma':[1e-2, 1e-3, 1e-4, 1e-5],
                 'C':[0.001, 0.10, 0.0001,0.00001]}

tuned_SVM=RandomizedSearchCV(SVC(),tuned_parameters,cv=3,scoring='f1_weighted',random_state=12,verbose=15,n_jobs=-1)

Any suggestions on how to proceed?


texts: Machine learning mode retraining after getting new data

texts:
I have trained a model for predicting the sale of items daily, such as daily car sales, with a machine learning model in Python. Now I get the new time real data (time series data). I want to retrain my mode. There are some issues:

How much data I should use for retraining model, namely what's the time
  range of data or when I should retrain my model. For example use the new one day or one week et al., data to retrain model.
What's condition of new data if I want to use the new data for retraining model.
How to evaluate the stability of new training model, and How to know abnormal prediction for new model.


texts: Can we use HDFS and big data Analytics for processing huge log files being processed through some application on some central server?

texts:
Detailed Question Explanation:

Suppose say our application X is processing huge logs (size varying from MBs to GBs) and giving insight results in these logs(NOT A Social Data logs or Security Logs)

now this logs are in format say log.y with different variety, using C++ as Engine to process these huge logs.(It generates imp. insights about data but need to be processed using our application X only and we don't want to change core way processing of application X)

If this processing happens on some server it under or over utilizes resources (That I already know).

If we use cloud computing for this processing we get that processing power with optimum usage.

How do we see help of BIG data analytics in this particular sort of usage?
Any help or suggestion is very deeply appreciated


texts: How Do I Find the Matthews Correlation Coefficient (MCC) in Python?

texts: I would like to find the Matthews Correlation Coefficient (MCC) for predictions made by a binary Sklearn machine learning model. How do I do it?


texts: Do I need to fit on train data for truncated SVD and then transform the test data on fitted train data?

texts: Regarding truncated SVD(single value decomposition) do I need to fit on train data and transform the test data on fitted train data? or can I fit on test and transform on test fit?


texts: what is the difference in terms namely Correlation, correlated and collinearity?

texts: A website says Correlation refers to an increase/decrease in a dependent variable with an increase/decrease in an independent variable. Collinearity refers to two or more independent variables acting in concert to explain the variation in a dependent variable.Could someone clarify the terms ?


texts: What is a logworth statistic, and how useful is it?

texts:
My teacher mentioned it today, and there is nearly zero good search results for it, other than one mention each in the SAS and JMP documentation.

It says it is -log10(p-value), but there is almost no explanations of this online. Also it seems like this statistic is widely ignored. It isn't mentioned in any tutorials or in any of the data science papers I have read in the past two years.

Question

So, what is this statistic, and where does it come from? How useful is it in data science?


texts: Décision tree, How to see under/over fitting with just looking at the leafs?

texts:
My question is: how with just looking at the leafs of a decision tree could you tell if the model is under/over-fitting?

Any sort of advice will be helpful.


texts: Is there a relationship between learning rate and training set size?

texts:
I have a large dataset to use for training a Neural Network model. However, I don't have enough resources to do a proper hyperparameters tuning on the whole dataset. Therefore, my idea is to tune the learning rate on the subset of data (let's say 10%), which won't obviously give as good estimate as the whole dataset would give, but since it's already a significant amount of data I would expect it might give the estimate which is sufficient enough.
However, I wonder if there is some relationship between learning rate and the training size, other than the fact that the estimate would be noisier when subsampling. And by the relationship I mean some rule of thumb, for example, increase your "optimal" LR, when you're increasing the training set size. I don't see any (apart from the relationship between LR and batch size, but it's actually other topic), but would love to make sure that I'm not missing anything.


texts:
What is the dimension of a time trend matrix in a VAR model? Consider the following VAR model:

I was thinking the dimensions is (3x2) where
-> 3 rows correspond to three X variables
-> 2 columns correspond to value of Beta and value of t

I'm not really good with matrices, but everything else in the VAR model has a dimension of (3x3) or (3x1), so I thought (3x2) was odd and not sure how it would work out...

Help would be greatly appreciated!!!


texts: How to calculate lexical cohension and semantic informaticveness for a given dataset?

texts:
In 'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures' they have mentioned;

There are two slightly different classes of measure: lexical cohesion (sometimes called ‘unithood’ or ‘phraseness’), which quantifies the expectation of co-occurrence of words in a phrase (e.g., back-of-the-book index is significantly more cohesive than term name); and semantic informativeness (sometimes called ‘termhood’), which highlights phrases that are representative of a given document or domain.

However, the review does not include the ways to calculate/derive these measures. Can someone please specify how to get these two measurements for a given text documents?


texts: Too high performances on a classification problem

texts:
I have a .json file as dataset of the type:

and I am working on a classification problem in which I have to predict 4 classes, which are rhe semantic. I have worked through the problem, and after splitting the dtataset into training and test sets, I get an accuracy of $1$ .I have an unbalanced dataset, so I have oversampled it:

my code is the following:
dataFrame = pd.read_json('dataset.json',lines = True)
df = dataFrame[["lista_asm", "semantic"]].copy()

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer=TfidfVectorizer()
df_x = df['lista_asm']

X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = df['semantic']

#oversampling
from imblearn.over_sampling import RandomOverSampler

 ros = RandomOverSampler()
 X_ros, y_ros = ros.fit_sample(X_all, y_all)

 print(X_ros.shape[0] - X_all.shape[0], 'new random picked points')

#splitting

X_train, X_test, y_train, y_test = train_test_split(X_ros, y_ros,
    test_size=0.2, random_state=15)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

#fitting the model
clf = svm.SVC(kernel='linear', C=10).fit(X_train,y_train)

y_pred = clf.predict(X_test)

#checking accuracy
acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc)          #from here I get accuracy 1

I have not written the imports in order to avoid making the code too long here, but if needed I can add them.
So I am not testing on the training set, but the results are too good so for sure there mmust be something wrong.I don't understand what is wrong.
Is this something that happen for any specific reason?
I have tried to change some stuff to make the code worse, in order to see if the accuracy would go down, but it remains to one.
[EDIT]I have tried to use your suggestions, and I am already sorry since for sure I am doing something wrong.
Here is what I have done:
X_all = df['lista_asm']
y_all = df['semantic']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
    test_size=0.2, random_state=15)

tf = TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True,
max_features = 20000)
tf_transformer = tf.fit(X_train)
xtrain = tf.fit_transform(X_train)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = svm.SVC(kernel='linear', C=10).fit(xtrain,y_train)

tf2=TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True,
max_features = 20000)
tf_transformer = tf2.fit(X_test)
X_test = tf2.fit_transform(X_test)

y_pred = clf.predict(X_test)

acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc)       #gives accuracy 0.277

and if I plot the other performance metrics:

with the following message:

UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))

[EDIT]I have also tried to use the same approch but with a .jsonl file, and in this case it works fine giving a reasonable accuracy. Maybe it is wrong to do so, but could the problem be here, so that I am not handling correctly a .json file?
Can somebody please help me?


texts: Python script/GUI to generate positive/negative images for CascadeClassifier?

texts:
For the first time, I am playing around with a Cascade Classifier with the OpenCV package (also new to the latter). I realized that it would probably be faster to write my own GUI/script to generate the needed positive and negative images from the set of images I have than to open each file in Photoshop or Paint, but I also suspect this has been done many times before.

In particular, I am looking for a GUI that lets users page through files in a directory and then use mouse clicks to draw rectangles on a particular image and have the coordinates of the rectangle recorded for later purposes. Any suggestions? If not, I'll  be sure to post a link when/if I finish this. It seems like something of general enough utility I am surprised I can't find it in the OpenCV package itself.


texts: How to improve accuracy of Named entity recognition (NER) tagger on local data?

texts:
I am using NER from spacy. Its giving incorrect results for few words. Its trained on general dataset. How can I customize on my local data.

For example,

Person -  {'Mike Miller', 'Miller', 'Infantino', 'Gianni Infantino'}
Location -  {'England', 'UK', 'Europe', 'Telegraph'}

Here, "Telegraph" is incorrectly is assigned to location.


texts: is it possible to decide model without any data?

texts:
Today I just faced a very unique demand from my superior. He asked me whether I can make a model first before we gather the data for training because we don't have any data yet.
I was utterly confused about what to do with this. Did anyone have any suggestion how should I approach modelling without any data at all? Thanks


texts: Machine Learning, Imputing values that should be blank

texts:
Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event.

As an example say a teacher wants to predict the grades of his students. Some of the students may have been in his class last year and he can use that grade as a variable. However maybe only 20% of the students were in his class so the rest of the 80% will have a Null value. Most ML algorithms cannot accept Null values so the variable would have to somehow be imputed.

I cannot think of an imputation method that would make sense here, the standard mean/mode would imply that all students were in the class and since the variable is pretty unbalance and 80% of the values would be imputed I don't imagine it would hold any valuable information.

Are there any methods to deal with this scenario?


texts: Finetuning a deep fully convolutional neural network with skip connections

texts:
What is the proper procedure for finetuning a deep fully convolutional neural network with skip connections for transfering between different tasks? For a 'typical' (no skip connections, not fully convolutional) neural network, at least the last layer is "fully connected", which is to say the nodes in this layer use every node in the previous layer as input, and the number of outputs is equal to the number of class labels M. In Shelhamer's original FCN paper and the follow-up PAMI paper, there are details about how to implement this type of finetuning (also sometimes referred to as feature extraction); however, there aren't any details as to how it should be done with a network with skip connections like a ResNet or a DenseNet (sorry linked removed due to reputation). Does anyone have any advice on best practice here?

(x-posted to Reddit and Caffe forums)


texts: TicTacToe Linear Regression low accuracy and R^2 score

texts:
Im using the python sklearn library to attempt a linear regression TicTacToe AI.
I create my training set by simply having the computer play random 'blind' games against itself. For example... Player one plays a random segment of the board. Next player two plays a random valid segment of the board etc. This goes on until the board is full or someone has won. Each time player one wins, i store the board states leading up to the win. Every loss, i simply mark that board state (and past board states of the same game )  as a loss for player one. For every tie game(full board) I do not count it as anything. I play about 20k of these games. At the end I get my training data set which includes the board state (the feature set) and the outcome which is the percentage (a floating pint value. eg .8 is 80%) of games won for that state.
So for example going from board top left to bottom right: [1, 1, 1, 2, 0, 2, 0, 0, 0] would be:
X X X
O - O
- - -

would have a '1' or 100 percent after playing 20k random games etc.
I'm trying to predict the success rate of player one's next move. Basically the success rate of any free segment based on the board state.
However; after training sklearn linear regression with my training data, I get a very low R^2 score of .14 and any test is highly inaccurate. I'm beginning to think there is a flaw in my data? Is this how data scientists would go about creating the training set for tic tac toe?


texts: Im trying to use gridsearch to find the best parameter for my model. Knowing that I have to implement nearmiss undersampling method while doing cross validation, should I fit my gridsearch on my undersampled dataset (no matter which under sampling techniques) or on my entire training data (whole dataset) before using cross validation?


texts: I am training an ensemble model using a 400 data set sample this led to a precision recall curve that looks like stairs ? what would be the reason beside the low number of samples ?


texts: Deep Learning for non-continuous dataset

texts:
I am working with this dataset which is record of student academic details and I want to predict the student's performance.

since the dataset is non-continuous I cannot apply CNN on this dataset.

How can I apply Deep learning on this kind(non-continuous) of dataset. I searched online but could not find anything relevant

Thank you!!


texts: ML on dataset with heavy dependency on prefix

texts:
I have a target function, which heavily depends on vector prefix:

(1, 2, 0, 0, 1, 2, 4) -> A
(1, 2, 0, 55, 1, 99, 1) -> A
(1, 2, 124, 55, 1, 99, 71) -> A
(1, 3, -5, 0, 1, 2, 4) -> B
(1, 3, 12, 55, 1, 99, 1) -> B
(1, 3, 124, 55, 1, 99, 71) -> C
(1, 3, 124, 21, 10, 195, 11) -> C
...

Prefix length may vary.
pre-condition: a training set of vectors and their target values
target: to be able to evaluate target value on arbitrary vector
I have tried HMM, but it doesn't seem to work well in this case. What algorithm should I use instead?


texts: What are Machine learning model characteristics?

texts:
This question I have received in some Machine Learning related interview and Here is the question

  What questions would you ask to learn about machine learning model characteristics?

This is what I think:

I did a bit research on the internet & found this resource, but still not very clear about How ML model characteristics are equivalent to asking trade-off between different algorithms in supervised learning settings. What I understood and framed my answer is:

First, for simplicity, I assumed that this model is used for some supervised learning task(classification/ regression) then,

I would first try to find a learning algorithm used to create this model, because this will provide me with a clue of the created model and help me to talk about different issues within it(such as,

How complex or simple your model is(feature engineering!)?
What are your training & test errors(Model performance)

Both will help me to talk about the bias vs variance trade-off.
Furthermore, I could talk about different learning algorithm trade-offs in supervised settings.

Is the model based on identifying correlations i.e output variable can be expressed in terms of the linear/non-linear combination of features?
(targeting linear regression, logistic regression, SVM or Neural network algorithms, or
it's decision tree-based algorithms.

I am not confident enough that my answer is complete or correct (since I am not able to understand what precisely characteristics of a model are?), so I need this community's help to provide feedback! please feel free to add your suggestion to it!


texts: How to train a Machine Learning model for blocked data

texts:
I'm concerned with a supervised classification problem for the following type of data. The data consists of $N$ rows (where $N$ is not very large - this is not a big-data problem) and $M$ columns (features) and each row has a certain label I'm interested in. Every row belongs to a certain block and each block consists of 1-50 rows. The size of a block depends on the duration of a block (so the rows in each block are correlated, but the correlation between different blocks can be neglected).

The aim is to learn a classification algorithm on the data that allows to classify a new row or a new block.

Now there are two important things: The labels are constant within each block and there are some features that allow the identification of blocks.

My question is: What might be the best way to learn a model on this specific type of data?

To illustrate this problem a bit further, let me report some problems I found when I worked with Random Forests on the data set.

Suppose I take a part of the data as validation data, which contains whole blocks. If I split the remaining rows randomly in training and test data, the accuracy of the learned Random Forest is very high on training and test data but very low on the validation data. This is due to the fact that the Random Forest learns in this case to identify the individual blocks and as the validation data contains some unknown blocks, the accuracy drops. This can be also seen in the resulting importance for the features: the most important ones are those features that allow an identification of a cycle very easily.

Splitting the remaining data into training and test data and leaving the blocks whole helps a bit but doesn't fix the problem completely.

Another approach would be to remove all features that allow the identification of blocks. But this is difficult a priori and some of these features could have important information for my problem.

A simple way to overcome the problems would be to take the row mean in each block and use the resulting data for the classification problem. However, in this case you loose a lot of information.

So I'm wondering if there are more natural ways to approach this classification problem which respect the block-structure of the data.


texts: What algorithms will stuck in the local minimum?

texts:
Algorithms like neural network are easily getting stuck in local minimum because the shape of the loss function (so there are parameters like momentum are designed to solve this type of problem). Logistic regression will always find global minimum because log-loss is a convex function (please feel free to correct me if I miss anything here).

Then I am wondering for other algorithms like RandomForest, GradientBoostTree, or SVM, which one would face the problem of stuck in local minimum, which one would not? And why? Thanks!


texts: Full convolution vs convolution operation

texts:
I am trying to implement backpropogation algorithm from scratch. For that I read this blog:backpropogation

You don't have to read the entire blog. At the end is mentioned formula to find gradients.

Now, I am unsure of difference between full convolution and convolution here.
Configuration of my Network
Following are the layers in my network

Convolution
Relu
Maxpool
FC
Softmax
Cross entropy
So the derivative for cross entropy is (p-y) {p = actual probability y = true probability}. Since, I have three classes, so, my vector would look something like this :

[0.8 0.1 0.1]
And since the input to my FC is a vector of size 384, so, convolving it with this vector will give me a vector of size 381. And my filters are of size 384. So how can I update filter of size 384 with gradient of size 381?
I am a newbie and my head is quiet stuck around the concepts. Any help is much much appreciated.


texts:
I'm working on a clustering problem. I have a training set composed of sets of points where the clusters are known and I want to find the good clusters on a testing dataset.
It's a kind of supervised clustering.

I looked for articles about supervised clustering but I didn't find a lot of informations. There is "semi-supervised clustering" which consists of using informations on couples of points (must-link or don't-link relations) but, in my task, I don't have this kind of information. There are also some kind of "metric learning supervised clustering" which uses the labelized clusters to estimate a metric that would produce the given clusters using k-means. That kind of technique could help me but there is not much articles about it and I wonder if I'm not finding the good keywords or something.

What are the techniques/algorithms to cluster data points using labelized data (training points with known clusters) ?


texts: Intuitive explanation of Lovasz Softmax loss for Image Segmentation problems

texts: Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.


texts: Add new attribute to the input of nerual network

texts:
I have few students'(38) sequential data. For example, one student sequential data is like:

A
B
B
A
C
D
E
.
.

Each student has different length of sequences and their used letter was within this 5 letter. So, to predict the next letter/letters I used feed-forward neural network and recurrent-neural network(LSTM). I got reasonable accuracy with these models.

Now, I want to add a new attribute of the students to the input. Suppose, the students are from four different departments.
I added the one hot vector of a particular department to the corresponding student.

I used Leave one out as a cross-validation. But for this new input, the testing accuracy is always 100% which is not possible.

My question is, Are there any methods for adding a new attribute to the sequential data?


texts: sklearn SimpleImputer too slow for categorical data represented as string values

texts:
I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but it takes too much time to fulfill the task as compared to pandas. Both methods produce the same output.

Here is the code to reproduce the behavior on a synthetic data:

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer

lst = np.array(['a', 'b', np.nan], dtype='object')
arr = np.random.choice(lst, size=(10**6,1), p=[0.6, 0.3, 0.1])
ser = pd.Series(arr.ravel())

Using SimpleImputer:

%%time
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)

Wall time: 13 s

Using pandas:

%%time
ser.fillna(value=ser.mode()[0])

Wall time: 64.8 ms

Things get even worse when string values are longer (e.g. ‘abc’ instead of one letter ‘a’). For numerical data pandas still outperform sklearn, but the difference is not that huge.

What am I doing wrong?


texts: Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass?

texts:
I want to replicate the Transformer from the paper Attention Is All You Need in PyTorch. My question is about the decoder branch of the Transformer. If I understand correctly, given a sentence in the source language and partial/incomplete translation in the target language, the Transformer is tasked with predicting the next token in the translation. For example:

English (source): I love eating chocolate

Spanish (target): Yo amo comer ...

The next token in the translation should be chocolate (thus, the full translation would be "Yo amo comer chocolate"). So, in this example, the encoder would process the sentence "I love eating chocolate", the decoder would process the partial translation "Yo amo comer" and the final output would be a softmax over the whole vocabulary (hopefully with chocolate being the token with the highest score).

The issue is, during training we want the Transformer to learn the full translation. This means that if the target sentence has length N, we want the transformer to predict the first word, the second word, the third word, and so on all the way up to the N-th word of the target sentence. One way to do that is by generating N training instances (one for each word of a target sentence of length-N). However this approach is computationally quite expensive, because instead of having a single training instance per source-target pair, we now have as many training instances as there are words in all the target sentences. So I was wondering if it's possible make all predictions for all words in the target sentence in a single forward pass. Is that possible?


texts: Recommendation for boolean dataset with apache mahout

texts:
I was trying to implement Item based Recommender System with the boolean dataset, Dataset example:

      User-id | movie-id | Action | Comedy | Drama
        1              200    0         1        1
        2              210    1         1        0

And I tried implementing it with item-based similarity algorithm as follows:

package prediction.contentrecommender;

import java.io.File;
import java.io.IOException;
import java.util.List;

import org.apache.mahout.cf.taste.common.TasteException;
import org.apache.mahout.cf.taste.impl.model.file.FileDataModel;
import org.apache.mahout.cf.taste.impl.recommender.GenericItemBasedRecommender;
import org.apache.mahout.cf.taste.impl.similarity.TanimotoCoefficientSimilarity;
import org.apache.mahout.cf.taste.model.DataModel;
import org.apache.mahout.cf.taste.recommender.RecommendedItem;
import org.apache.mahout.cf.taste.similarity.ItemSimilarity;
import org.apache.mahout.cf.taste.impl.recommender.GenericBooleanPrefItemBasedRecommender;

public class ContentRecommender {

/**
 * @param args
 */
public static void main(String[] args) {
    int userid=10;
       int noOfRecommendations=1;

    try {

        DataModel dm = new FileDataModel(new File("data/dataset.csv"));
        ItemSimilarity sim = new TanimotoCoefficientSimilarity(dm);
        GenericItemBasedRecommender recommender =
            new GenericBooleanPrefItemBasedRecommender(dm,sim);

     List<RecommendedItem> recommendations =
         recommender.recommend(userid, noOfRecommendations);
            for(RecommendedItem recommendation : recommendations)
             {
                System.out.println(recommendation.getItemID());
            }

    }
    catch (IOException e)
    {
        System.out.println("there is an error");
        e.printStackTrace();
    }
   catch (TasteException e)
    {
        System.out.println("error");
        e.printStackTrace();
    }

}

}

When I am running this code, I am not getting any output, although there is no error.I don't know what can be the problem. Can anyone figure out how to implement item-based similarity algorithm with Boolean data-set.


texts: How to train a machine learning model if there is a relationship between two different data points? I mean not relationship between two variables among the point but that between two different data examples or points, so that i want to classify similar such set of datapoints into one single class . Which algorithm is most suited in this case . I want to write the algorithm on my own.


texts: Pre-processing data - Dataframe manipulation Time Series

texts:
I have a question in which I'm not entirely sure in which path to take. I'd appreciate if you could point me in the right direction.
Below a screenshot of the a few records of my dataset.

As you can see, all the data is related to the same patient and admission id.
I'm going to tackle this problem using time-series. So, I was told to pre-process this dataset like the following:
Get a block of 12 records. Each block is 4 hours
Look at the 13th block which we want to predict
Check the column death_on_hr to see if there's a value there. It will contain in which hour the patient died if they did. If it is within that 48 hours block, I create one single row that combines all the 12 rows block with the values and label 1 if they died and 0 if not.
Then, we move one step forward. Now we start from the second row and get a block of 12 and perform the same steps as above until all the records for that patient is done.
I'm confused and how to approach this and maybe there's an easier way with pandas and numpy that someone could point me towards on how to achieve this.
Here is a screenshot of patient that died within the 48 hours.

Thank you.


texts: Handling missing values to optimize polynomial features

texts:
I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on the predicted output.

Unfortunately my data has missing values (np.NaN) and sklearn's PolynomialFeatures() can not handle these values. What is the best way to impute these values?

I've been trying to replace them with 0, 1, mean and median and for my dataset using the median seems to be the best solution. But can this be generalized and what is the intuition behind it?

I was also wondering if filling methods like ffill, bfill or even KNN modelling can be useful in this context.

Thanks a lot!


texts: Convert a pdf into a conditional pdf such that mean increases and std dev falls

texts:
Let success metric(for some business use case I am working on) be a continuous random variable S.
The mean of pdf defined on S indicates the chance of success. Higher the mean more is the chance of success. Let std dev of pdf defined on S indicates risk. Lower the std deviation lower the risk of failure.

I have data,let's call them X, which affects S. Let X also be modelled as bunch of random variables.

P(S|X) changes based on X.
The problem statement is I want to pick Xs such that P(S|X) has mean higher than P(S) and std deviation lower than P(S).

Just to illustrate my point I have taken X of 1 dimension.
Scatter plot between X(horizontal) and Y(on vertical):

You can see that P(S|X) changes for different values of X as given in the below plot:

For X between 4500 and 10225, mean of S is 3.889 and std dev is 0.041 compared to mean of 3.7 and std dev of 0.112 when there is no constraint on X.

What I am interested in is given S and bunch of Xs... pick range of Xs such that resulting distribution of P(S|X) has higher mean and lower standard deviation... Please help me find a standard technique that would help me achieve this.

Also I don't want to condition on X such that number of samples are too small to generalise.I want to avoid cases such as on the left most side of tree where number of samples is 1.


texts: What did DeepMind do with AlphaGo between the Fan Hui and Lee Sedol games?

texts:
In January, DeepMind published the article (see video) about its win against Fan Hui, which happend in October 2015. The article and other interviews say, it used 100.000 human games, and then 13.000.000 self-plays. In January, many commentators where sceptical whether AlphaGo can win against Lee Sedol.

In March 2016 (5-6 months after Fan Hui's defeat), AlphaGo won against Lee Sedol.

I was searching through many documents, watched all games, read the papers, but could not find the answer to the question: What did DeepMind do in the 5-6 months between the two games to improve the quality of AlphaGo to that level?

Was it only further self-play? If so, are there some numbers of games that were played? Were there algorithmic advances?

I would love to understand that. Thank you!


texts: Are there any machine learning algorithms that focus on comparing items, rather than classification or regression?

texts: This is more of a hypothetical than something I'm actively trying to solve. It just struck me that a machine learning algorithm that specifically looked at two pieces of data and had to label one as greater than the other or something of that nature might be inherently different than classifying each separately and comparing the strength of the classifications.


texts: Find all recurring subgraphs/patterns of maximal size in a single undirected, labeled, connected graph

texts:
I would like to identify all subgraphs of maximal size (maximum number of nodes) that are recurrent in a single undirected, labeled, connected graph. I provide exemples of input and expected output below.

I'm a freshman to graph theory so the terminology may not be the most appropriate. Please correct me if you read between the lines.

By labeled graph I mean the nodes are labeled and labels can be redundant.
By connected graph I mean all nodes are connected.
By undirected graph I mean edges are not oriented/directed.
By all subgraphs of maximal size I am not sure if I mean all possible non-overlapping isomorphisms.

So I have a single graph that I know will contain some symmetries (it contains subgraphs that will be repeated N times in the graph). And I want to find the repeated patterns.

I understand this can be a (very) complex problem but the graphs I have to use will never contain more than 150-200 nodes, so that's already something.

I have to do this using python 3.x and I can use any libraries. I have tried some brute-force detection but it gets complicated quickly and I feel like I shall use some clustering, maybe with trees representations. Possibly it can be solved with the right functions from packages likes networkX, but I can't figure it out atm.

Also, in case there is a central node, I don't really care if it is included in multiple subgroups.

I can't tell you how grateful I already am for any insights you may provide :)

Exemple 1

Exemple 2

Exemple 3

But this last criterion is really bonus at this point.


texts: Is this approach for Active Learning correct?

texts:
Let's say that we have 10,000 unlabeled documents, and we want to use pool-based sampling with batch size of 5. And we will use least confidence (LC) for querying strategy.

Is this outline of active learning reasonable:

1) Randomly select 30 documents, and get them labelled:=> labelled: 30, unlabelled: 9970.

2) Train the model on 30 labelled data

3) Use the model to make predictions for 9970 unlabelled documents

4) Use LC to select 5 out of all predicted documents that the model is least certain about

5) Get labels for the 5 documents:=> labelled: 35, unlabelled: 9965

6) Retrain the model on labelled data

7) Keep doing steps 2 - 6 until, the labeling resource expires or other stopping criteria is met.

Is this correct? I am mostly concerned about step 3) Should we retrain on all the data on every iteration? It seems like it could get computationally slow, but on the other hand if we don't evaluate every document how would we know which one is the most helpful to get labelled?


texts: Is there a metric for counting AND classification simultaneously?

texts:
I'm working on a project that mixes object detection and crowd counting.

The metric for object detection is mAP, which combines the regression of the bounding boxes with the precision of the classification. The metrics for crowd counting are MAE and MSE, a regression problem.

The problem I want to solve involves counting and classifying objects in an image. I don't want the precise location of the bounding boxes, hence mAP is not useful, and I need to account for the classification errors, then MAE and MSE are out.

Is there a meric/loss that combines counting and classification?


texts: Visualizing community composition using network of pie charts

texts:
Given a social network, I want to perform community detection and compare the result to known node metadata, such as gender, age, etc. to see if certain communities are largely composed of "similar" people. I have seen this done before in visualizations like this:

(image from https://arxiv.org/pdf/0809.0690.pdf)

where each circle represents a community and the coloring of the circle shows the breakdown of some attribute (e.g. nationality) within that community.

Does anyone know what tool can be used to create such a visualization? I have seen it in multiple papers, so there must be something...


texts: Will this MAPE implementation work for multidimensional output?

texts:
I'm currently working on a CNN problem where the output is a 60x59 array of numerical values. I want to verify if the mean absolute percentage error (MAPE) function I'm employing will properly consider the error by matching each corresponding point to the true value, as opposed to unexpected behaviour. Will this formulation work?

def percent_mean_absolute_error(y_true, y_pred):
    if not K.is_tensor(y_pred):
        y_pred = K.constant(y_pred)
    y_true = K.cast(y_true, y_pred.dtype)
    diff = K.mean(K.abs((y_pred - y_true)) / K.mean(K.clip(K.abs(y_true),
                                                           K.epsilon(),
                                                           None)))
    return 100. * K.mean(diff)

K here is keras.backend. All I want to know is if this function properly considers the error by mapping the function properly from point to point from the input to the output?


texts: I did image classification using CNN and it successfully classified the images  but How to save predicted images to separate folder for example i have two classes cat and dog after prediction how to save predicted cats images to cat folder and dog images to dogs folder


texts: Sampling in Text Classification: can the results be considered 'reliable'?

texts:
I am testing different models (SVM, Logistic Regression, Naive Bayes, Random Forest) for predicting the class of a spam email. My target is a binary variable.
I am analysing only text, no other fields.
My dataset includes
Label
0.0    3333
1.0     768

As you can see there is a big problem with classes imbalanced.
I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset.
I got good results in terms of F1, recall and accuracy for upsampling (above 88%; max 97%), bad for downsampling (<=76%).
For instance:
Down
              precision    recall  f1-score   support

         0.0       0.79      0.43      0.56       102
         1.0       0.61      0.87      0.76       114

Confusion Matrix:
 [[ 49  60]
 [ 12 100]]

Up
              precision    recall  f1-score   support

         0.0       1.00      0.85      0.91       873
         1.0       0.87      1.00      0.94       884

Confusion Matrix:
 [[772 141]
 [  20 822]]

I would like to ask you if these values can be considered good results or they can't. I am considering  a publication (not only to include similar analysis), so I would like to check if such results can be considered reliable, despite of the imbalance.
Any suggestions and advice will be greatly welcome.


texts: I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?


texts: MobileNetv1 - High and Fluctuating validation metrics

texts:
i've been using the MobileNetv1 model, pre-trained on ImageNet in Keras. I replaced the last layer with my own softmax layer to classify my images into one of 10 classes (maybe too few for such model and makes it overfit?)
During training, I froze the first 47 layers of the model.
The final results are overall good, the confusion matrix is pretty promising and the evaluation shows a 0.95 accuracy, despite having a 1.25 loss.
The questions I would like to get help on are the following:

Why is the validation loss so high despite the accuracy being so good?

Why is validation loss fluctuating so much during training, is it potentially due to overfitting?


texts: Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

texts: The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?


texts: Converting VOC2012 annotation .xml to segmentation .png

texts: Does anyone know a good way to convert VOC2012 annotations files in xml to SegmentationClass (class annotations) files in png?


texts: What metrics determine the quality of the model?

texts:
Working on this Kaggle competition, and have some questions. Using this code:

def r2_keras(y_true, y_pred):
    SS_res =  K.sum(K.square(y_true - y_pred))
    SS_tot = K.sum(K.square(y_true - K.mean(y_true)))
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

The output of my training looks like:

0s 138us/step - loss: 0.1340 - mean_squared_error: 0.1340 - r2_keras: 0.7565 - val_loss: 0.4112 - val_mean_squared_error: 0.4112 - val_r2_keras: 0.4064
Scaled Validation r2: 0.5182
Unscaled Validation r2: -152.1261

I am using 20% of the training data for validation.

I am tracking these metrics during training:

Training loss, mse and r2
Validation loss, mse and r2

I get these metrics on the model after training:

R2 for Validation on scaled data
R2 for Validation on unscaled data

scaler = StandardScaler()
scaled_train = scaler.fit_transform(train_df)
scaled_test  = scaler.transform(test_df)
...
m.fit( X_train, Y_train, epochs=epochs, validation_data=(X_test,Y_test))
....
from sklearn.metrics import r2_score
scaled_r2 = r2_score(prediction, scaled_test_df[[target]].values)
unscaled_r2 = r2_score(descaled_prediction, test_df[target].values)

So, my questions are:

Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?
Why isn't the unscaled r2 the same as the scaled r2?
The model r2 is not the same as any of the validation r2's during training (val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?


texts: How do I visualize data for a natural language processing project?

texts:
I am using a question-and-answer dataset. My neural network takes a question and an article content, and outputs where an answer starts (as an integer). To visualize my data, how should I process it and what plot(s) should I use?

I'm considering:

Word/N-gram frequency histogram for the questions. Another one for the answers.

Plots mapping word/n-gram frequency to output features

Plots mapping word/n-gram frequencies to Shannon entropy values.

On that note, maybe using a smaller machine learning model - such as a decision tree - qnd graphing the resulting probabilities.

What is the best plot for a project like mine?


texts:
I am getting attr(, "nn.index") as part of my KNN output in R. What is meant by that and how is this value getting calculated?

knn.pred <- knn(tdm.stack.nl_train, tdm.stack.nl_Test, tdm.cand_train)
print(knn.pred)

> knn.pred

[1] Silent      Silent      Silent      Silent      Silent      Silent      Silent
 [8] Silent      Silent      Silent

attr(,"nn.index")

      [,1]
[1,]  292
[2,]  292
[3,]  343
[4,]  444
[5,]  250
[6,]  445
[7,]  270
[8,]  228
[9,]  302
[10,]  355


texts: Setting "missing" distance values to zero when training a neural network

texts:
Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of up to 32 objects at once. If the sensor can't find 32 objects (which is always the case) it sets the x and y coordinates of the objects not found to zero. Could there be a problem training a neural network on this kind of data? The sensor is set on a car, the objects are other cars and the networks job is to predict the next move.

Another problem is that when the sensor finds a new object, already existing/found object might change id. Any tips on that? I am thinking about making randomly permute the indices of the object since that should not make any difference?

Are there any standard solutions to these kind of problems? Especially Setting the distance of non existing objects to zero.


texts: Python First Project: Algorithm that find a book and its author from a quote

texts:
I hope this is the good site for my question.

I am new with Python and I am neither a programmer nor a data scientist. I would like to code an algorithm that given a book quote can find the book's name and author.
Do you have any suggestion/sources that can help me with my project?

I think that the simplest approach will be to include in my code the Google Books API. In fact, it is really difficult to find a large database including Books' text in order let the algorithm research the quote. Do yo agree with my first approach? Which are the limitations of using Google APIs? Can I share my code with other users or they will be not able to use it because of some kind of Google restriction?

I am sorry for the stupid questions, but I have just begun this adventure in Datascience and I think that the best way to learn is to fix some objectives and not just to do courses.

Thank you.


texts: Feature importance difference in two similar machine learning models

texts:
Situation 1:
I have trained a text classification model (Model 1) which gives me a probability of true class as X. I have also trained a classification model (Model 2) using only the categorical and numeric data. Both the models are used to predict the same true class; just the features differ.
I  used a random forest classifier on the probabilities returned by Model 1 and Model 2(taking them as input features) and got similar performance metrics(Accuracy, Precision recall). feature importance was 49% for model 1 and 51% for model 2.
Situation 2:
I used the probability X of text classification model as an input feature to the Model 2(which contained categorical and numeric features). The performance was almost similar to Situation 1 but here the feature importance of the final model indicated that text model probability had higher importance around 68% and rest of the features had lesser importance.
I want to understand the difference in feature importance of both situations.


texts: Drawing Neural Network diagram for academic papers

texts:
Is there any tool that one can use to draw neural network architecture diagram for research papers?
Example diagram:


texts: How to treat time based ticket prices for train/test split

texts:
I have a dataset of airfare price tickets that were scraped throughout a 6 month period where each observation represents a particular price for a specific flight on a specific date that it was scraped. In other words, a specific unique flight may appear multiple times in the dataset if it's scraped multiple times on different days. For example,

Scrape Date: 11/13/19, Days To Trip: 42, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 290
Scrape Date: 11/22/19, Days To Trip: 33, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 330
Scrape Date: 12/01/19, Days To Trip: 24, Flight: DL1345 , Departure: 12/25, Time: 5:00PM, Price: 349

I know that with time-series data such as stock prices, you want to split your training/testing data so that the data in testing is in the future and comes after the data in training. However, I don't believe the dataset I have would warrant a split like this and I can instead randomly shuffle the data for train/test split but I am not 100% sure on the right call. Should I split the data based on time or can I randomly sample since the price of the tickets don't depend on each other?


texts: Why after adding categorical data the Linear Regression fails?

texts:
Based on a training set we applied a simple Linear Regression on some attributes that all were numeric.

Now we have more attributes in terms of categories and of course we applied one-hot-encoding to transform the categories to binary attributes

Take for example this simple python code:

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8, test_size=0.2)

model = LinearRegression(normalize=True).fit(X_train, y_train)

printErrorMetrics(trueTargets=y_test, predictions=model.predict(X_test))

When the table X has only the original numeric attributes the scores from the printErrorMetrics function (RMSE, etc.) are all good enough

We were expecting better results after adding the one-hot-encoded categories but the results are so worse that the method does not seem to work.

Are we missing anything?

Do we need to preprocess the data after adding the one-hot-encoded columns/attributes?


texts: Is the gradient descent the same if cost function has interaction?

texts:
We know how to determine regression parameters using gradient descent. If

and the cost function is C=|Y-Y(X)|^2, we update b as

where  is the learning rate and  is the partial differential of the cost function C with respect to b.

If in multiple regression there exist an interaction and we want to stick on the linear model formulation (not using tree or other non-linear regressors), such that

and the cost function is still the same, do we just do the same way to update b? i.e. the existence of interaction terms doesn't have impact on gradient descent. I didn't see any difference of gradient descent between with/without interaction.


texts: Best model for Antimicrobial Resistance (AMR) prediction?

texts:
Some classes of problem are best solved by a specific class of machine learning model, due to the structure of the data (e.g. Deep Learning for computer vision).
Prediction of bacterial resistance/susceptibility to antimicrobials (from genotypic data) using Machine Learning methods is a problem that has started receiving interest in recent years.
The following paper (from 2017) analysed the then current literature and found that:

To date, there has not been a consensus about the optimal machine learning model to be used for AST genotype–phenotype prediction, as reflected by the diverse algorithms authors have implemented (Table 1).

Machine learning: novel bioinformatics approaches for combating antimicrobial resistance
Macesica, Polubriaginof, Tatonettib (2017)

Has this changed in the past 2 years?
Is there now a consensus about which models are most effective?

References: Table 1 from paper:


texts: Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function)

texts: A friend of mine has recently started working on R-studio and is interested in filling the NA values in different columns using the above-mentioned function. Also, since he intends to run a time series analysis for every column, what should be the correct approach?


texts: after augmentation validation accuracy going down?

texts:
My main question is about augmentation.
if I process the augmentation I believe it always better than less data
but in my case the validation accuracy going down

train : 7000 images , validation: 3000 images : validation accuracy:0.89

train : 40000 images , validation: 17990 images : validation accuracy:0.85

my augmentation code

def data_augmentation_folder(trainImagesPath,saveDir):
    #X_train=load_training_data(trainImagesPath,"train")
    print("=====================================================")

    X_train = cleanData(trainImagesPath)
    X_train = np.array(X_train)
    print(X_train[0].shape)

    for i in range(5):

        #print(i)

        datagen = ImageDataGenerator(rotation_range=15,
                       width_shift_range=0.1,
                       height_shift_range=0.1,
                       shear_range=0.01,
                       zoom_range=[0.9, 1.25],
                       horizontal_flip=True,
                       vertical_flip=False,
                       fill_mode='reflect',
                       data_format='channels_last',
                       brightness_range=[0.5, 1.5])

        if i==1:
            datagen = ImageDataGenerator(
                       featurewise_center=True,
                                     featurewise_std_normalization=True,
                                     rotation_range=90,
                                     width_shift_range=0.1,
                                     height_shift_range=0.1,
                                     #zoom_range=0.2
            )
        if i==2:
            datagen = ImageDataGenerator(
                   featurewise_center=True,
                                 featurewise_std_normalization=True,
                                 rotation_range=100,
                                 width_shift_range=0.1,
                                 height_shift_range=0.1,
                                 #zoom_range=0.2
            )
        elif i==3:
            datagen = ImageDataGenerator(
                    rescale=1./255,
                    shear_range=0.2,
                    zoom_range=0.2,
                    horizontal_flip=True)
        elif i==4:
             datagen = ImageDataGenerator(
                    rescale=1./255,
                    shear_range=0.1,
                    rotation_range=80,
                    zoom_range=0.1,
                    horizontal_flip=True,
                    brightness_range=[0.5,1.5])

        datagen.fit(X_train)

        for x, y in datagen.flow(X_train, np.arange(X_train.shape[0]),shuffle=True, save_to_dir=saveDir,save_format='jpg',save_prefix='aug'):
            #print(y)
            assert x.shape[1:] == X_train.shape[1:]
            break

  questions

  in which case the validation going down even though I proceeded augmentation?
  what you have to worry about when You proceed the augmentation?


texts:
I'm trying to analyze database performance over a period of time and detect anomalies. The database server consists of many threads that perform different actions. I run a query to determine the number of active threads and the action they are performing.

A sample dataset is below:

My Objective:
I need to analyze over a period of time and determine what is normal at a given timestamp and detect any abnormalities. For example, Monday at 10 am, there are 10 active threads; out of which, there are 2 threads with the action 'Preparestatement' and 10 threads with the action 'Readtable'. Any other thread actions is potentially an anomaly.

As you can see from the image above, the actions (executestatement, Fetchcursor and so on) could be different at each timestamps. I want to understand if the structure of the pandas dataframe is the right choice to meet my objective.


texts: Can we cross-validate the neuron no. and hidden layer no. of a feed-forward neural network?

texts:
Confused about tons of relevant info on the web, so I appreciate if anyone of you can clarify for me. I will use the model to binary classify for your reference.
As for the problem, let's say in a feed-forward network with hidden layer no. to be decided, I want to know how many neurons I should put in (each) hidden layer(s), so

What are the most commonly referred metrics to decide the no. of the neurons (in each hidden layer)? Recall, accuracy, precision or...?
Can we use some kind of cross validations (e.g. Stratified K-Fold) to testify the performance of the network with different no. of hidden neurons and different no. of hidden layers in terms of the above metrics?

Thanks a lot in advance.


texts: What are the disadvantages of having a left skewed distribution?

texts:
I'm currently working on a classification problem and I've a numerical column which is left skewed. i've read many posts where people are recommending to take log transformation or boxcox transformation to fix the left skewness.

So I was wondering what would happen If I left the skewness as it is and continue with my model building? Are there any advantages of fixing skewness for classification problem (knn, logistic regression)?


texts: Logic behind the Statement on Non-Parametric models

texts:
I am currently reading 'Mastering Machine Learning with scikit-learn', 2E, by Packt. In Lazy Learning and Non-Parametric models topic in Chapter 3- Classification and Regression with k-Nearest Neighbors, there is a paragraph stating-
Non-parametric models can be useful when training data is abundant and you have little prior knowledge about the relationship between the response and the explanatory variables. kNN makes only one assumption: instances that are near each other are likely to have similar values of the response variable. The flexibility provided by the non-parametric models is not always desirable; a model that makes assumptions about the relationship can be useful if training data is scarce or if you already know about the relationship.
My doubt is that I am able to reason out the logic behind the statement- "a model that makes assumptions about the relationship can be useful if training data is scarce or if you already know about the relationship." But, I am not able to see any logic for the first statement- "Non-parametric models can be useful when training data is abundant and you have little prior knowledge about the relationship between the response and the explanatory variables."
Please help me to find out the logic behind it other than for computation. Thanks for your time and consideration.


texts: feature selection using genetic algorithm in Python?

texts:
I have a dataset of 4712 records and 60+ features working on a binary classification problem. I already tried out all the feature selection approaches like filter, embedded and wrapper but am just curious to learn and try genetic algorithm for feature selection.

The reason for choosing genetic algorithm is because I guess it will just provide me the best model fit based on best features.

1) I understand it might take time but would you people help me know how can I do this in Python?

2) In addition, is genetic algorithm any different or better than all other feature selection approaches discussed above? What are its disadvantages?

Is there any python packages and tutorial available on how to use this?

I see tutorials but they all about the theory of genetic algorithm

Can you help me by sharing any tutorial or package for genetic algorithm?

post update


texts:
I have a set of climate data (temperature, pressure and moisture for example), $X$, $Y$, $Z$ which are matricies with dimensions $n \times p$ where $n$ is the number of observations and $p$ is the number of spatial points.

Previously, to investigate modes of variability in dataset $X$, I simply performed a empirical orthogonal function (EOF) analysis OR Principle component Analysis (PCA) on $X$. This involved decomposing (via SVD), the matrix $X$.

To investigate the coupling of the modes of variability of $X$ and $Y$, I used maximum covariance analysis (MCA) which involved decomposing a covariance matrix proportional to $XY^{T}$.

However, if I wish to looked at all three datasets, how do I go about doing this? One idea I had was to form a fourth matrix, $\Sigma$, which will be the 'feature' concatenation of the three datasets:

$\Sigma = [X, Y, Z]$

so that my matrix $\Sigma$ will have dimensions $n \times 3p$.

I would then use standard PCA/EOF analysis and use SVD to decompose this matrix $\Sigma$ and then I would obtain modes of variabiilty with size $3p \times 1$ and thus subsequently the mode associated with $X$ is the first $p$ values, the mode associated with $Y$ is the second set of $p$ values and the mode associated with $Z$ is the last $p$ values.

Is this correct? Or can anyone suggest a better way of looking at the coupling of all three (or more) datasets?

Thank you so much!


texts: What is the "novel reinforcement learning algorithm" in AlphaGo Zero?

texts:
For some reason, AlphaGo Zero isn't getting as much publicity as the original AlphaGo, despite its incredible results. Starting from scratch, it's already beaten AlphaGo Master and has passed numerous other benchmarks. Even more incredibly, it's done this in 40 days. Google names it as "arguably the best Go player in the world".

DeepMind claims this is a "novel form of reinforcement learning" - is this technique truly novel? Or have there been other times when this technique was used- and if so, what were their results? I think the requirements I'm talking about are 1) no human intervention and 2) no historical play, but these are flexible.

This appears to be a similar question, but all the answers seem to start from the assumption that AlphaGo Zero is the first of its kind.


texts: How do I find all the drastic changes in the stock and fetch news for those days of the stock?

texts: I am analyzing MindTree company's stocks as an internship project using python pandas and have been given the task to find all the drastic changes by looking at the produced plot for the stock and fetch news for those days of the stock. I am not sure what is considered a drastic change in the stock market. How do I find all the drastic changes in the stock? Also which news API should I use to find the news for those specific days?


texts: Loss function for Hierarchical Multi-label classification

texts:
I am looking to try different loss functions for a hierarchical multi-label classification problem. So far, I have been training different models or submodels (e.g., a simple MLP branch inside a bigger model) that either deal with different levels of classification, yielding a binary vector. I have been using BCE and summing all the losses existing in the model before backpropagating.

I am considering trying other losses like MultiLabelSoftMarginLoss and MultiLabelMarginLoss. What other loss functions are worth to try? hamming loss perhaps or a variation?
Is it better to sum all the losses and backpropagate or do multiple backpropagations?

Thanks in advance!


texts:
I have several sequences of univariate real-valued time-series data. The sequences are of different lengths and right now I cannot batch them and feed them to a network.
What is the correct procedure to pad these sequences? Is it even possible in this case since I can't use any number as a special symbol?

UPDATE 1

I'm working with arbitrary univariate time-series data (not related to one specific domain, unbounded range). To give example of one such a series consider standardized stock dataset (only first 10 elements shown):

d = array([-0.37807043, 0.14321786, -0.37807043, 0.13478392, 0.18733381,
   1.19576774, 0.25675156, 0.26064414, 0.30930144, 0.38650436])


texts: Error: float() argument must be a string or a number, not 'StandardScaler'

texts:
I am getting this error for the mentioned code. Please help me with this as I am a beginner and learning machine learning

# Polynomial Regression

# Importing the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('Position_Salaries.csv')
x = dataset.iloc[:, 1:2].values
y = dataset.iloc[:, 2].values

#Feature scaling
from sklearn.preprocessing import StandardScaler
sc_x=StandardScaler()
sc_y=StandardScaler()
x=sc_x.fit(x)
x=sc_x.fit(y)

#Fitiing SVR to the dataset
from sklearn.svm import SVR
regressor = SVR(kernel = 'rbf')
regressor.fit(x,y)

#Predict the results
regressor.predict(np.array([6.5]).reshape(1, 1))

#Visualizing the results
plt.scatter(x,y,color = 'red')
plt.plot(x,regressor.predict(x))
plt.label("Positon Level")
plt.label("Salary")
plt.show()


texts: Neural Network Hidden Neuron Selection Strategy

texts:
I'm trying to determine what is the best number of hidden neurons for my MATLAB neural network. I was thinking to adopt the following strategy:

Loop for some values of hidden neurons, e.g. 1 to 40;
For each NN with a fixed number of hidden neurons, perform a certain number of   training (e.g. 40, limiting the number of epoch for time reasons: I was thinking to doing this because the network seems to be hard to train, the MSE after some epochs is very high)
Store the MSE obtained with all the nets with different number of hidden neurons
Perform the previous procedure more than 1 time, e.g. 4, to take into account the initial random weight, and take the average of the MSEs
Select and perform the "real" training on a NN with a number of hidden neurons such that the MSE previously calculated is minimized

The MSE that I'm referring is the validation MSE: my samples splitting in trainining, testing and validation to avoid overfitting is 70%, 15% and 15% respectively)

Other informations related to my problem are:
fitting problem
9 input neurons
2 output neurons
1630 samples

This strategy could be work? Is there any better criterion to adopt? Thank you

Edit: Test done, so the result suggest me to adopt 12 neurons? (low validation MSE and  number of neurons lower than 2*numberOfInputNeurons? but also 18 could be good...


texts: Normalizing / standardizing training and validation data

texts:
Say I split my data to 80% training and 20% test/validation and I want to standardize it, I think I'm right in saying I shouldn't standardize across 100% of the data, and then do the split, because then the validation has some insight into the training data?

I'm not sure if I should either

1) Generate the mean and standard deviation stats on the 80% of training data and then apply the same mean/standard deviation to standardize the validation data.

Or 2) Standardize the training data, and then standardize the validation data, i.e mean and SD is derived from the 80% for the training data, and then mean/SD is derived separately on the 20% of validation data?

Many thanks


texts:
I have some trained models that I could import and compare, but then the same code gave:

"WARNING tensorflow:From 'where its saved' colocate with (from tensorflow.python.framework.ops)
  is deprecated and will be removed in a future version.
Instructions for updating:
  Colocations handled automatically by placer."

As well as 2 other deprecation warnings.

The code that I am using to import the models is:

"model_1 = load_model("filepath\\filename.h5")"

Restating my computer fixed the issue, but I'm curious as to why this happened, if anyone can explain.


texts: Google Collaboratory Downloaded Libraries and Files Location

texts:
So, when using Google Collab, by default it's actually a built-in Python environment in the browser, right? Usually, user will import library from Unknown location into Collab.

I said Unknown location because I didn't know where all these libraries are stored. Let say, I'm downloading new library such as Numpy with !pip install numpy, and of course can be checked by invoking !pip list.

So, where does all this libraries are installed? I checked on my Drive and I can only found the .ipynb files, not the libraries.


texts: Can categorical features be linearly distributed?

texts:
Still on my early days, so I cannot really say whether categorical values can be used when linear regression is a key element.

Judging from the below plot, based on my dataset, I'd say this possible (judging from this answer) and the graph below shows I can further use them, can't I?

My question is, can I use the linear regression (e.g. the one from sklearn) to calculate the RSME?


texts: Drop NA values in an imported xlxs document with R

texts:
I have the attached document.  I've imported an xlxs document into R.  I would like to get rid of all rows containing NA.  I've tried the following but it isn't working.

# install packages Note# tidyverse installs packages for dplyr and ggplot2
install.packages("tidyverse")
install.packages("readxl")

#open the library
library(tidyverse)
library(readxl)

setwd("~/Documents/UofL/Data_Science/CECS_635/week_3/")

un <- read_excel("UnitedNations.xlsx")

un1 %>% drop_na(un,)


texts: How can I improve the performance of my DQN?

texts:
I created a deep Q network to play snake. The code works fine, except for the fact that performance doesn't really improve over the training cycle. At the end, it's pretty much indistinguishable from an agent that takes random actions. Here's the training code:

def train(self):
        self.build_model()
        for episode in range(self.max_episodes):
            self.current_episode = episode
            env = SnakeEnv(self.screen)
            episode_reward = 0
            for timestep in range(self.max_steps):
                env.render(self.screen)
                state = env.get_state()
                action = None
                epsilon = self.current_eps
                if epsilon > random.random():
                    action = np.random.choice(env.action_space) #explore
                else:
                    values = self.policy_model.predict(env.get_state()) #exploit
                    action = np.argmax(values)
                experience = env.step(action)
                if(experience['done'] == True):
                    episode_reward += 5 * (len(env.snake.List) - 1)
                    episode_reward += experience['reward']
                    break
                episode_reward += experience['reward']
                if(len(self.memory) < self.memory_size):
                    self.memory.append(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']))
                else:
                    self.memory[self.push_count % self.memory_size] = Experience(experience['state'], experience['action'], experience['reward'], experience['next_state'])
                self.push_count += 1
                self.decay_epsilon(episode)
                if self.can_sample_memory():
                    memory_sample = self.sample_memory()
                    #q_pred = np.zeros((self.batch_size, 1))
                    #q_target = np.zeros((self.batch_size, 1))
                    #i = 0
                    for memory in memory_sample:
                        memstate = memory.state
                        action = memory.action
                        next_state = memory.next_state
                        reward = memory.reward
                        max_q = reward + self.discount_rate * self.replay_model.predict(next_state)
                        #q_pred[i] = q_value
                        #q_target[i] = max_q
                        #i += 1
                        self.policy_model.fit(memstate, max_q, epochs=1, verbose=0)
            print("Episode: ", episode, " Total Reward: ", episode_reward)
            if episode % self.target_update == 0:
                self.replay_model.set_weights(self.policy_model.get_weights())
        self.policy_model.save_weights('weights.hdf5')
        pygame.quit()

Here are the hyperparameters:

learning_rate = 0.5
discount_rate = 0.99
eps_start = 1
eps_end = .01
eps_decay = .001
memory_size = 100000
batch_size = 256
max_episodes = 1000
max_steps = 5000
target_update = 10

Here's the network architecture:

model = models.Sequential()
model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros', input_dim = 400))
model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros'))
model.add(Dense(5, activation = 'softmax', kernel_initializer = 'random_uniform', bias_initializer = 'zeros'))
model.compile(loss='mean_squared_error', optimizer = 'adam')

For reference, the network outputs 5 values because of the 4 directions the snake can move, and 1 extra for taking no action. Also, instead of being a screenshot of the game like in a traditional DQN, I pass in a 400 dimensional vector as representation of the 20 x 20 grid that the game takes place in, though I believe that it should still work fine with this. The agent receives a reward of 1 for moving closer to the food or eating it and receives a reward of -1 if it dies. How can I get the performance to improve?


texts: Avoid reloading DataFrame between different python kernels

texts:
Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks?

I'd be looking for something, which is conceptually similar to MATLAB's persistent variables. There it is possible to call a custom function / library from multiple individual editors (notebooks), and have that external function cache some result (or large table).

Mostly I would like to avoid reloading a heavily used table (which is loaded through a custom library that is is called from the notebooks ), since reading it takes around 2-3 minute whenever I start a new analysis.


texts: Tokenizing words of length 1, what would happen if I do topic modeling?

texts:
Suppose my dataset contains some very small documents (about 20 words each). And each of them may have words in at least two languages (combination of malay and english, for instance). Also there are some numbers inside each of them.

Just out of curiosity, while usually customizable, why are some tokenizers choose to ignore tokens that are just numbers by default, or anything that doesn't meet certain length? For example, the CountVectorizer in scikit-learn ignores words that do not have more than 1 alphanumeric. And the tokenizer utility in gensim ignores words with digits.

I used CountVectorizer in the end and made it to accept words containing digits and words with length 1 as well. This is because I needed exact match, as slight differences in those words of length 1 may point to a different document.

I am currently trying topic modeling (gensim's LSI) to perform topic analysis, but the main intention of doing that is to be able to reduce the dimension so I can feed it to spotify's annoy library for quick approximate searching (from 58k features, to 500 topics). Also I expect it to reduce the time and memory taken to compute classifier models.

So the question is really, if I tokenize words of length 1, would it makes sense to perform topic modeling? Even if I do a brute force search by just comparing cosine similarity (no fancy ANN), would it affect the precision or accuracy (i.e. would it be able to recognize the slight change in words of length 1 in query and retrieve the right document)?


texts:
As it is mentioned in the F1 score Wikipedia, 'F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0'.

What is the worst condition that was mentioned?

Even if we consider the case of: either precision or recall is 0.
The whole F1-score value becomes undefined. Because when either precision or recall is to be 0, true postives should be 0. When the true positives value becomes 0, both the precision and recall become 0.


texts: Classification technique for unsupervised data?

texts:
I have unsupervised data (i.e this data doesn't have any target variable through which I can learn it's prior behaviour) it is a mix of continuous and categorical data. Now I want to classify the test data into three categories on basis of my unsupervised data.

The approach I took is to first do the clustering of unsupervised data, use this categorised data as a base data for preparing a new  model that predicts on top of it.

I want know whether this approach correct or not or is there better way for classifying test set? Particular algorithm I need to follow for this?

I am doing this in R.

The approach is to modify the training set data so that this can be used to properly predict the test data. Here target variable is missing in train and test set.


texts:
I want to know if my thinking is correct?

Total Images = 120,000

So in the Keras Fit Generator function,
I made the batch_size = 24 and steps per epoch = 500 ,which is 12000 and only one-tenth of the total dataset.

Therefore I should make the "epochs = 100" if I actually want the real epoch to be 10.

i.e. It will actually take 10 epochs to go through the entire 120,000 images once.


texts: Gradient descent and partial derivatives

texts:
I'm on the first chapter of Nielsen's book on neural networks. He gives an example of a gradient descent and that change in function $C(v_1,v_2)$ is described by this formula:

$$\begin{eqnarray}\Delta C \approx \frac{\partial C}{\partial v_1} \Delta v_1 +\frac{\partial C}{\partial v_2} \Delta v_2\tag{7}\end{eqnarray}$$

Though I'm familiar with partial derivatives, i'm confused about how you would compute partial derivatives of this function.

Could someone give an example please?


texts: Extrapolating GLM coefficients for year a product was sold into future years?

texts:
I've fit a GLM (Poisson) to a data set where one of the variables is categorical for the year a customer bought a product from my company, ranging from 1999 to 2012. There's a linear trend of the coefficients for the values of the variable as the year of sale increases.

Is there any problem with trying to improve predictions for 2013 and maybe 2014 by extrapolating to get the coefficients for those years?


texts: Calculate confidence score of a neural network prediction

texts:
I am using a deep neural network model to make predictions. My problem is a classification(binary) problem. I wish to calculate the confidence score of each prediction. As of now, I use predict_proba of sklearn to get probability of a class and consider it as the model confidence for that prediction. Is it the right approach? What could be a better way to calculate the confidence score?
Kindly add relevant literature to support your approach.


texts: Determining whether a Machine Learning model is overfitted with regard to the stability of the features

texts: I need to know how would I get to know if I have overfitted my Machine Learning model on the train data. The performance metric I have used is Logistic Loss. Does the stability of the features affect the performance of my model? If yes, how do they relate?


texts: Simultaneous inference on many gpu, question

texts:
I want to do boosting for instance segmentation. Lets say I have 5 different models for this and I have 5 GPUs. WHat I want to do is to do the inference on each model on each GPU, and combine the masks on CPU for example. I want to do this in this configuration, because the speed is the limiting factor. If all 5 cards starts to do inference which takes a second lets say, after a second I will have 5 masks generated which normally would take 5 seconds.
Did somebody did try to do it ?
How to approach it.


texts: What is search engine life cycle from information retrieval perspective

texts: I am already aware of usage of information retrieval for search engine. But still trying to understand what can be possible life-cycle phases for a search engine. I mean for a search engine is it only about build a search engine, deploy and occasional maintenance? I assume so. Perhaps at best augment it with some deep learning or NLP algorithm. Is my assumption accurate or there can be more to it for the search engine life cycle, purely from information retrieval perspective?


texts: Available data about 'likers' as a page on Facebook

texts:
I am hoping to model the characteristics of the users of a specific page on Facebook, which has roughly 2 million likes. I have been looking at the Facebook SDK/API, but I can't really see if what I would like to do is possible. It seems that the users share quite different amounts of data so I probably discard a lot of users and only use the ones with a quite open public profile. I would like to have the following data:

1) See the individuals that have 'liked' the page.

2) See the list of friends for each person that have 'liked' the page.

3) See gender for each person (optional)

4) See other pages that each person has liked (optional)

Could anyone tell me if it is possible to get this data? As mentioned earlier it is okay if I discard data for users that don't like to share this data.


texts: Confusion matrix, when mistakes below diagonal are better than above the diagonal

texts:
I have a classification problem and I am producing a confusion matrix. Ideally one wants to get all results in the diagonal. I get quite many points around diagonal for different algorithms. Still for my use-case I want to favor algorithms that underpredict the class (I have ordinal data) and not overpredict.
Is there a metric that can measure under and overprediction and rate those errors with a different weight? The typical accuracy, precision terms assume that all mistakes are the same.
Of course I can try to implement my own metric but I am quite sure that I am not the first that is having this issue.
Any metric available that you know already?
Thanks
Alex


texts:
I am a bit new to this, but I just had a quick question about clustering vs. classification. I have a bunch of texts that I want to classify. There are 4 classes I have at the moment, but texts can belong to more than one class. What I have seen so far is to do 4 binary classifications, but I was wondering if there is a classification algorithm where I can achieve texts belonging to more than one class. Or could I do this with clustering having overlapping clusters?
I am trying to do this in Python.


texts: Incorporating new features in document similarity task

texts:
I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we can infer its TfIDF vector. Finally, we compare the query's TfIDF vector to all the vectors of the documents in the corpus using cosine similarity, which finds us the "most similar" texts.

My question is, is there a way to incorporate more micro structure features into the pipeline such that similar document retrieval will perform better, and if so how?

By this I mean, can we use Part of Speech tagging, intent recognition or other methods, as extra features in the pipeline. I am looking for more unsupervised methods here as I have a lot of data (1TB) and it is all unstructured and un-tagged / un-labelled, but supervised suggestions will also be welcome. I appreciate that by incorporating new features, we will most likely have to move to a completely different model paradigm as TfIDF and cosine similarities will not work depending on the structure of the new features.

NOTE: I have already performed a significant amount of text pre-processing on the corpus e.g. stemming, tokenizing, word replacement, stop word removal etc.


texts: What to do with stale centroids in K-means

texts: When I run Kmeans on my dataset, I notice that some centroids become stale in the they are no longer the closest centroid to any point after some iteration. Right now I am skipping these stale centroids in my next iteration because I think those centroids no longer represent any useful set of the data, however I wanted to know if there are other reasonable ways to deal with these centroids.


texts: How do I choose number of clusters when Eigengap heuristic suggest 1 for spectral clustering?

texts:
Eigengap heuristic Method suggest number of clusters k is usually given by the value of k that maximizes the eigengap (difference between consecutive eigenvalues).
I plotted the Eigenvalue distribution for my Data:
[

Eigengap heuristic would suggest cluster number: 1.
How should I choose number of clusters in this case ?


texts: Applying a keras model working with greyscale images to RGB images

texts:
I followed this basic classification TensorFlow tutorial using the Fashion MNIST dataset.
The training set contains 60,000 28x28 pixels greyscale images, split into 10 classes (trouser, pullover, shoe, etc...).
The tutorial uses a simple model:

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])

This model reaches 91% accuracy after 10 epochs.

I am now practicing with another dataset called CIFAR-10, which consists of 50,000 32*32 pixels RGB images, also split into 10 classes (frog, horse, boat, etc...).

Considering that both the Fashion MNIST and CIFAR-10 datasets are pretty similar in terms of number of images and image size and that they have the same number of classes, I naively tried training a similar model, simply adjusting the input shape:

  model = keras.Sequential([
     keras.layers.Flatten(input_shape=(32, 32, 3)),
     keras.layers.Dense(128, activation='relu'),
     keras.layers.Dense(10)
  ])

Alas, after 10 epochs, the model reaches an accuracy of 45%. What am I doing wrong?

I am aware that I have thrice as many samples in an RGB image than in a grayscale image, so I tried increasing the number of epochs as well as the size of the intermediate dense layer, but to no avail.

Below is my full code:

import tensorflow as tf
import IPython.display as display
from PIL import Image
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import pdb
import pathlib
import os
from tensorflow.keras import layers #Needed to make the model
from tensorflow.keras import datasets, layers, models

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

IMG_HEIGHT = 32
IMG_WIDTH = 32

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck']

train_images = train_images / 255.0
test_images = test_images / 255.0

def make_model():
      model = keras.Sequential([
         keras.layers.Flatten(input_shape=(IMG_HEIGHT, IMG_WIDTH, 3)),
         keras.layers.Dense(512, activation='relu'),
         keras.layers.Dense(10)
      ])
      model.compile(optimizer='adam',
                   loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                   metrics=['accuracy'])
      return model

model=make_model()
history = model.fit(train_images, train_labels, epochs=10)


texts: Why a restricted Boltzman machine (RBM) tends to learn very similar weights?

texts:
These are 4 different weight matrices that I got after training a restricted Boltzman machine (RBM) with ~4k visible units and only 96 hidden units/weight vectors. As you can see, weights are extremely similar - even black pixels on the face are reproduced. The other 92 vectors are very similar too, though none of weights are exactly the same.

I can overcome this by increasing number of weight vectors to 512 or more. But I encountered this problem several times earlier with different RBM types (binary, Gaussian, even convolutional), different number of hidden units (including pretty large), different hyper-parameters, etc.

My question is: what is the most likely reason for weights to get very similar values? Do they all just get to some local minimum? Or is it a sign of overfitting?

I currently use a kind of Gaussian-Bernoulli RBM, code may be found here.

UPD. My dataset is based on CK+, which contains > 10k images of 327 individuals. However I do pretty heavy preprocessing. First, I clip only pixels inside of outer contour of a face. Second, I transform each face (using piecewise affine wrapping) to the same grid (e.g. eyebrows, nose, lips etc. are in the same (x,y) position on all images). After preprocessing images look like this:

When training RBM, I take only non-zero pixels, so outer black region is ignored.


texts: Unbalanced classes -- How to minimize false negatives?

texts:
I have a dataset that has a binary class attribute. There are 623 instances with class +1 (cancer positive) and 101,671 instances with class -1 (cancer negative).

I've tried various algorithms (Naive Bayes, Random Forest, AODE, C4.5) and all of them have unacceptable false negative ratios. Random Forest has the highest overall prediction accuracy (99.5%) and the lowest false negative ratio, but still misses 79% of positive classes (i.e. fails to detect 79% of malignant tumors).

Any ideas how I can improve this situation?

Thanks!


texts: Apply LSTM to each matrix element with Keras

texts:
I'm trying to apply a LSTM/GRU to each entry of a matrix $X$

note: Each matrix element is a time-series, so shape of X is (batch_size, rows, cols, time_steps, dims)

$
y_{i,j}=
    \begin{cases}
      0, & \text{if}\ x_{i,j}\small[0\small] = 0 \\
      LSTM(x_{i,j}), & \text{otherwise}
    \end{cases}
$

, where:

$x_{i,j} := \text{element (i,j)  of matrix X. This element is a time series}$
$x_{i,j}\small{[0]} := \text{ First tiem step of time series $x_{i,j}$ }$

$y_{i,j} := \text{LSTM output for $x_{i,j}$}$

PROBLEM

I've managed to evaluate the condition and to apply the LSTM to each entry, but I can not build a keras model with these outputs, beacuse of the typical error:

AttributeError: 'NoneType' object has no attribute '_inbound_nodes'

CODE

import keras

def convert_to_layer(X):
    return X

def gather(r):
    def gather_(X):
        return X[:,r,:]
    return gather_

n_rows = 2
n_cols = 2
time_steps = 20
dims = 2
units = 32

M = keras.layers.Input(batch_shape=(1,n_rows,n_cols,time_steps,dims))

# Serialize inputs
M2 = keras.layers.Reshape((n_rows*n_cols,time_steps,dims))(M)

# Gather element 0 (only work on this element for simplicity...)
seq = keras.layers.Lambda(gather(0))(M2)

# Create condition
cond = keras.backend.equal(keras.backend.sum(keras.backend.abs(seq),axis=-1),0)

# Apply if statement
y = keras.backend.switch(cond,
                          lambda: keras.layers.GRU(units,return_sequences=True)(seq),
                          lambda: keras.backend.zeros(shape=(1,time_steps,units)))

y = keras.layers.Lambda(convert_to_layer)(y)
print(y)

model = keras.models.Model(M,y)

OUTPUT

Tensor("lambda_311/Identity:0", shape=(1, 20, 32), dtype=float32)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-76-983e9545cca1> in <module>()
     29 print(y)
     30
---> 31 model = keras.models.Model(M,y)

5 frames

/usr/local/lib/python3.6/dist-packages/keras/engine/network.py in build_map(tensor, finished_nodes, nodes_in_progress, layer, node_index, tensor_index)
   1378             ValueError: if a cycle is detected.
   1379         """
-> 1380         node = layer._inbound_nodes[node_index]
   1381
   1382         # Prevent cycles.

AttributeError: 'NoneType' object has no attribute '_inbound_nodes'

QUESTIONS

The problem is with the keras.backend.switch (if the switch is removed and the output of the model is the output of the gru_cell unconditionally, the model can be build)

Why isn't the model compiling?
Any other alternative way of computing the LSTM over the matrix $X$


texts: Matrix dimensions issue on calculating Softmax Derivatives

texts:
I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function.
For simplicity I am only interested to get derivatives of W2 w.r.t Loss.
I think for a single input x1, my math works out OK(below) and the dL/dW2 correctly comes same dimension as W2 (DxC). However I struggle on vectorizing it for all inputs X.
In specific, in non-vectorize version the derivative of softmax(here A2) w.r.t its input (here Z2) has dimensions CxC since the function maps . This is explained very well in this link. But what is the dimension of this matrix in vectorized version for all inputs? It cannot be still CxC since that is only for a single input.
I have marked in red where I am not sure what the correct dimensions are. Please if anyone can review my math and post an answer. I know there might be different formulation than what I have done here but I appreciate if you show me where I got it wrong. Thanks.
]4


texts: How can I filter and balance a Windowed Tensorflow dataset with a binary classification label, based on the label?

texts:
I have an unbalanced tensorflow windowed dataset with labels (over 90% negative examples) which I am trying to balance by filtering. I am having trouble filtering as my windowed dataset with labels does not fall under the cases I have come across in my searches, or the tensorflow documentation.

I'm working on a model to predict a binary classification based on time series data. I start with a time series dataframe with a number of columns (price, volume, etc.) where each row is one minute.

Currently I am still stuck on filtering the different labels. My next step after filtering would be to get the size of both filtered datasets, find the smaller size (n), and then concatenate the smaller dataset with (n) elements from the bigger dataset, after shuffling the bigger dataset. This way I would have a balanced dataset with an equal number of 1 and 0 labels. If you have a better Idea I would be happy to hear it.

EXPLAINING MY CODE: DFrame is a pandas dataframe with columns such as price, volume, etc., and each row is a different minute, with the first row being the earliest/oldest time period. The last column of DFrame is the classifier 0 or 1.

I then create a tensorflow dataset from slices with the first input being all the DFrame columns except for the label which is in the last column, and the second input (the label) being the last column which is the classifier.

I then use the window function to create windows of size (hindsight) which is currently 512, meaning (If i'm not mistaken) it takes the previous 511 minutes as well as the current minute and uses this as a rolling window to associate with the label of the current minute. So my understanding is that x is then an array of 512 arrays, from the row of the current minute to the row of 511 minutes ago, and the y is the label of the current minute. So x is an array of 512 arrays (rows for each minute, from the dataframe), and y is just one integer, 1 or 0.

Ideally I would like to be able to apply the same balancing logic to a multiclass classification problem, where I essentially add additional labels for additional price movement ranges.

The error comes from the filter. The model seems to run without that, and even trains my keras model. as explained I would actually want to add more code after the filter once I get it working to balance the dataset but I need to filter it first.

tensor= tf.data.Dataset.from_tensor_slices((tf.constant(DFrame[DFrame.columns.values[:-1]].values), tf.constant(DFrame[DFrame.columns.values[-1]].values)))

tensor = tensor.window(hindsight,1,1,True)

tensor = tensor.shuffle(1000)

tensor = tensor.filter(lambda x,y: tf.equal(y, 0))

tensor = tensor.flat_map(lambda x,y:tf.data.Dataset.zip((x.batch(hindsight), y.batch(1))))

tensor = tensor.batch(Batch_size).prefetch(1)

TypeError: Failed to convert object of type <class 'tensorflow.python.data.ops.dataset_ops._VariantDataset'> to Tensor. Contents: <_VariantDataset shapes: (), types: tf.int64>. Consider casting elements to a supported type.


texts: What are the individual models within a machine learning ensemble called?

texts:
I am aware that an ensemble machine learning model is a stack of two or more machine learning models. Is there a word to refer to those individual models that go into the ensemble model? (i.e. a specific term/jargon?)

I wish to refer to them and am not aware of any specific terminology to describe them. I've considered:

"sub-models"
"component models"
even "pancake models" (since people
generally understand what's being referring to when they hear 'one of the
pancakes in the model')


texts: Affinity propagation fit, fit_predict and predict methods

texts:
I am using sklearn affinity propagation algorithm as below.

affprop = sklearn.cluster.AffinityPropagation(affinity="precomputed", damping=0.5)

I also have a similarity matrix created for the data I am using. Now I want to use my similarity matrix to use in the affinity propagation model.

In sklearn they have different methods for this such as fit, fit_predict, predict. So, I'm not sure what to use.

Is it correct if I use,

affprop.fit(my similarity matrix)


texts: In Machine Learning, what is the point of using stratified sampling in selecting test set data?

texts:
I am currently learning machine learning via this book "Hands-On Machine Learning with Sci-kit learn and Tensorflow" by Aurelien Geron.

In page 76 and 77, the author talks about using stratified sampling so that your test set would be more representative of the whole data. I didn't really understand the point of this since it would not affect the accuracy of your training model ? or how would selecting better test sets affect the accuracy of your training model ?