texts: Is it vital to use weighted metrics with weighted loss functions?

texts:
When one has an imbalanced machine learning problem, let's say a binary classification problem where class 0 is a majority class and class 1 is a minority class. Here class 1 is the most important class for you. Now your dataset is composed of 100 samples: 70 samples from class 0 and 30 samples from class 1.
In this case, usually, one defines and uses a weighted binary cross-entropy loss function to calculate the loss.
classweights = torch.FloarTensor([1.000, 1.300])
weighted_loss = torch.nn.BCELoss(weight = classweights)

For calculation of metrics like F1-score, is it required that these weights are used?
sample_classweights = sklearn.utils.class_weight.compute_sample_weight(class_weight, y_true)
classification_report = sklearn.metrics.classification_report(y_true, y_pred, sample_weight=sample_classweights)

Any hint or help is appreciated.


texts: Classification for High dimensional data

texts:
I am trying to build a classification model with 1000+ features and
I am performing the below steps but I am not sure if it's the correct way to do it

Step 1. Finding Variable Importance using the entire dataset

Step 2. Build a model using the subset of the features returned from Step 1
Once I get my new data I am splitting the data in training and testing and then performing cross validation on training data.

My question is should I use CV while performing variable importance on the entire dataset?

Thanks for your help!


texts: storing a huge dataset in h5py file format

texts: I work on preparing the luna16 dataset for feeding into the CNN model, after reading all '.mhd' files and the labels(0, 1) in the annotated CSV file, I get a memory error, I know the problem is because of the data size is need a lot of RAM that I do not have on my computer, is there any solution to that?


texts:
Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short questions and would appreciate if you could clarify those:

Are the attention scores the only place (besides the loss) where masks are needed or should the input be masked out as well?

I am asking because is see implementations where a linear layer for the query, key and values is with bias=False is used.

Is the reason for setting bias=False to have zeros preserved in the output of the layers or is there a different explanation?
Should a padding_idx be used when learning word embeddings in order to zero out the padded tokens?


texts: What is the difference between KL-divergence, JS-divergence, Wasserstein distance and MMD?

texts:
I was reading about different distribution distances, and came across

Kullback-Leibler divergence
Jensen-Shannon divergence
Wasserstein distance
Maximum mean discrepancy (MMD)

The book was too abstract for me to absorb. What are the differences between them, actually?


texts: Given a regression based model with many feature variables; what tools would you utilize to figure out which feature variables add the most variance?

texts:
Given a hypothetical dataset {S} with 100 X feature variables and 10 predicted Y variables.

X1
...
X100
Y1
....
Y10

1
..
2
3
..
4

4
..
3
2
..
1

Let's say I want to improve the accuracy of Y1. I am prepared to constraint/remove the input variables in order to increase the accuracy. How would I go about finding the culprits for making Y1 more variable than needed?
E.g. I find that X49 adds the biggest swing in variance with Y1 and after constraining it Y1 is fitted better.
How would I go about finding it's X49?
EDIT: I'm asking for approaches towards sensitivity analysis. Not deciding which variables need to be removed. Let's assume all 100 X variables are important but some need to be constrained (e.g. X49)


texts: XGboost with group-specific and individual-specific data

texts: My dataset consists of a combination of two databases. One database consists of individual-level data on the characteristics and compensation of the top five executive officers of big American companies. The second database consists of balance sheet information on the companies of these executives. I use 4 individual-specific variables (e.g. executive age) from the first database and about 15 to 20 company-specific variables from the second database (e.g. total company profit). This means that for executives of the same company, only 4 variables are unique to the specific executive and all company-specific variables are the exact same for all 5 executives. I want to use XGboost to predict executive compensation using these individual-specific and company-specific variables. Is it okay to use 5 executives from the same company or should I use only 1 executive (CEO) for each company? This would decrease the sample size from 8926 to 1703 observations, but it would ensure that all of my observations are truly “unique”. Thank you very much!


texts: What are features in the context of reinforcement learning?

texts: In machine learning, "feature" is a synonym for explanatory variables. I know what a feature is. However, in the specific case of RL, it's not clear to me what features are. What are "features" in the specific case of RL? What are examples of features in RL? Which RL algorithms require the specification of features? Why do we need "features" in RL?


texts: Formal definition for parameter setting in data mining context

texts:
While reading this material on decision trees, I came across the following statement:

  The construction of decision tree classifiers does not require any domain knowledge or parameter setting, and therefore is
  appropriate for exploratory knowledge discovery.

I have couple of doubts related to this statement. First one is the definition of parameter setting in this context and another one is how being without parameter setting is apt for exploratory knowledge discovery?


texts: TensorFlow / Keras: What is stateful = True in LSTM layers?

texts:
Could you elaborate on this argument? I found the brief explanation from the docs unsatisfying:

  stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.

Also, when stateful = True is to be chosen? What are practical cases of its use?


texts: What is the best architecture for Auto-Encoder for image reconstruction?

texts:
I am trying to use Convultional Auto-Encoder for its latent space (embedding layer), specifically, I want to use the embedding for K-nearest neighbor search in the latent space (similar idea to word2vec).

My input is 3x224x224 (ImageNet), I could not find any article that elaborates a specific architecture (in terms of number of filters, number of conv layers, etc.)
I tried some arbitrary architectures like:

Encoder:

Conv(channels=3,filters=16,kernel=3)
Conv(channels=16,filters=32,kernel=3)
Conv(channels=32,filters=64,kernel=3)

Decoder:

Deconv(channels=64,filters=32,kernel=3)
Deconv(channels=32,filters=16,kernel=3)
Deconv(channels=16,filters=3,kernel=3)

But I'd like to start my hyper-parameters search from a set up that proved itself on a similar task.
Can you refer me to a source or suggest an architecture that worked for you for this purpose?


texts: How can autoencoders be used for clustering?

texts:
Suppose I have a set of time-domain signals with absolutely no labels. I want to cluster them in 2 or 3 classes. Autoencoders are unsupervised networks that learn to compress the inputs. So given an input $x^{(i)}$, weights $W_1$ and $W_2$, biases $b_1$ and $b_2$, and output $\hat{x}^{(i)}$, we can find the following relationships:

$$z^{(i)} =W_1x^{(i)}+b_1$$
$$\hat{x}^{(i)} =W_2z^{(i)}+b_2$$

So $z^{(i)}$ would be a compressed form of $x^{(i)}$, and $\hat{x}^{(i)}$ the reconstruction of the latter. So far so good.

What I don't understand is how this could be used for clustering (if there is any way to do it at all). For example, in the first figure of this paper, there is a block diagram I'm not sure I understand. It uses the $z^{(i)}$ as the inputs to the feed-forward network, but there is no mention to how that network is trained. I don't know if there is something I'm ignoring or if the paper is incomplete. Also, this tutorial at the end shows the weights learned by the autoencoder, and they seem to be kernels a CNN would learn to classify images. So... I guess the autoencoder's weights can be used somehow in a feed-forward network for classification, but I'm not sure how.

My doubts are:

If $x^{(i)}$ is a time-domain signal of length $N$ (i.e. $x^{(i)}\in\mathbb{R}^{1\times N}$), can $z^{(i)}$ only be a vector as well? In other words, would it make sense for $z^{(i)}$ to be a matrix with one of its dimensions greater than $1$? I believe it would not, but I just want to check.
Which of these quantities would be the input to a classifier? For example, if I want to use a classic MLP that has as many output units as classes I want to classify the signals in, what should I put at the input of this fully-connected network ($z^{(i)}$,$\hat{x}^{(i)}$, any other thing)?
How can I use the learned weights and biases in this MLP? Remember that we assumed that absolutely no labels are available, so it is impossible to train the network. I think the learned $W_i$ and $b_i$ should be useful somehow in the fully-connected network, but I don't see how to use them.

Observation: note that I used an MLP as an example because it is the most basic architecture, but the question applies to any other neural network that could be used to classify time-domain signals.


texts: Do I need an Artificial Intelligence API?

texts: I'm wondering if programmers tend to use AI APIs. And if so, what are they like? And where can I find a nice one for Java?


texts: How to set different weights for different training samples?

texts:
Suppose that your supervised learning training set is made out of 3 different datasets, merged into a big one. Because of the way each of those was labeled before merging, you might be suspicious that one of them (maybe the smallest one) is more "important" than the other ones, meaning that their labels are more reliable. The others might contain more errors.
How could you weight the most reliable data points for the ML model to pay more attention to them and increase the loss when it makes a mistake on those samples? And is there a simple way to implement this using scikit-learn?


texts: Thesis on avalanche prediction using One Class SVM

texts:
I'm doing my thesis on avalanche prediction using machine learning.

For my input features I'm using avalanche accidents with features like slope, altitude, facing direction of the slope, combined with according weather data from the day the avalanche occurred.

I want to predict an avalanche when certain variables combine and create a deadly avalanche situation. So 1: the avalanche occurs. 0: The avalanche does not occur.

The only data in my database are occurred avalanches, I got around 200 samples. So I don't have any data of a non deadly avalanche situation, which is mostly the case.

My question is if a One Class SVM is a good approach to take on this classification?


texts: Feature extraction; similarity and classification of accelerometer data

texts:
I have several expert persons performing the same specific action (for example, squat or leap forward) multiple times. Say 5 persons do 100 squats each. They have an accelerometer attached to the same body parts. I record the accelerometer readings and get 100*5 = 500 data samples. They do it for multiple different actions (squat, push up, leap forward, etc). The way they record the action is as follows:

Start recording (push button)
Do the action
Stop recording (push button)

Now I need to see that another person is doing the actions in the correct order. For example, squat, leap forward, stand up, drop down, push up. I take his accelerometer data and continuously feed it to the classifier that needs to tell me if he now has done exactly a squat action, not a leap or a push-up. So, when the first action, namely squat, was identified, I check against leap forward and so on.
There are several problems with this:

These data samples have different amounts of values, since somebody is squatting a bit slower, others do it a bit faster. So, some data
samples have 250 XYZ values, others have 220 or 270, etc. (in range of
+-50). What I do for now is make stricter rules. I discard all the data samples that exceed 250 readings and for ones that have fewer values than 250, I append the values from the beginning to the end so that it gives 250 in total. Works fine, since there is a windup for every action where the person is standing still for a brief moment before he performs the action. This is not optimal, because the experts need to redo the action if they were too slow (the windup was too long) + I append fake data. What would be a better solution to handle this?
For now I am using Random Forest, AdaBoost classifiers with low/high
pass filtered accelerometer data that I map to 750 columns (250 X, 250 Y, 250 Z) with 1
class column. So the prediction tells me something like 70% leap, 25% squat, 5% push up. The classification is sometimes wrong or not precise enough. Thus, I was thinking of extracting some features from my signal series and feed them to the algorithm instead. My problem is that I do not know what features to extract.

The majority of papers that I found focus on human activity recognition to differentiate between walking, running, ascending, and descending stairs. They were not very helpful in the regard that they have continuous data flow of a person walking/running for hours and they use much more sample data. In contrast, with my task, I have data set instances that are separate from each other.
I am not asking to solve those whole tasks for me, just guide me into some direction with a good explanation of why it might be useful.


texts: Random Forests Feature Selection on Time Series Data

texts:
I have a dataset with N features, each one with 500 instances in time.

For example, let's say that I have the following:
Features x, y, v_x, v_y, a_x, a_y, j_x, j_y,
A sample with 500 examples (rows in a table) for each feature,
A sample with 500 other instances, and a class.

I'd like to select a subset of the features automatically with the Random Forests algorithm. The problem is that the algorithm (I'm using ScikitLearn, RandomForestClassifier), accepts a matrix (2D array) as X input, of size [N_samples, N_features]. If I give the array as it is, that is a vector (len 500) for the feature x, another (len 500) for the feature y, etc., I get a N_samples x N_features x 500 array, which is incompatible with the requirements of RandomForestClassifier.

I tried to unroll the matrix in a vector, like having so 500 x N_features array, but in that way, in the reduction, it considers all the elements independent feature, and breaks my structure.

How can I select/reduce the features keeping the time instances consistent?
(I can use this algorithm, but i'm also open to other libraries and/or algorithms)

My goal is to do classification, so forecasting resources are limitedly useful to me. Also I have the requirement that each sample has those occurrences, and I don't have them as separate samples unfortunately.


texts:
I have a data set of 2300 entries, with 5 variables one of them the dependent variable which is binary.

I fitted a decision try using the rpart function in R over the 4 independent variables, and I had this tiny tree, which means the split was done using only one predictive variable.

Is it normal? How can I add the rest of the nodes?


texts:
I'm trying to use the function torch.conv2d from Pytorch but can't get a result I understand...

Here is a simple example where the kernel (filt) is the same size as the input (im) to explain what I'm looking for.

import pytorch

filt = torch.rand(3, 3)
im = torch.rand(3, 3)

I want to compute a simple convolution with no padding, so the result should be a scalar (i.e. a 1x1 tensor).

I tried this with conv2d:

# I have to convert image and kernel to 4 dimensions tensors to use conv2d
im_torch = im.reshape((im_height, filt_height, 1, 1))
filt_torch = filt.reshape((filt_height, im_height, 1, 1))
out = torch.nn.functional.conv2d(im_torch, filt_torch, stride=1, padding=0)
print(out)

But the result is not what I expected:

tensor([[[[0.6067]], [[0.3564]], [[0.5397]]],
    [[[0.2557]], [[0.0493]], [[0.2562]]],
    [[[0.6067]], [[0.3564]], [[0.5397]]]])

To give an idea of what I'd like, I want to reproduce scipy convolve2d behavior:

import scipy.signal
out_scipy = scipy.signal.convolve2d(im.detach().numpy(), filt.detach().numpy(), 'valid')
print(out_scipy)

which prints:

array([[1.195723]], dtype=float32)


texts: PCA, SMOTE and cross validation- how to combine them together?

texts:
I was reading a lot recently about PCA and cross validation and it seems that the majority call it malpractice to do PCA before cross validation. I would also like to perform SMOTE, but there is a split between those who perform SMOTE before or after PCA. Yes, i am very confused, but even authors of published papers seem to be confused as well about this (Me being a newbie). Even here (on CrossValidate) there are different answers for the same Q about SMOTE and PCA, and the gist is, see what suits your data. Anyway, I know how to perform each by it's own, but don't know how to do PCA in each fold, for example in Kfold, or any other cross validation method.

1- Is it possible for someone to share with me how to do PCA inside Kfold? Through a loop or pipeline maybe?

2- I personally have a bipolar opinion on the SMOTE/PCA thing: a- I think that PCA should be performed before SMOTE so that the dimentionality reduction is done on the main (real) training data before we sample it (In my case upsampling). b- My other thought on it, is that if you upsample and then perform SMOTE then your dimentionality reduction is based on "non-biased" group sample (where both groups now have same number). My Q here is, if i CV (cross validated) PCA, can i include SMOTE in the mix?

3- Last Q, I saw a video of a prof. who said that if you test/split your data and then CV PCA on the training set, this is wrong. Since cross validate e.g. Kfold already does the train/test split for you. Is this the case? If so, then how would i eventually select my X_test X_train y_test y_train for further analysis?


texts:
I've an ECG data spread over time. The duration for each data is around 3 minutes (approx 180 seconds).

Each second around 200 recordings were taken. So total length for each sample is approx 36000.

Now how to use this data for lstm. My purpose is to make an encoder-decoder network to encode this data into small size. I'm planning to use reconstruction error for loss function.


texts: How to compute the R2-score for multivariate data?

texts: I have a model that for each instance predicts some values and I have the real measurements for each instance, but the number of data points varies among instances, say, for the first one I have 4 data points, so I predict 4 values, for the next one 5, etc. How do I evaluate the R2-score of my model. Do I just compute the R2-score for each instance and then take the average? Or is there a better way to do it?


texts: What to input into machine learning algorithm for image recogniton?

texts: I am working on a project that involves classifying images as either that of a cat or that of a dog, without using CNNs. I used SKImage to convert the images to a matrices and changed it to grayscale to reduce dimensions and complexity. I then flattened the matrices to vectors and input them into an SVC, Logistic Regression and Stochastic GD Classifier, but all of them were very innacurate and had less than 60% accuracy. This leads me to believe that I converted the images to the wrong format, which is why the ML algos are unable to fit the data. So what should I convert them to? Should I use feature extraction?


texts: Solving problem with variational dimension with deep learning

texts:
I have some data as below

sample_id  start end class
   1        3     7   1
   1        10    12  1
   1        14    18  1
   2        5     7   2
   2        9     11  2
   2        14    20  2
   2        21    26  2
   3        4     6   1
   3        8     11  1
   .....

It could be viewed as below

reference    0.............................................................30
    sample 1:  3------7     10-----12   14--------18
    sample 2:     5---7   9-----11      14----------------20  21---------26
    sample 3:   4---6   8-------11   13---------------19  20----22  24-------28
    sample 4:   4------------------12       15--17

In short, it could be described like this. Suppose, there is a 1000m long straight line. I randomly select a number n between 20-100. Then randomly select n segments from the reference line and create a sample. Similarly, for second sample, select a random number between 20-100 and take that many random segments from the reference line. Then randomly select 50% of the samples as class 1 and rest of the samples as class 2.

So, there are two types of samples, class 1 and class 2. Each sample consists of n number of segments. In above example, sample 1 consists of 3 intervals and second sample consists of 4 intervals. Number of intervals vary from sample to sample.

Now I want to use a deep learning algorithm to classify sample type as either class 1 or 2. Which deep learning algorithm should I use? Or what would be the solution for the problem using deep learning technique.


texts: How to pass more than 2 input columns to a Deep learning Keras model for sequence tagging/labeling

texts:
I have to build a neural network which extract relationship between two entities.Input should be:
Input text/paragraph, vocabulary of entities and relationship phrases that system should recognize.

Output is sequence of tags and length of output sequence and input text/paragraph is same.

Dataset is a CSV file having 3 input columns(input text, entities in text, relationship between 2 entities) and 1 output column. I am using Keras library to build this model.

Example-input1: zomato acquires uber; input2: zomato, uber; input3: acquires
; Output: some-tag some-tag some-tag (note: these are not actual tags just an example)

I planned to use a char embedding for input text using time distributed layer with Bi-LSTM but now got stuck with 3 inputs.
I am aware of keras functional api but how can I use it in a sequence tagging problem with time distributed layer.
If any other approach can be used to avoid this problem, please suggest.


texts: Having issues in converting dataframe to Bipartite graph

texts:
I have 2 columns which are in a dataframe and need to be converted to a bipartite graph: the entries of the columns should be vertices, with an edge for each row, joining the two vertices in that row. I am having issues in doing that.

My data looks like this:

Col1        Col2
9d7051e2    da48d749
611cebdb    93ef5eb4
758f1c7b    6acae826
d09360ac    a33fe922

I tried two methods to convert this dataframe to bipartite graph:

Method #1

G = nx.from_pandas_edgelist(df_train, source='Donor ID', target='Project ID',
                            edge_attr=True, create_using=None)

Method#2

G = nx.Graph()
G.add_nodes_from(df_train['Donor ID'], bipartite=0)
G.add_nodes_from(df_train['Project ID'], bipartite=1)
G.add_edges_from(
     [(row['Donor ID'], row['Project ID']) for idx, row in df_train.iterrows()])

In either case, bipartite.is_bipartite(G) returns FALSE.

Not sure what I am missing here. Any help on the same would be appreciated.

Few things which I tried to resolve the issue:

Find duplicate records (col1+col2) and removed duplicate ones
Make sure there are no missing values in both the table so that it confirms that there is an edge for every record.
When I tried with a subset of records out of 1Million records, it worked.
So as I understand its because of some faulty data but how to find all faulty records and how to fix it is challenging.
The data type is Object. Tried to convert it to string but not able to do so.
Any help will be appreciated.

Nothing above helped.

Please guide me what I am missing here or how can I further debug the issue.


texts: Syntax error in pandas dataframe slice in my sklearn data prep?

texts:
I'm new to sklearn and having trouble formatting the data to predict and evaluate a confusion matrix.  I'm using this Random Forest tutorial.

Here is my code

 from sklearn.ensemble import RandomForestClassifier
 import numpy as np
 import pandas as pd
 dataframe = pd.read_csv('output.txt', sep='\t')
 df = pd.DataFrame(dataframe)
 df['is_train'] = np.random.uniform(0, 1, len(df)) <= .75

 train, test = df[df['is_train']==True], df[df['is_train']==False]
 features = df.columns[1:5]
 clf = RandomForestClassifier(n_jobs=2)
 y, _ = pd.factorize(train['event_count'])
 clf.fit(train[features], y)

It's this line for my predictions which is giving me the error:

 preds = df[:6][clf.predict(test[features])]
 IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices


texts: Predicting a variably-placed value in a vector

texts:
I have $m$ vectors in $\mathbb{R}^n$, where $m >> n$, and I want to train a model to impute a value $x_i$ in $\mathbf{x}$, where $1 \leq i \leq n$ (and can vary by vector).
For instance, I may have the vector $\mathbf{x} = [12.34, 12341, 234, 21.5643, \cdot, 42.2]$, where I want to predict the second-to-last value.
I am guessing in theory this is similar to the masked word problem and approaches similar to BERT may work (though modified for a continuous domain), but I am wondering if their is an existing model architecture that does this?


texts:
My code:
from sklearn.impute import SimpleImputer

fill_values=SimpleImputer(missing_values=0,strategy='mean')

X_train=fill_values.fit(X_train)
X_test=fill_values.fit(X_test)

imp_X_train=fill_values.transform(X_train)
imp_X_test=fill_values.trainsform(X_test)

Gives the error:
TypeError: float() argument must be a string or a number, not 'SimpleImputer'

Where did I go wrong?


texts: PCA, why variance of eigen values is measure of its utility?

texts:
Source - Murphy, 12.3

Heuristic for assessing applicability of PCA.

Let the empirical covariance matrix Σ have eigenvalues λ1≥λ2≥···≥λd>0, with mean λ. Explain why the variance of the eigen values, σ2=(1/d)∑(λi−λ)2 is a good measure of whether or not PCA would be useful for analysing the data (the higher the value of σ2 the more useful PCA)


texts:
What are some podcasts which are related to data science?

This is a similar question to the reference request question on CrossValidated.

Details/rules:

The podcasts (the theme and the episodes) should be related to data science. (For example: A podcast which is about some other domain, with an episode which speaks about data science in that domain, is not a good reference/answer.)
Personal opinions/reviews (if any) would be very helpful too.


texts: Repeated k-fold Cross Validation for time series data?

texts:
I have a relative small sample size (330 with 45 features) + it's time series data.
I want to train my LightGBM regression model for best generalized RMSE score and want to use repeated CV. I use hyperopt to do hyperparameter optimization to optimize for lowest RMSE.
The last 2 months include data post-corona, so a standard k-fold CV would probably fail when testing on such data because there was quite a shift in the target variable y.
Standard ways to do repeated CV use resample / reshuffle which is not useable with time series data.
What is best practice in this case? How can I do repeated Cross Validation while working with time series data?
At the moment I do this for the fmin() function.
def lightgbm_cv_repeated(params):
    params = {
        'n_estimators': int(params['n_estimators']),
        'max_depth': int(params['max_depth']),
        'learning_rate': params['learning_rate'],
        'min_child_samples': int(params['min_child_samples']),
        'min_child_weight': params['min_child_weight'],
        'feature_fraction': params['feature_fraction'],
        'bagging_fraction': params['bagging_fraction'],
        'bagging_freq': int(params['bagging_freq']),
        'num_leaves': int(params['num_leaves']),
        'max_bin': int(params['max_bin']),
        'num_iterations': int(params['num_iterations']),
        'objective': 'rmse',
        }

    scores = []

    for i in range(5, 11):
        print(i)
        cvTSS = TimeSeriesSplit(max_train_size=None, n_splits=i)

        model = lgb.LGBMRegressor(random_state=i, **params)
        score = -cross_val_score(model, X=X, y=y, cv=cvTSS, scoring="neg_root_mean_squared_error", n_jobs=-1).mean()
        scores.append(score)

    result = mean(scores)
    return result


texts: Why does dice coefficient equally weigh false positive and false negative?

texts:
According to this article:
Dice coefficient=

The 2-class DSC variant for class c is expressed in
Equation 1, where gic ∈ {0, 1} and pic ∈ [0, 1] represent the
ground truth label and the predicted label, respectively. The
total number of pixels in an image is denoted by N. The epsilon
provides numerical stability to prevent division by zero.

And the author said:

With highly imbalanced data
and small ROIs such as skin lesions, FN detections need to be
weighted higher than FPs to improve recall rate.

I can't understand why does dice score weigh FP and FN. Could you help me to understand this?


texts: How to develop small workable embeddings for debugging

texts:
Processing language data with deep learning models often involves a lookup of a pre-trained embedding model. In the model development phase, it's very annoying that every time the entire embeddings are loaded, as embeddings could be very big (e.g., Glove) and would consume a lot of timing loading it.

Is there a way to build/find a smaller and workable embedding just for debugging?

Right now what I've done is shrink the dimensions of Glove, for example, reduce the dimension of embeddings from 300d to 30d by taking the first 30 dims, but that might potentially induce a risk that some of words will share the same embedding.


texts: Should boolean features be normalized and should false be -1 or 0

texts:
I am attempting to train an SVM from a set of features which are both numeric and categorical, for example:

Distance X (Numeric)
Distance Y (Numeric)
Font Size Difference (Numeric)
Word 1 Bold (Boolean)
Word 2 Bold (Boolean)
Word 1 Font Size (Numeric)
...

For mapping the features to a feature array I am treating true as 1 and false as 0 and then normalizing the entire feature array using Z-Scores.

Should I instead encode false as -1 so it has a numerical impact on the generation of the support vectors, with a value of 0 it does not modify the chosen vector (I assume)?

Should boolean features be normalized in the same way as numerics or should they be left with their encoded values?


texts:
I read about the Rosenblatt Perceptron Learning Algorithm. Often there is an explicit note:

  It is important to note that all weights in the weight vector are being updated simultaneously

But why are all weights updated simultaneously? I tried another approach where I iterated over all weights and updated them in different iterations. It also worked on some simple test cases.

Could someone explain to me, why they are updated simultaneously and why should this approach be better?


texts:
I am working on an academic project where I go through the bug database and classify the bug (a Multiclass Classification). After this classification, I would like to assign a weight to assign a weight based on the occurrence.
As an example, let us assume that we have a class called "Cli-related" which produces around 40% of the bug: How to assign a weight for this?
Simple way is # of bugs in this class / total No. of bugs. is this the way to do or does ML has different ways of assigning it.
I see that there is a class_weight parameter but was not aware of this usage.
Can someone help here please?


texts: How to best forecast simple binary data?

texts: I have a set of timeseries binary (boolean) data, with intervals of 1 day. Each day can either be 1 or 0 (true/false). What is the best way to forecast the next day/week's data based on the data I already have?


texts: MCMC for finding Bayesian Neural Network

texts:
Is someone familiar with such an approach:

Suppose I want to build a bayesian neural network, with distributions over my parameters instead of point estimates. First I train my network with standard backprop. After training I start some MCMC sampling over my network weights in order to estimate the distributions.

Somehow this approach seems very natural to me and probably there are already some papers or drawbacks, which I do not see. I am aware that sampling can be a slow process, but with a good starting point it should run quite fast. Also sampling could be run in batchmode in order to speed up.


texts: BERT - How Question answering is different than classification

texts:
Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is:

class BertForSequenceClassification(PreTrainedBertModel):
    def __init__(self, config, num_labels=2):
        super(BertForSequenceClassification, self).__init__(config)
        self.num_labels = num_labels
        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, num_labels)
        self.apply(self.init_bert_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
        _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            return loss
        else:
            return logits

In Above code pooled_output is considered useful in line  _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)

And in below QnA code encoder layer output (i.e., sequence_output) is considered useful in line: sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)

class BertForQuestionAnswering(PreTrainedBertModel):
    def __init__(self, config):
        super(BertForQuestionAnswering, self).__init__(config)
        self.bert = BertModel(config)
        # TODO check with Google if it's normal there is no dropout on the token classifier of SQuAD in the TF version
        # self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.qa_outputs = nn.Linear(config.hidden_size, 2)
        self.apply(self.init_bert_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, start_positions=None, end_positions=None):
        sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        logits = self.qa_outputs(sequence_output)
        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

Now my questions are:

Why there are 2 logits being returned in sequence_output for Question Answering case
What is different in encoder layer and pooled layer
Why is encoder layer (sequence_output) is considered in QnA case and Pooled layer in classification case


texts: How to upload a saved file from google colab to a site such as kaggle or github?

texts:
We are able to load a file from a site such as kaggle and git hub to google collab, we apply the below code to download from other sites into google colab:

!wget nlp.stanford.edu/data/glove.6B.zip

but I am very curious to know if I am able to upload a saved file (such as an output from google colab) to kaggle or github with a command like above?


texts: Anomaly detection in Time Series Data - Help Required

texts:
I am looking for algorithms on Anomaly detection for time series data. It is uni-variate analysis, considering single parameter (inlet pressure) of air compressor sensor data. The objective is to detect if there is any significant deviation in parameter value in real time.

If any one has worked on similar projects, please share your thoughts.


texts: Find most important features which differentiate two sets

texts: Assume that we have two small sets of feature vectors (each feature vector representing an item). What is a good way of finding which features have the biggest difference (in distribution) between the two (small, 50 samples) sets? Given that the distribution of the data per feature is not necessarily normal.


texts: Understanding Embeddings input and output sizes

texts:
I have been trying for a while to understand the dimensionality of embeddings in neural networks and I think that finally things have clicked on my brain, however I would love to check whether or not my understanding is correct.

Embeddings are an effective way to transform words into vectors, or at least to reduce the dimensionality of the data (essentially the Bag of Words approach does not work well as data is sparse)
If I have a text corpus that contains say, 5000 sentences, I could then pad each sentence to a standard size, for example 150 and then use embeddings (possibly the Glove pretrained ones) to get an output with dimensionality of 100, that means that I would have 5000x150x100 elements.

Is my understanding correct? If so this means that I can start training my network using mini-batches of say 16x150x100 elements, the layer after the embedding one could be then a LSTM and so on...


texts: Keyword/phrase extraction from Text using Deep Learning libraries

texts:
Perhaps this is too broad, but I am looking for references on how to use deep learning in a text summarization task.

I have already implemented text summarization using standard word-frequency approaches and sentence-ranking, but I'd like to explore the possibility of using deep learning techniques for this task. I have also gone through some implementations given on wildml.com using Convolutional Neural Networks (CNN) for sentiment analysis; I'd like to know how one could use libraries such as TensorFlow or Theano for text summarization and keyword extraction. Its been about a week since I started experimenting with Neural nets, and I am really excited to see how the performance of these libraries compares to my previous approaches to this problem.

I am particularly looking for some interesting papers and github projects related to text summarization using these frameworks. Can anyone provide me with some references?


texts: Logistic Regression optimal threshold is a negative value

texts:
I run the code below:
import pandas as pd
import numpy as np
from sklearn.model_selection  import train_test_split
from sklearn import linear_model
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from numpy import sqrt
from numpy import argmax
from sklearn.metrics import roc_curve
from sklearn.preprocessing import StandardScaler

def standardize(variable):
      return (variable - np.mean(variable)) / np.std(variable)

def normalize(x):
    return (x-x.min()/(x.max()- x.min()))

data.columns = np.arange(len(data.columns))

trainX, testX, trainy, testy=train_test_split(X,y,test_size=0.5,random_state=2, stratify=y)

# fit a model
model = LogisticRegression(solver='lbfgs')
model.fit(trainX, trainy)

#yhat = model.predict_proba(testX)
yhat = normalize(testX.values)
yhat = yhat[:, 0]
print(yhat)

# calculate roc curves
fpr, tpr, thresholds = roc_curve(testy, yhat)

#print(thresholds)
# calculate the g-mean for each threshold
gmeans = sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = argmax(gmeans)
print('Best Threshold=%f, G-mean=%.3f' % (thresholds[ix], gmeans[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.', label='Logistic')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

The optimal threshold score is: Best Threshold= -0.049752, G-mean=0.889
Why is the optimal threshold a negative number? What does it mean? And why am I getting a negative number?


texts: build a classification model under constraint

texts:
Suppose I have n features $(x_1, x_2, ...., x_n)$ (all float) and want to classify $y$ (0 or 1)

For now I have a legacy expert system to do the classification.

Expert system rules:

Categorize all $x_i$

If $x_1 \lt 1$, set $x_1' = a_{11} $,

If $x_1 \in [1,10]$, set $x_1' = a_{12} $,

If $x_1 \gt 10$, set $x_1' = a_{13} $,

...

If $x_n \gt 100$, set $x_n' = a_{nk} $,

If $w_1 x_1'+...+w_n x_n' \geq z$, then predict $y = 1$, otherwise $0$

$a_{11}, a_{12}, .... a_{nk}, w_1, ..., w_n, z$ are determined by intuition instead of calculation, and I wanted to improve the expert system. How to do it? logistic regression cannot be simply applied on this case...


texts: What is representation in optical character recognition?

texts:
I am learning OCR and reading this book

The authors define 8 processes to implement OCR that follow one by one (2 after 1, 3 after 2 etc):

Optical scanning
Location segmentation
Pre-processing
Segmentation
Representation
Feature extraction
Recognition
Post-processing

This is what they write about representation (#5)

  The fifth OCR component is representation. The image representation
  plays one of the most important roles in any recognition system. In
  the simplest case, gray level or binary images are fed to a
  recognizer. However, in most of the recognition systems in order to
  avoid extra complexity and to increase the accuracy of the algorithms,
  a more compact and characteristic representation is required. For this
  purpose, a set of features is extracted for each class that helps
  distinguish it from other classes while remaining invariant to
  characteristic differences within the class.The character image
  representation methods are generally categorized into three major
  groups: (a) global transformation and series expansion (b) statistical
  representation and (c) geometrical and topological representation.

This is what they write about feature extraction (#6)

  The sixth OCR component is feature extraction. The objective of
  feature extraction is to capture essential characteristics of symbols.
  Feature extraction is accepted as one of the most difficult problems
  of pattern recognition. The most straight forward way of describing
  character is by actual raster image. Another approach is to extract
  certain features that characterize symbols but leaves the unimportant
  attributes. The techniques for extraction of such features are divided
  into three groups’ viz. (a) distribution of points (b) transformations
  and series expansions and (c) structural analysis.

I am totally confused. I don't understand what is representation. As I understand after segmentation we must take from image some features, for example topological structure like Freeman chain code and must match to some saved at the learning stage model - i.e. to do recognition. By other words - segmentation - feature extraction - recognition. I don't understand what must be done on representation stage. Please, explain.


texts: Response variable at the group level, independent variables at the entity level

texts:
I have a dataset of entities which each belong to a particular group (i.e. entity=schools and group=school district). I also have lots of auxiliary variables on each entity.
However, for my response variable I only have information at the group level (i.e. response variable only at the district level, but regressor variables at the school level).
Can someone recommend an algorithm or class of algorithms that might be appropriate here? I'd rather not aggregate all the auxiliary information up to the group level and throw away all info contained in the correlations among the auxiliary variables.


texts: Need help with LDA for selecting features

texts:
I am currently selecting features of products by using LDA to group 6000 keywords of product into topics.
Here is the sample of my dataset after being organized into list of keywords for each product id.

I consider each id as a "document" and each keywords as the "word" in a "document" for the case of LDA.
It didn't work out as I expected as each topic have many identical keywords with different weight. I removed 100 most common keywords but there are still some identical keywords in the topics. Here is the sample output:

How can I deal with the identical keywords in my topics? and also how to deal with the 100 most common keywords I removed?

Here is the graph showing the frequency of the words in all "documents"

Each word only presents once in each document, but may present in different documents. I updated a new graph of the frequency of the words

Thank you so much. Any suggestion is appreciated.


texts: Categorical encoded variables in scikit-learn diabetes dataset

texts: When using sklearn.datasets import load_diabetes, the sex variable which is categorical, is scaled to continuous values. Is it even legal to scale such variables?


texts: Training algorithm for finding relationships within a data set

texts:
Given a dataset that looks like the following:

-----------------------------------------------------------
|Category|Phrase         |Related 1       |Related 2  |...|
-----------------------------------------------------------
|Art     |Art exhibitions|                |           |   |
|Art     |Art galleries  |Art exhibitions |Art museum |   |
|Art     |Art museum     |Art galleries   |           |   |
|Sports  |Football       |Sports coaching |NFL        |   |
|Sports  |Basketball     |NBA             |           |   |
|Sports  |Sports coaching|                |           |   |
-----------------------------------------------------------

If I have an input phrase, how can I list possible related phrases inside the dataset? For example, I would relate "art exhibitions" to "art museum" and "art galleries". The dataset has some relationships filled out already (I am not sure if this counts as usable training data or if I have to transform it further) and I am looking to fill in the blanks, that is find relationships for some rows.

Eventually we would want to be able to support input phrases that don't exist yet in this dataset and establish relationships for them based on historical info, which is why I think I should be looking at machine learning.

What is the best algorithm/approach I can make use of for this problem?


texts: Should I drop Nan before before creating a scatter plot and computing the correlation matrix?

texts:
I have some NaN values in my data and I cannot replace it with median or something else.
Now I have to check out the correlation, using scatter plot and Pearson correlation.
Should I drop these NaNs or there is no difference? Actually, I see the difference, but a very, very small difference. What is the best solution here?


texts: Keras ImageDataGenerator.flow_from_directory doesn't find images

texts:
I'm working on a deep learning (CNN) problem. I have structured my images into folders correctly (I think), like this:

train_dir:
   - class 1
   - class 2
validation_dir:
   - class 1
   - class 2
test_dir:
   - class 1
   - class 2

The images are PNG in a (196,256,3) format but will I'd like to convert them to (196,256,1) because they are grayscale. I do this with the flow_from_directory argument color_mode='grayscale'

As such the code I have so far to get on with this is:

train_datagen = ImageDataGenerator(rescale=1./255) # re-scale pixel values between [0,1]
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size = (196,256),
        color_mode='grayscale',
        batch_size=20,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size = (196,256),
        color_mode='grayscale',
        batch_size=20,
        class_mode='binary')

With train_dir and validation_dir being strings with the equivalent path of the correct folders I mentioned earlier above.

However I get this error when running this section of the code:

Found 0 images belonging to 2 classes

After trying for sometime I can't figure out what's wrong. Does anyone have a clue?


texts: why does my calibration curve for platts and isotonic have less points than my uncalibrated model?

texts:
i train a model using grid search then i use the best parameters from this to define my chosen model.
model = XGBClassifier()
pipeline = make_pipeline(model)

kfolds = StratifiedKFold(3)
clf = GridSearchCV(pipeline, parameters, cv=kfolds.split(x_train, y_train),
                           scoring='roc_auc', return_train_score=True)

clf.fit(x, y)

model = clf.best_estimator_

using this model from gridsearch i then calibrate it and plot uncalibrated vs calibrated..
y_test_uncalibrated = model.predict_proba(x_test)[:, 1]
fraction_of_positives, mean_predicted_value=calibration_curve(y_test,y_test_uncalibrated,n_bins=10)

plt.plot(mean_predicted_value, fraction_of_positives, 's-', label='Uncalibrated')

clf_isotonic = CalibratedClassifierCV(model, cv='prefit', method='isotonic')
clf_isotonic.fit(x_train, y_train)
y_test_iso = clf_isotonic.predict_proba(x_test)[:, 1]
fraction_of_positives, mean_predicted_value = calibration_curve(y_test, y_test_iso, n_bins=10)

plt.plot(mean_predicted_value, fraction_of_positives, 's-', color='red', label='Calibrated (Isotonic)')

i do the same for platts above.. however i get the following results:

i don't understand why there are less points now for isotonic and platts? i don't feel as though i am doing anything wrong in my code. am i making any mistakes?


texts:
I created an RNN model for text classification with the LSTM layer, but when I put the batch_size in the fit method, my model trained on the whole batch instead of just the mini-batch _size.
This also happened when I used GRU and Bidirectional layer instead of LSTM.
What could be wrong?
def create_rnn_lstm():
    input_layer = layers.Input((70, ))
    embedding_layer = layers.Embedding(len(word_index) + 1, 300, weights=[embedding_matrix], trainable=False)(input_layer)
    embedding_layer = layers.SpatialDropout1D(0.3)(embedding_layer)
    lstm_layer = layers.LSTM(100)(embedding_layer)
    output_layer1 = layers.Dense(70, activation="relu")(lstm_layer)
    output_layer1 = layers.Dropout(0.25)(output_layer1)
    output_layer2 = layers.Dense(2, activation="softmax")(output_layer1)
    model = models.Model(inputs=input_layer, outputs=output_layer2)
    model.compile(optimizer=optimizers.Adam(), loss='binary_crossentropy')
    return model

LSTM_classifier = create_rnn_lstm()
LSTM_classifier.fit(X_train_seq, y_train, batch_size=128, epochs = 10, shuffle=True)


texts: SVM on BERT-Embeddings with very small dataset does not converge

texts:
I am trying to reproduce the results from this paper where they use a linear SVM on top of BERT-Embeddings for text-classification. They use the average of the token-embeddings which results in a 768 dimensional vector for each sample.
As this is active-learning, the number of samples used is very small (between 40 and 500).
So: nb_samples < nb_features.
I thought that a SVM converges most of the times (see this question), and surely in the cases where nb_samples < nb_features. However, this is not the case, even with only 40 samples, the SVM (LinearSVC from sklearn) does not converge.
I am quite sure my embeddings are somehow meaningful, they look also ok (values between between -1 and 1) no need for normalization etc.
What could be my problem?
I tried dual=True and dual=False, increasing max_iter to 10000 etc.


texts: performances evaluation of image classification with different distribution for train and test set

texts:
I am implementing a CNN to do image classification of 4 classes representing different weathers : Haze, Sunny, Rainy, Snowy. I have as training set 3200 images, and as test set 3038 images.

The problem is that the test and training set are different set of pictures(so to be clear, I didn't split the dataset, but I have been given a training set and a test set in different folders) and they have so a different distribution one with respect to the other.

After training and evaluating, I have that my accuracy on the test set is 0.475313 and the loss always on the test set is 3.060471.

Also if I plot the hihstory of accuracy and of the loss I have the following:

what I don't understand is if I have the accuracy so low because I am overfitting, or because the distribution is different. And also, how do I interpret the plots I have posted about the histories of loss and accuracy? Is it normal that the gap between test and train accuracy is so large?

Thanks in advance.

[EDIT] By searching online, I have found that to cope with this kind of problems, it is possible to use domain adaptation, but honestly I have no idea how to implement it because I have just read about it.


texts: Solution for in Time/Space Complexity challenge in Recommendation System?

texts:
I have a book Recommendation System project and have a huge data set of feature vectors. What is the best solution for in memory computation? I mean, the program should:

calculate the cosine Similarity among Database Books
be REAL TIME, because it's an interactive business site.

I've already thought about big data analysis tools and management such as Spark, Hadoop, ... . However, I am really new to Spark Techs and therefore not sure whether it's practical or not. I am really mixed up how can it be helpful? I have studied spark's documentation in java but it confused me more about how spark can be helpful?


texts: I'm familiar with using trigonometric functions to transform cyclic variables for use as features in training a model (most commonly hour of the day or month of the year); I'm now trying to figure out the best way of doing this for using these types of variables as the target for my model. (Imagine a model predicting the month when a particular event is most likely to occur, for instance). Neither using a strictly-increasing representation (so that January, 2018 is close to December, 2017 but very far from January, 2017) nor treating the month as a categorical variable is ideal, but the trigonometric encoding done for features requires both sin and cos parts to have a unique representation of each month, so using one of the two isn't a workable approach either.  Is there a better option that I'm missing?


texts: What do I need to learn in order to really understand Machine Learning?

texts:
I've graduated with a math major in undergrad, but mostly focused on algebra (Galois Theory, Knot theory, etc.). I work in something unrelated right now, but now I want to study machine learning. The question is, what kind of knowledge should I have if I want to really understand machine learning?

Say, here are some things I can think of, but obviously I'm missing a lot, I'm assuming.

"Fundamentals" (Calculus, Linear Algebra, Discrete Mathematics, Coding, etc.)
Probability (but what specific areas?)
Statistics (but what kind?)
Algorithms
Differential Equations

But what else? Or what subfields of what I've mentioned above are particularly important (i.e. Bayesian statistics)?

Edit: I am currently considering a graduate program in ML, and wanted to know if this is something I really want to do / know more about it / prepare myself.


texts: Gaussian process regression: sudden increase of the prediction's variance

texts:
I am trying to do a variance-based uncertainty sampling like in the book "Active Learning" by Burr Settles, page 18, Figure 2.6. Link to the book: http://active-learning.net/

I use Gaussian process regressor from scikit-learn to fit a 1-D Gaussian function. The parameters of the regressor are the same as in this example (a noise-free case) http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html

kernel = C(1.0, (1e-3, 1e3)) * RBF(10, (1e-2, 1e2))
gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=9)

Initially, I randomly choose two training data points and fit the curve based on them. The Figure below shows the fit (top) and the variance (bottom)

Then I add one more training data point taken at the place where the variance is the largest, and re-fit the data.

I repeat this process for a few iterations and the interpolation improves with each iteration, as expected. For example, this is the iteration 7:

However, at the next iteration, the interpolation is worse. The variance suddenly increases:

The next few iterations are fine again. For example, iteration 9:

But at the iteration 11 I get a large variance again:

Iterations 12 through 15 show very small variance again - the curve is interpolated very well at those iterations. I stopped at 15.

Why does this sudden increase of variance happen at certain steps? How to avoid it?

When I change the parameter n_restarts_optimizer, this behavior is still observed but at different iteration steps. For example, when n_restarts_optimizer=8, the increase in variance is at iteration 9. When n_restarts_optimizer=7, the increase in variance is at iteration 4, 10, 11. When n_restarts_optimizer=4, the increase in variance is at iteration 7, 14.

When I change the parameter length_scale in the RBF kernel from 10 to 1.0 I don't see this increase in variance anymore. However, when I make it even smaller (length_scale = 0.1) it appears again.


texts: Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

texts:
I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.

It seems to me that the number of LSTMs in the encoder would have to be the same as number of words in the text (if word embeddings are being used) or characters in the text (if char embeddings are being used). For char embeddings, each embedding would correspond to 1 LSTM in 1 direction and 1 encoder hidden state.

Is this understanding correct?

E.g. If we have another model that uses encoder-decoder for a different application (say text-to-speech synthesis described here https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html) tha uses 256 LSTMs in each direction of the bidirectional-encoder, does it mean the input to this encoder is limited to 256 characters of text?

Can the decoder output has to be same length as the encoder input or can it be different? If different what factor describes what the decoder output length should be?


texts: Creating an "unclassified" class in Random Forest

texts:
I am trying to classify satellite based images by creating a region of interest and then classifying according to it.
I am using a Jupyter notebook using python to do that.
I used a Random forest classifier and got a nice model and result, but the problem is that the image is "over classified" meaning that all the pixels et value and force to be classified.
I would like to define level of similarity that a pixel has to have in order to be classified, otherwise, it will not get any class.
For example, the black suppose to be asphalt:

However, in the RGB, you can see it's not asphalt:

Is there any way to define in random forest or any other algorithm "level os similarity"? (For example something similar to n-D angle to match pixels to reference like ised in SAM, but under random forest, or another algorithm that allows define that)
SAM- https://www.harrisgeospatial.com/docs/SpectralAngleMapper.html
My end goal: to get "unclassified" values based on similarity level to the calibration data


texts:
I'm a Java developer and I want to pursue career in Data Science and machine learning.

Please advise me where and how to begin? What subjects and mathematical/statistical skills are required and so on?


texts: I need to find the baseline for application metrics (eg CPU utilization,Response time).pls help

texts:
I have historical values for these metrics.Please suggest some approach to find this.The baseline has to be dynamically generated based on the selected duration.

Data ::

6/6/18 3:40 AM  26  TargetResponseTime  0.00
6/6/18 3:35 AM  26  TargetResponseTime  0.00
6/6/18 3:35 AM  26  TargetResponseTime  0.00
6/6/18 3:30 AM  26  TargetResponseTime  0.07
6/6/18 3:30 AM  26  TargetResponseTime  0.05
4/22/18 2:00 AM 26  TargetResponseTime  0.05
4/22/18 2:00 AM 26  TargetResponseTime  0.05
4/22/18 1:50 AM 26  TargetResponseTime  0.09
4/22/18 1:50 AM 26  TargetResponseTime  0.09
4/22/18 1:00 AM 26  TargetResponseTime  0.00
4/22/18 12:50 AM    26  TargetResponseTime  0.17
4/22/18 12:45 AM    26  TargetResponseTime  0.40
4/22/18 2:05 AM 26  TargetResponseTime  0.00
4/22/18 1:00 AM 26  TargetResponseTime  0.00
4/22/18 12:50 AM    26  TargetResponseTime  0.17
4/22/18 12:45 AM    26  TargetResponseTime  0.40
4/22/18 8:05 AM 26  TargetResponseTime  0.00
4/22/18 8:05 AM 26  TargetResponseTime  0.00
4/22/18 7:55 AM 26  TargetResponseTime  0.00
4/22/18 7:55 AM 26  TargetResponseTime  0.00
4/22/18 7:25 AM 26  TargetResponseTime  0.57
4/22/18 7:15 AM 26  TargetResponseTime  0.00
4/22/18 7:00 AM 26  TargetResponseTime  0.00
4/22/18 7:25 AM 26  TargetResponseTime  0.50
4/22/18 7:25 AM 26  TargetResponseTime  0.28
4/22/18 7:15 AM 26  TargetResponseTime  0.00
4/22/18 7:00 AM 26  TargetResponseTime  0.00
4/22/18 11:45 AM    26  TargetResponseTime  0.16
4/22/18 11:45 AM    26  TargetResponseTime  0.16
4/22/18 10:45 AM    26  TargetResponseTime  0.00
4/22/18 10:45 AM    26  TargetResponseTime  0.00
4/29/18 4:05 AM 26  TargetResponseTime  0.00
4/29/18 4:00 AM 26  TargetResponseTime  0.00


texts: What are some of the resources to learn practical issues in machine learning and data science?

texts: I feel like a lot of the resources on ML have been focused on algorithms/modeling whereas in practice (industry applications) what's arguably the harder part, IMO, is what's before and after modeling (collecting unbiased training data, error diagnosis, conduct appropriate A/B testing, explore/exploit, etc) What are some of the resources to learn these practical issues in ML/DS ? Is experimental design the right keyword?


texts: How to deal with disconnected components in isomap?

texts: While creating a nearest neighbor graph for isomap, there is a possibility that the graph is disconnected. In this case finding graph distances between all pairs of points will not be possible. Are there any simple methods other than iteratively changing the nearest neighbor search parameters  till we get a connected graph?


texts: Is there a library that would perform segmented linear regression in python?

texts: There is a package named segmented in R. Is there a similar package in python?


texts:
I have a methodology question: are hold-out and CV generalization-optimization techniques mutually exclusive? It gets really confusing to me at times, because in the most recent project I have been doing something as follows:

I have split the dataset into TRAIN and TEST sets (with stratified distribution)
I have applied gridsearchCV on the TRAIN set with cv = 10, so effectively the model was splitting TRAIN set into TRAIN' and VALIDATION sets at each fold
I have used the optimized trained model with the TEST set, and ended up with results 0.97 (TRAIN set) vs 0.68 (TEST set).

Normally, when I see results like that my immediate reaction would be to assume that I am overfitting to the TRAIN set, and that the model does not generalize well enough. However, I have already used CV with the TRAIN set to make sure I was tuning the model to generalize as well as possible. I do not have any immediate tools (nor I should touch the already trained model) to improve this score now, since as I understand Hold-out technique, it just gives me the final score of how my model performs on previously unseen data.

At the same time, it feels kind of strange, since with using both CV and Hold-out, I am guaranteed to get some outliers in the hold-out set (TEST in my text above), which will fail.

Any suggestions or ideas?


texts: What does it mean when the model Learning Curve displays this behavior?

texts:
I made a classification LSTM model and it seemed to be working as intended until I increased the number of epochs. That's when I noticed that the Validation and Training curves cross each other at a certain point:

I have never seen this kind of curve... also couldn't find anything like it on my searches.
Does it mean my model is over-fitting?
Should I worry with this behavior?
What can I do to avoid this?
Thanks.


texts: Difference between Ridge and Linear Regression

texts: From what I have understood, the Ridge Regression is just having the loss function for an optimization problem with the addition of the regularization term (L2 Norm in the case of Ridge). However I am not sure if the loss function can be described by a non-linear function or it needs to be linear. In this case, if the loss functions needs to be linear, then from what I understand the Ridge regression, is simply performing Linear regression with the addition of the L2-Norm for regularization. Please correct  me if I am wrong.


texts:
Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (vanilla gradient descent did). Why might this be?
Later steps where affected by momentum etc but I would assume these effects wouldn’t come into play for the first few steps.


texts: implementing temporal difference in chess

texts:
I have been developing a chess program which makes use of alpha-beta pruning algorithm and an evaluation function that evaluates positions using the following features namely material, kingsafety, mobility, pawn-structure and trapped pieces etc..... My evaluation function is derived from the

$$f(p) = w_1 \cdot \text{material} + w_2 \cdot \text{kingsafety} + w_3 \cdot \text{mobility} + w_4 \cdot \text{pawn-structure} + w_5 \cdot \text{trapped pieces}$$

where $w$ is the weight assigned to each feature. At this point i want to tune the weights of my evaluation function using temporal difference, where the agent plays against itself and in the process gather training data from its environment (which is a form of reinforcement learning). I have read some books and articles in order to have an insight on how to implement this in Java but they seem to be theoretical rather than practical. I need a detailed explanation and pseudo codes on how to automatically tune the weights of my evaluation function based on previous games.


texts: How did the author got this final result from this Gaussian Distribution formula?


texts: Making bigram features from a particular dataset

texts:
I have a folder which has a number of files which have a format like these

madvise
write
write
write
write
read
read
madvise
ioctl
ioctl
getuid
epoll_pwait
read
recvfrom
sendto
getuid
epoll_pwait

that is it is a set of words which repeat.This is how all the files are like.
Now I have created a feature vector table using unigram that is each word becomes a feature and each file becomes a row where I put the frequency of that word occuring in the respective columns.

Now I want to create a similar FVT using bigrams.I was wondering how to do that in this case.


texts: Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

texts:
My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels are not exclusive, a headline could be about crime and politics at the same time. I would like to compare three algorithms for this problem. I use the following pipeline to predict the labels:

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])

parameters = [
    {"clf": [RandomForestClassifier()],
     "clf__n_estimators": [10, 100, 250],
     "clf__max_depth":[8],
     "clf__random_state":[42]},
    {"clf": [LinearSVC()],
     "clf__C": [1.0, 10.0, 100.0, 1000.0],
     "clf__random_state":[42]}
    {"clf": [GaussianNB()]}
]

rkf = RepeatedKFold(
    n_splits=10,
    n_repeats=2,
    random_state=42
)

cv = GridSearchCV(
    pipeline,
    parameters,
    cv=rkf,
    scoring='accuracy',
    n_jobs=-1)

The pipeline works fine for the random forest, but breaks at the LinearSVC() with the following error:

ValueError: bad input shape (20972, 20)

If I remove the LinearSVC(), it stops at:

A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.

There is a list "Support multilabel:" in the sci-kit learn documentation on multilabel classification (https://scikit-learn.org/stable/modules/multiclass.html) and only the random forest is included. However, the documentation states that "Multioutput classification support can be added to any classifier with MultiOutputClassifier."

I am bit confused, do LinearSVC() and GaussianNB() support multilabel classification when wrapped in MultiOutputClassifier()? If not, is there a workaround?


texts:
Recently I read about path ranking algorithm in a paper (source: Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion).

In this paper was a table (Table 3) with facts and I tried to understand how they were calculated.

F1 (harmonic mean of precision and recall) = 0.04
P (precision) = 0.03
R (recall) = 0.33
W (weight given to this feature by logistic regression)

I found a formula for F1 via Google which is

$F1 = 2 * \frac{precision * recall}{precision + recall}$

The problem is that I get the result of 0.055 with this formula, but not the expected result of 0.04.
Can someone help me to get this part?
Also, does someone know how 'W' can be calculated?
Thanks.


texts: Machine Learning Based Algorithm for Image Inpainting?

texts:
I have set of images which they look like the below sample.

You can think of it as a person's 2-week schedule represented by different color. Inside the images. However, there are some small missing areas shown in black. I'm interested in inpaint/fill the missing areas using ML methods.

Note that I have already tried classical methods like some those mentioned in the paper: "Image Inpainting, Christine Guillemot and Olivier Le Meur", but the result was not satisfactory.

Since almost all images have the same pattern, I'd like to know if anyone can suggest the ML-based approaches tackle this problem.


texts: Advantages of CNN vs. LSTM for sequence data like text or log-files

texts:
When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what does it depend on? Are there any papers or statistics that confirm this?
I'm thinking of data like Linux log entries or short sentence of length of less than 20 words/tokens.
Personally i would almost always use LSTM but I'm curious if CNN wouldn't be better in some cases, if its possible to implement them in a meaningful way. On short sentence there isn't much buffer to use CNN if i'm not mistaken.


texts:
I have two questions :

Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated ) ?
When we normalize the training set we ought to normalize the target set too . Won't it affect the performance ? I mean won't the data set change completely because the model had different Ranges of features as compared to the ranges of features in target set .

I tried googling the questions but was not able to come to conclusion . Any help would be appreciated .

Thanks !


texts: Issue with custom loss function including parameters in keras autoencoder

texts:
I am writing an Autoencoder that tries to find parameters for 3D Meshes. For example, how bent an object is. I input the Mesh vertices but would like to include the true parameters versus the Autoencoders latent space in the loss function.
I already wrote this function after some searching around but I don't know how to index from the current batch to get the right parameters from the numpy array param_true.

So basically what I would like is to get the same batch from the param_true array as the Autoencoder is using at the moment so I can calculate the loss from the right elements.
At the same time, I do not want to use the parameters as inputs.
Thanks in advance!


texts: Why is the reported loss different from the mean squared error calculated on the train data?

texts:
Why the loss in this code is not equal to the mean squared error in the training data?
It should be equal because I set alpha =0 , therefore there is no regularization.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error

#
i = 1 #difficult index

X_train = np.arange(-2,2,0.1/i).reshape(-1,1)
y_train = 1+ np.sin(i*np.pi*X_train/4)

fig = plt.figure(figsize=(8,8))
ax = fig.add_axes([0,0,1,1])
ax.plot(X_train,y_train,'b*-')
ax.set_xlabel('X_train')
ax.set_ylabel('y_train')
ax.set_title('Function')
nn = MLPRegressor(
    hidden_layer_sizes=(1,),  activation='tanh', solver='sgd', alpha=0.000, batch_size='auto',
    learning_rate='constant', learning_rate_init=0.01, power_t=0.5, max_iter=1000, shuffle=True,
    random_state=0, tol=0.0001, verbose=True, warm_start=False, momentum=0.0, nesterovs_momentum=False,
    early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999, epsilon=1e-08)

nn = nn.fit(X_train, y_train)

predict_train=nn.predict(X_train)

print('MSE training : {:.3f}'.format(mean_squared_error(y_train, predict_train)))

When I ran this code I found loss = 0.02061828 and the MSE in the training (MSE training) = 0.041


texts: How to create a language translator from scratch?

texts: I want to create a translator which can translate English, Korean and Tamil sentences into English sentence, I tried googletrans but is there any way to create something better than that using DL and NLP techniques?


texts: Is it possible to do Normalization before Xgboost?

texts:
Currently I am working on a project which uses Xgboost Regression.
Before putting data into model, I implemented Normalization, the accuracy significantly increased compared with without Normalization.
But I saw someone said that Xgboost doesn’t need Normalization .
Did someone meet the same issue as I met ?
Thanks in advance.


texts: Why do I get such a high MSE when I choose a multivariate target?

texts:
I have this dataset:

Here's how I currently create my data and univariate target:

X = data_copy.drop(columns=["Date"]).values
y = data["Date"].values
X_train, X_test, y_train, y_test = train_test_split(X, y, shuffle=True)

With this model:

lin_reg = Pipeline([("MinMaxScaler", MinMaxScaler()),
                   ("LinearRegression", LinearRegression())])

I get an MSE of 4.376690269172973e+34.

However, if I do this:

y = data_copy.drop(columns=["Date"]).values
X = data["Date"].values.reshape(-1, 1)

I get such a high MSE that is close to six digits! Why? Ideally, I want X to be my date, and y to be my multivariate target, not the other way around. What's the issue here?


texts: In first visit monte carlo are we assuming the environment is the same over episodes?

texts:
Watching this video (11:30) that presents the simplest algorithm for reinforcement learning: Monte Carlo Policy Evaluation, which says in general:

The first time a sate is visited:

increment N(s): N(s) = N(s) + 1
increment total state's return function by current episode's return so far S(s) = S(s) + G_t

State's value is estimated by mean return over many episodes:

V(s) = S(s) / N(s)

by law of large numbers, V(s)-->V_true(S) as N(S)-->inf

My question is - should the environment always behave the same over different episodes, or can it change randomly, and we sill have the true value over a large enough number of episodes?


texts: How to add date column in python pandas dataframe


texts: Is there a paper accomplishing finding physical law from observation without premade perception, using machine learning?

texts:
For example:
Isaac Newton finds law of universal gravitation just by looking a falling apple, without any premade perception of that phenomenon. Is it possible to accomplish that kind of discovery using machine learning?


texts: is it beneficial to use high-order n-grams as feature vectors for web anomaly detection?

texts:
i am studying about the use of n-gram models to classify web attacks based on several parameters like, requested resources, query parameters and attributes, characters distribution and so on. Most research papers use n-grams frequencies as feature vectors for their machine learning algorithms.

Ascii code only allows the use of 256 characters, so if i want to use 1-gram, my feature vector will have 256 dimensions, and for  n-grams it will $ 256^n\,$ dimensions.

A feature vector with such a long dimension leads to the so called "curse of dimensionality".

The use of bi-grams frecuencies as feature vectors results in good accuracy by classification algorithms.

If the use of bi-grams gives greats results, the use of tri-grams give better results against bi-grams.

So, my question is as follows :
If we increase n, we should have better results compared to lower values of n when classifying queries?
Can i conclude that for example the use of 7-grams would produce greater accuracy when compared to 4-grams?

Thanks


texts: What is the differences in the Gini Index, Chi-Square, and Information Gain splitting methods?

texts: I am looking through decision trees, and I do not understand what makes each of these methods different. Could someone explain clearly what the difference between these is? Thank you.


texts: Why it is taking more training time on one machine comparatively to similar another?

texts:
I have a HP15-R203TX notebook with CPU i5-5200U 2.20GHzx4 / RAM 4Gb /2 Gb NVIDIA GeForce 820M. My friend has the same notebook. Another one has a dell notebook but same graphic card.

When i ran same jupyter notebook on same dataset to train the same number of layers, it takes 20 minutes on mine but 6-8 minutes on both of my friend's.

I tried resetting the whole environment, but didn't helped.


texts: Is vanilla RNN suitable for time series prediction?

texts:
I read this document: https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/
It was pretty simple, but I don't understand how to use it for predict the next sequence (for example) in trading systems.

In the above article it "loops" and returns each cycle a "bit", but if do I need only ONE output (like in trading systems)?

I suppose that I have to feed "outputs" also/only, is this right?

Thank you


texts: I have a large labeled dataset with 29 classes. Is is possible to use a clustering algorithm (like k-means) in this dataset, or it's not possible since clustering algorithms are unsupervised ?


texts: Predict the next player's option for betting based on playing history?

texts:
I have a project which is to predict in next time, which option player will select for betting. This prediction based on his history. For example, the player have time series data, the last options they selected, how much money they spend, did they loose or win...ect. There are many options for one bet, around 300. Based on that data, I want to predict next time which option (or possibility of options) they will choose for betting.

I don't know which kind of problem it is. Is it kind of product recommendation, multi-class classification or multi-label classification? Any keyword, or suggestion to solve this kind of problem? Thanks.


texts: Designing a destination prediction model based on historical trips

texts:
I am working on a dataset with more than 100,000 users who ask for ride in Uber-like services.
The features are the following: user.id ,trip-origin-lat-long,trip-destination-lat-long, request timestamp.

I want to make a model that predicts destination of user based on his/her trip history.

I think it is a classification problem.
so, I have 2 questions:

1-which kind of classification algorithm should I use?
2-should I train a model for every single user or should I train a model for all users based using their user id as a feature?


texts: How can I classify specific types of words in a document given I have the full text of the document and the labels

texts:
I am working on a project that involves picking out specific kinds of objects from text. The documents I am going though are life sciences and biomedical in nature, and in these documents there are specific biomedical "objects" I want to pick out. The nature and variety of the text objects means I can't use regex or string matching. It has to be some kind of classification.
These text objects can be one word, or multiple words, but they are always in sequence.
An example sentence would be like
During the process of protein synthesis, X was used.

I need to pick out X. Luckily, I have plenty of labeled documents, and plenty of labels to go along with it. So I know a human can pick out these objects. So the challenge now is to get a machine to be able to pick out these types of objects from unseen text. I am working under the assumption that these specific text objects all fall under somewhat similar grammatical and textual context, so given enough labeled data, a machine should be able to learn how to pick out the text object.
Two Main Questions.

How do I label specific words in a document such that some model will understand that given a sequence of text, the object at position Y is a labeled and what we should be trying to classify.

Does anything I just said make any sense? Is there any research on what I've been talking about, because I've looked around and have not been able to find much.


texts: How to make multiple regression perform better for outliers? (without reducing effect of them)

texts:
I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values.

So I wonder what technics exist to make algorithm perform better for high values that can't be dismissed as outliers. You see, these few high values make MSE very large, because the models tends to underestimate these high values, predicting them to be 2 or more times smaller.

I have an idea to generate fake data for outliers, but I haven't found how to do it the right way for regression. Is it right to generate fake data for high values based on their proximity by mixing their features? So it would be smth like SMOTE, but instead of classes we have nearby values?

Or, perhaps, the other idea is to cluster targets by density and then to generate balanced clusters by generating fake data with SMOTE or ADASYN?

P.S. Note that I don't want to reduce effect of outliers. On the opposite, these high values are extremely important, so I want model to perform well for them.


texts:
I understand that there is a stack of RBM's and then a supervised step where the feed forward neural network is initialized with the weights from the RBM.
However, what I don't understand is- does it only have to initialise the weights from the input to the first hidden layer in the feed forward neural network? What about the weights from the first hidden layer to the second layer?


texts: will increasing threshold always increase precision?

texts: here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before?


texts: How to prefer no choice instead of bad choice with sklearn decision tree

texts:
I'm using sklearn decision trees to classify documents in two possible types "type1" and "type2".

I've isolated few features that seem pertinent and tried to combine them manually to evaluate the result of a model.
When classifying the documents manually i use the following outcomes:

type 1
type 2
unknown

Then I give the same features to a decision tree. The results are worse in that case because it always tries to classify documents in one of the categories "type1" or "type2" but is unable to classify documents in "unknown"

Is it possible to configure a sklearn decision tree in a way that in case of high uncertainty the document won't be classified instead of selecting a category that is likely to be wrong?


texts: How many repititions of a stochastic algorithm should be done before the results can be considered trustworthy?

texts:
I have written a stochastic machine learning algorithm.

Each time I run the algorithm on the same test data I get different results. I determine accuracy each time compared to ground truth.

In order to properly assess the system's performance I must run the algorithm on the same test data multiple times and take the mean of the accuracy of all the runs.

I have been asked to justify the number of runs of the algorithm that I do. Is there a scientific method for determining this or a piece of research I can cite that suggests a value?


texts:
In some places, perceptron is described as having added bias, while in some places, bias is not added.
Which one is right for you?


texts:
I am trying to build a simple feedfoward neural network with three layers for an autonomous car control. The network takes image frame inputs from a live video stream,  and gives steering value as output ranging from 0 to 1000. The inner layer has 38400 units  and output layer 1000 units. My training dataset is 2592 large and the problem I am having is finding the right number of neurons in the hidden layer. I have followed some rules of thumb that I found online , like

The number of neurons in the hidden layer should be between the input and output layer
The training dataset should be at least twice as large as the number of neurons in the hidden layer.

That gave me a boundary of 1000 and 1296 neurons. I’ve tried different architectures of networks within this range but the best accuracy I’ve got is not more than 2%,  with a single  epoch. I don’t know what’s going wrong. What should I do ?

I will appreciate any help I get.


texts: CNN Multi-Class Model Only Predicts 1 class for all test images

texts:
I am trying to build a CNN model to predict 42 classes. I used pre-trained models for this. I used Xception.
This is how I have imported my dataset:
train_datagen = ImageDataGenerator(rescale =1.0/255.0,
                                   zoom_range = 0.2,
                                   shear_range = 0.2,
                                   horizontal_flip = True,
                                  validation_split = 0.2)
training_data = train_datagen.flow_from_directory(train_path,
                                                  target_size = (299,299),
                                                  batch_size = 32,
                                                  class_mode = 'categorical',
                                                 subset = 'training')
validation_data = train_datagen.flow_from_directory(train_path,
                                                   target_size = (299,299),
                                                   batch_size = 32,
                                                   class_mode = 'categorical',
                                                   subset = 'validation')

I then built my model:
import keras
prior = keras.applications.Xception(include_top = False, weights = 'imagenet', input_shape = (299,299,3))
model = Sequential()
model.add(prior)
model.add(Flatten())
model.add(Dense(256, activation = 'relu'))
model.add(Dropout(0.1))
model.add(Dense(42, activation = 'softmax', name = 'Output'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

My model performs quite well but I do not know why when I predict it on the test set, it predicts all the same class.
This is the epochs:
Epoch 1/3
2636/2636 [==============================] - ETA: 0s - loss: 1.9625 - accuracy: 0.4816
Epoch 00001: val_accuracy improved from -inf to 0.56412, saving model to xception.hdf5
2636/2636 [==============================] - 5041s 2s/step - loss: 1.9625 - accuracy: 0.4816 - val_loss: 1.6792 - val_accuracy: 0.5641
Epoch 2/3
2636/2636 [==============================] - ETA: 0s - loss: 1.4015 - accuracy: 0.6224
Epoch 00002: val_accuracy improved from 0.56412 to 0.64584, saving model to xception.hdf5
2636/2636 [==============================] - 5101s 2s/step - loss: 1.4015 - accuracy: 0.6224 - val_loss: 1.3240 - val_accuracy: 0.6458
Epoch 3/3
2636/2636 [==============================] - ETA: 0s - loss: 1.2084 - accuracy: 0.6747
Epoch 00003: val_accuracy did not improve from 0.64584
2636/2636 [==============================] - 4968s 2s/step - loss: 1.2084 - accuracy: 0.6747 - val_loss: 1.4000 - val_accuracy: 0.6373

However, when I predict its all one category:
['39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',

Any help is appreciated. I would also like to know how I can use pre-trained models for these.


texts: Is data subsampling appropriate for hyperparameter optimisation?

texts:
Fundamentally, under what circumstance is it reasonable to do HPO only on a subsample of the training set?
I am using Population Based Training to optimise hparameters for a sequence model. My dataset consists of 20M sequences and was wondering if it would make sense to optimise on a subsample due to restricted budget.


texts: Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad?

texts:
This is the update equation of a weight by AdaGrad:
$$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$
Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is just a small value to prevent division by zero and $G_{w_{old}}$ is the gradient of $w_{old}$
Here $G$ is the sum of all the gradients, and so we are giving equal importance to all the gradients irrespective of the timestep.
Now, wouldn't it make more sense to give more importance to gradients than the one in distant past. Because, this is what we do in SGD with momentum. We give more importance to the gradients than the ones in the distant past.
So, why don't we do the same thing over here, Giving more importance to the gradients than the ones in the distant past?


texts: Whether solution of Kernel ridge regression through inverse calculation yields global optimal solution or not?

texts: As Kernel Ridge Regression computes inverse (Not a pseudo inverse) of a square matrix, so whether obtained solution would always be global optimal or not?


texts: is perceptron based implementation of Logical conjunction (AND) regression or classification

texts:
A perceptron model could be used to implement a Logical conjunction (AND) operator.

In this case, is the problem regression or classification?


texts: Data Cleaning for Discrete Features. Data list that contains null or N/A data

texts:
I ran into this kind of problem in my projects and I want to see if there more ways to solve it.

EXAMPLE
There is some data about apples and pears and the features are dominant color (red, green) and weight in grams. Some apples and pears in the data are missing the color. I know that I can just ignore and remove those apples and pears but is there another way to solve this problem? Since the data for the color is discrete, one or the other, and not continuous, I can't just plug in the median or average color of the other apples and pears. Let's also say that I can't assume or make educated guesses from the data. How should I solve this problem without removing data?


texts: Creating a neural network for predicting next vote in a series of votes

texts:
We are working on a project for creating music based on crowd sourcing. People vote for every note until the vote is closed, and then move on to the next vote until the canvas for the music is filled. A similar project is crowdsound, if you want to get an idea of what it looks like.

Now the fun part is, based on all the votes we get from various people, we would like to be able to build a Neural Network that can build an entire song on its own. The idea is for it to take in account every preceding vote and predict the one that will follow. That way, when trained, we could give it one note and let it predict the rest of the votes on its own and thus create a song on its own.

So I've read a few things here and there about neural networks, but there are two things I don't understand:

How to build one that takes into account a dynamic number of inputs (all preceding votes).
How exactly should I decide the number of hidden layers (I still only vaguely understand what those hidden layers represent) I need for it to work well.

We are using Java for the project and we were planning on using Neuroph for the neural network.


texts: Dealing with a dataset with a mix of continuous and categorical variables

texts:
How do the choice of machine learning algorithm and preprocessing change when some of the independent variables are categorical while others are continuous? Can such data be directly applied to the algorithm with categorical data converted using one-hot encoding?

For example, 2 columns are: Age and Race, Age being continuous and Race is categorical.

Update: Ideal buckets for a continuous variable is not known.


texts: How to gather training data for simple voice commands?

texts:
I'm trying to build a machine learning model for recognizing simple voice commands like up, down, left, etc.

On similar problems based on images, I'd just take the picture and assign a label to it.

I can generate features and visualize them using librosa. And I hear CNNs are amazing at this task. So, I was wondering how I'd gather training data for such audio based systems, since I can't record an entire clip considering my commands are only going to be a few milliseconds long (and while predicting, it'll only be a few milliseconds too).

My first idea was to generate an image (using matplotlib and librosa.features) for every few milliseconds and just use the ones where I'm saying the command as positive labels (and the rest negative labels), but it seems tedious and there might be a better way to do it.

Is there an effective way to do achieve this ?


texts:
I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs?

For example, in a list (of millions) that includes strings

['rs12346','rs1212122',...,'sxs-rs333',...,'kgp222']

..is there a way to extract the following patterns?

'rs' plus one or more digits
'sxs-rs' plus one or more digits
'kgp' plus one or more digits

Some other parameters:

cleaning up beforehand isn't an option
some manual tweaking would be possible (e.g. manually changing a pattern because of outside knowledge)
exceptions (i.e. classifying less than 100% of the strings) can be tolerated
an ideal solution would be using a built-in python library.


texts: I am writing a research paper and I am looking for reliable sources that provide information on disasters of machine learning. Especially in the field of autonomous driving. Have there been any incidents when something went really wrong? Any links to articles or research papers would be helpful.


texts: How can I fix this "convex" problem ? Is it just a matter of overfitting?

texts:
I get some metrics on validation data while training a model , and in my case the they are :

  (0.25, 0.31, 0.46, 0.57, 0.65, 0.75, 0.77, 0.78, 0.84, 0.84, 0.85,
  0.84, 0.84, 0.84, 0.82, 0.8, 0.8, 0.79, 0.78, 0.77, 0.77, 0.77, 0.75, 0.74, 0.73, 0.73, 0.73, 0.73, 0.73, 0.73)

They can described like this :

In my view , the ideal result should be like :

Is it a matter of overfitting ?

Unfortunately , I tried few times to change the regular coefficients to avoid overfitting , and adjust learning rate coefficients to slow down , but it was still "convex" .

How can I achieve the ideal result showed above ?

Much appreciated if anyone would give me some constructive tips ?


texts: How to quantitatively evaluate raw neural network activations?

texts:
Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam optimizer.

What kinds of trends in the numerical values of the activations should I be looking for with regard to a NN that is being trained successfully?

Do you want most activations to be either high or zero... not somewhere in the middle?
What values would indicate that you need to add/ remove a layer?
What maximum activation value is too high? Like if I saw an 8 should I be worried when the rest of my high values are around 4?


texts: Get a portion of a long field in Pandas?

texts:
I have a Pandas dataframe that has some fields that contain very verbose text. I want to be able to iterate through the DF but only display a limited set of words. I have code similar to:

for index, row in df.iterrows() :
    print(row['A'], row['B'])

How can I make sure that I only print the first 300 characters from 'A'?


texts: Why is hard for neural machine translation model to learn rare words?

texts: i'm kinda new with neural machine translation. I've read some papers, authors usually limit the size of vocabulary by replace rare words by unk token. In this paper, they said that "...NMT model cannot learn the translation of rare words...". I want to understand why is hard for NMT mode to learn rare words and also the impact of word counts to NMT model. Thanks