texts: How many epochs should all the data be trained on after training with validation finds when validation and training diverge?

texts: One uses and train/test split to use their training data to get an idea of how many epochs to train with. If the validation accuracy starts going down while the training accuracy is still going up, this would show a sign of overfitting, so one should probably stop the number epochs around there. But should training with all the data take longer to overfit? If so, should one add a few epochs? And if so for that, is there a good rule of thumb for how many? Maybe training different amounts of data could give an idea of what the differences are for number of epochs for the same amount of accuracy, even though they might not reach the same maximum amount. Any ideas on this?


texts: Change raw count to a proportion based on a filter in Tableau

texts:
In Tableau, I commonly want to carry out a task like distinguishing 2 groups by various measures.

The example below shows the difference in the count of 2 treatment and control for variable levels of a factor (measure):

This view is misleading though, because the overall group sizes of treatment and control are not equal.

What I'd like to do is to represent this as a proportion for each levels of the factor (measure) instead of using just a count. For instance, perhaps 25% of the treatment group is in Business Development whereas only 10% of the control group is in Business Development.

Is this possible to do in Tableau, or does it strictly require pre-processing?


texts: Hierarchical clustering in R filtering variable

texts:
I would like to test the added value of features compared to currently used predictors.
First, I checked if features were not correlated to the predictors (volume and intensity) I already use, and for the ones that are not correlated I would like to only keep the ones that are not highly correlated. So for each hierarchical cluster, I would like to keep the feature that is least correlated with volume and intensity, so far I only found how to delete all high correlations within the correlation matrix, but I would like to keep one per cluster. Is there any package or code in R that can do so?
So far I have this:
tmp <- cor(data)
tmp[upper.tri(tmp)] <- 0
diag(tmp) <- 0

data.new <- data[,!apply(tmp,2,function(x) any(x > 0.7))]

I'm quite new to coding, so any help will be much appreciated!


texts: Should the depth on convolutional layers be set to a figure divisible by 2?

texts:
I'm reading a book titled Python Deep Learning, and in Convolutional layers in deep learning on the chapter 5, the following is written:

  One more important point to make is that convolutional networks should generally have a depth equal to a number which is iteratively divisible by 2, such as 32, 64, 96, 128, and so on. This is important when using pooling layers, such as the max-pool layer, since the pooling layer (if it has size (2,2)) will divide the size of the input layer, similarly to how we should define "stride" and "padding" so that the output image will have integer dimensions. In addition, padding can be added to ensure that the output image size is the same as the input.

As far as I know, the width and the height should be a figure divisible by 2, in order to use a pooling layer. However, I don't understand why the depth must be a figure divisible by 2. The pooling layer just operates on a 2-dimensional screen based on width and height, and it operates on each filter (depth) separately, right?

Why should the depth also be set to a figure divisible by 2?


texts: Dimensionality reduction with prior knowledge of colinearity between features

texts:
Let's say that I have sparse feature vectors and I'd like to use dimensionality reduction in order to visualize them more easily.

Dimensionality reduction techniques like PCA will estimate colinearity between the features given the data. What if I have some prior knowledge on the colinearity between my features? As in, I would be able to create an approximate distance matrix between my features, and therefore between my data points.

I am aware that if features are actually colinear, methods like PCA will find a way to reduce them, however, I am afraid that I don't have enough data points to infer that colinearity strictly from the data and that PCA would remove meaningful features and keep uninformative ones.

Let's say that my data looks like something like this:

$x_0 = \{1,NaN,-1,NaN\}$

$x_1 = \{1,NaN,NaN,-1\}$

In my case, I know that the 3rd and 4th features are colinear and that the distance between $x_0$ and $x_1$ is close to zero. Therefore, they should be mapped to a very similar data point in a lower-dimensional space.

Can PCA still do it with so few data points? Are there ways to "force" the known colinearity or distance measures? T-SNE perhaps?


texts:
I am currently trying to build a self adjusting network, such that given any number of inputs, should always provide an output of shape (15,145)

The network structure is pretty simple and looks like this:

    inputs = 36
    list_of_input = [Input(shape = (45,5,3)) for i in range(inputs)]
    list_of_conv_output = []
    list_of_max_out = []
    for i in range(splits):
        list_of_conv_output.append(Conv2D(filters = 145 , kernel_size = (30,3))(list_of_input[i]))
        list_of_max_out.append((MaxPooling2D(pool_size=(3,2))(list_of_conv_output[i])))

    merge = keras.layers.concatenate(list_of_max_out)
    #reshape = Reshape((merge.shape[0],merge.shape[3]))(merge)

    dense1 = Dense(units = 1000, activation = 'relu',    name = "dense_1")(merge)
    dense2 = Dense(units = 1000, activation = 'relu',    name = "dense_2")(dense1)
    dense3 = Dense(units = 145 , activation = 'softmax', name = "dense_3")(dense2)

    model = Model(inputs = list_of_input , outputs = dense3)
    model.compile(loss="sparse_categorical_crossentropy", optimizer="adam")

    print model.summary()

    raw_input("SDasd")
    hist_current = model.fit(x = [train_input[i] for i in range(100)],
                        y = labels_train_data,
                        shuffle=False,
                        validation_data=([test_input[i] for i in range(10)], labels_test_data),
                        validation_split=0.1,
                        epochs=150000,
                        batch_size = 15,
                        verbose=1)

It been adjusted for having 36 inputs which would given it an output shape of (15,1,145) - but how can i determine the number of filters, kernel size and pooling size that would give me the desired output size.  The network is supposed to be used for classification, and the output vector of length 15 with classes for each third entry in the first axis (45 = 15*3). the total number of classes is 145, hence output dimension (15,145)


texts: Does Min Max algorithm leads to information loss?

texts:
Simplifying the question with an example.
Lets say I have a time series data and the variables are time, temperature, growth_of_microbe. Now I have observed usually with a higher high of temperature there is an effect on growth of microbe. Sample of the data is as follows

| Hour of Day(H) | Temp(T) | Microbe(P) | Min-Max of P(P#) | Why shouldn't P# be? |
|:--------------:|:-------:|:---------:|:----------------:|:-------------------:|
|        1       |    30   |     12    |   (12-1)/(23-1)  |
|
|        2       |    32   |     14    |   (14-1)/(23-1)  |   (14-12)/(14-12)   |
|        3       |    31   |     5     |   (5-1)/(23-1)   |     (5-5)/(14-5)    |
|        4       |    33   |     23    |   (23-1)/(23-1)  |    (23-5)/(14-5)    |
|        5       |    35   |     1     |   (1-1)/(23-1)   |     (1-1)/(23-1)    |

With respect to the above table, Min-Max uses the lowest/highest data  within the total available time rather than the lowest/highest till the time the value is calculated

Doesn't this lead to information loss?


texts: Is it possible to train a machine learning trainer?

texts:
That's just a weird thought experiment I recently came up with. I am new to this field actually. Anyway, here is the idea:

Suppose we have data D, and a machine learning algorithm A1. Then we use A1 to train D to get the hypothesis H. I denote this process as follows:

A1(D) = H

Then, suppose we have a machine learning algorithm trainer A2, which we can use to train a better machine learning algorithm, A1*, by inputing all the row data, D, the hypothesises trained by the machine learning algorithms, H, and the algorithms itself A1:

A2(D,H,A1) = A1*(actually there should be a lot of H and A1)

And this process can go on and on. Hopefully you can get my idea. Sorry in advance if that doesn't make sense to you since English is not my native language. So do you guys think it is possible to train, let's say, a algorithm A3, which can be used to train a machine learning algorithm trainer?


texts: Neural Network cannot learn nonlinear function

texts:
I am currently creating a neural network to learn a function of the following form Data that I want to learn x corresponds to x axis and y to y axis(one dependent and one independent variable)

I am using both keras and tensorflow and with both scripts I get the following result

I am currently creating a neural network to learn a function of the following form Data that I want to learn x corresponds to x axis and y to y axis(one dependent and one independent variable)

I am using both keras and tensorflow and with both scripts I get the following result Predictions orange line Data blue line. Somehow my neural network doesn't capture the non-linearity of the data and only tries to fit a linear function. Do you maybe have a suggestion what I am doing wrong? Also is the architecture appropriate for the following task or there exists some problems.

Additionally as an information I also include a snippet of the architecture that I am using in keras

    def individual_model(keys, labels, config):
model = Sequential()
model.add(Dense(32, input_dim=1))
model.add(LeakyReLU())
for i in range(2):
    # if str(i) not in config:
    #     break
    model.add(Dense(32))
    model.add(LeakyReLU())
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse', metrics=[
              max_absolute_error, 'mse', 'mae'])
model.fit(keys, labels, epochs=100, batch_size=32, verbose=1)
return model


texts: Measuring quality of answers from QnA systems

texts:
I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of answer along with their logits.
The answer is formed by choosing the best logits span and final probability is calculated by summing the start and end logits.

Now the problem is, I have multiple answer and many times the good answer is at 2nd or 3rd place (after sorting on the result of sum of start and end probability).  Is there any metric in search engine science using which I can rank the best answers?

Followings have been tried:

cosine similarity between question words and answers  -  This works many times but  fails when question semantic meaning is complex
TFIDF - gives good score but fails when there is synonym in answers rather than matching word.
gensim semantic similarity - fails badly.
BLUE score and new BERTF1Score also tried

Few terms I heard of but I doubt if these work, like Mean Reciprocal Rank which I think gives search quality rather than answer quality and also the correct response is required to calculate MRR (Please correct if I am wrong). Or the PageRank which is not valid in my case as the answer semantic meaning is preferred in QnA rather than the document popularity.

Kindly suggest other metrics which search engines generally use to rank the answers.


texts:
I am dabbling in active learning and was wondering how to combine this in seeking out the best architecture for the network.
In my understanding, active learning uses a heuristic for selecting the best instances to label in order to learn as quickly as possible. However, the way these instances are chosen are dependent on the model itself.
Is there a way to handle this model dependency?
It seems to me that models architecture is dependent on the train size, for example. If that's the case, wouldn't it be beneficial to allow the model to change its architecture during active learning?
Is there a way to do this or do we have to be very careful when selecting a model before performing active learning.
Another possibility I see is to perform a grid search on the networks architecture on all the labeled data after every so-often queries. But then again, the model that comes out on top is still dependent on the initially chosen model...


texts: I am working on super resolution projects using Generative Adversarial Networks. I need to validate the generated image whether they are close to original one. I already worked with SSIM and PSNR. is there any recent algorithms to compare two images?


texts: Extract canonical string from a list of noisy strings

texts:
I have thousands of lists of strings, and each list has about 10 strings. Most strings in a given list are very similar, though some strings are (rarely) completely unrelated to the others and some strings contain irrelevant words. They can be considered to be noisy variations of a canonical string. I am looking for an algorithm or a library that will convert each list into this canonical string.

Here is one such list.

Star Wars: Episode IV A New Hope | StarWars.com
Star Wars Episode IV - A New Hope (1977)
Star Wars: Episode IV - A New Hope - Rotten Tomatoes
Watch Star Wars: Episode IV - A New Hope Online Free
Star Wars (1977) - Greatest Films
[REC] 4 poster promises death by outboard motor - SciFiNow

For this list, any string matching the regular expression ^Star Wars:? Episode IV (- )?A New Hope$ would be acceptable.

I have looked at Andrew Ng's course on Machine Learning on Coursera, but I was not able to find a similar problem.


texts: Can someone spot anything wrong with my LSTM forex model?

texts:
The model below reads in data from a csv file (date, open, high, low, close, volume), arranges the data and then builds a LSTM model trying to predict next day's close based on a number of previous days close values.

However, validation accuracy is about 53.8% no matter if i...
- change hyperparameters
- make it a deep model
- uses many more features than just close

To test if I made a simple mistake I generated another data source that was a signal I created from adding sin, cosine and a little noise so that i KNEW a model should be able to be trained and it was. The model below got about 94% validation without any tuning.

With that in mind. How come when I try to use it on actual data (eurusd 1-min data) it doesn't seem to work..?

Anyone that sees an error or can point me in the right direction?

import pandas as pd
import numpy as np

fpath = 'Data/'
fname = 'EURUSD_M1_1'
df = pd.read_csv(fpath + fname + '_clean.csv')
# file contains date, open, high, low, close, volume

# "y" is whether the next period's Close value is higher or lower than current Close value
outlook = 1
df['y'] = df['Close']<df['Close'].shift(-outlook)

# Drop all NAN's
df.dropna(how="any",inplace=True)

# Get X and y. To keep it simple, just use Close
X_df = df['Close']

# "y" is whether the next period's Close value is higher or lower than current Close value
outlook = 1
y_df = df['Close']<df['Close'].shift(-outlook)

# Train/test split
def train_test_split(X_df,y_df,train_perc):
    idx = int(train_perc/100*X_df.shape[0])
    X_train_df = X_df.iloc[0:idx]
    X_test_df = X_df.iloc[idx:]
    y_train_df = y_df.iloc[0:idx]
    y_test_df = y_df.iloc[idx:]
    return X_train_df.as_matrix(), X_test_df.as_matrix(), y_train_df.as_matrix(), y_test_df.as_matrix()

X_train_df, X_test_df, y_train, y_test = train_test_split(X_df,y_df,90)

# Scaling
def scale(X):
    Xmax = max(X)
    Xmin = min(X)
    return (X-Xmin)/(Xmax - Xmin)

X_train_scaled = scale(X_train_df)
X_test_scaled = scale(X_test_df)

# Build the model
import tensorflow as tf
from keras.models import Sequential
from keras.layers import LSTM, Dense

# ### Constants
num_time_steps = 5   # Num of steps in batch (also used for prediction steps into the future)
num_features = 1   # Number of features
num_neurons = 97
num_outputs = 1 # Just one output (True/False), predicted time series
learning_rate = 0.0001 # learning rate, 0.0001 default, but you can play with this
nb_epochs = 10 # how many iterations to go through (training steps), you can play with this
batch_size = 32

# Reshaping
X_train_scaled = np.reshape(X_train_scaled,[-1,1])
nb_samples_train = X_train_scaled.shape[0]  - num_time_steps

X_train_scaled_reshaped = np.zeros((nb_samples_train, num_time_steps, num_features))
y_train_reshaped = np.zeros((nb_samples_train))

for i in range(nb_samples_train):
    y_position = i + num_time_steps
    X_train_scaled_reshaped[i] = X_train_scaled[i:y_position]
    y_train_reshaped[i] = y_train[y_position]

model = Sequential()

stacked = False
if stacked == True:
    model.add(LSTM(num_neurons, return_sequences=True, input_shape=(num_time_steps,num_features), activation='relu', dropout=0.5))
    model.add(LSTM(num_neurons, activation='relu', dropout=0.5))
else:
    model.add(LSTM(num_neurons, input_shape=(num_time_steps,num_features), activation='relu', dropout=0.5))

model.add(Dense(units=num_outputs, activation='sigmoid'))

model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics=['accuracy'])

history = model.fit(X_train_scaled_reshaped,
                    y_train_reshaped,
                    batch_size = batch_size,
                    epochs = nb_epochs,
                    validation_split=0.3)


texts: How to pass inputs (interactively) to a model?

texts:
Let me give you a high-level design (blueprint) of my model.

Input data::

CATEGORY    SERVICE     TITLE                               DEPARTMENT

apple       fruits      i love eating fruits.               fruitshop
mango       fruits      mangoes are yellow in color.        fruitshop
cycle       vehicle     that cycle is really expensive.     motorshop

I am trying to predict the value of DEPARTMENT (i.e. my output class), by consuming - CATEGORY, SERVICE and TITLE - as my input classes.

Pseudo code::

Converted CATEGORY and SERVICE columns into fixed numerical values -

df['CATEGORY'], mapp0 = df['CATEGORY'].factorize()
df['SERVICE'], mapp1 = df['SERVICE'].factorize()

For the TITLE, used NLTK and Tf-IDf techniques to create a sparse matrix with the individual word (tokens) as features and the weights forming the matrix values.
Then outputs of 1 and 2 are combined to form another sparse matrix, which is my main feature matrix (It has approximately 1100 features).
I'm so new to ML that not even sure if step#3 is even allowed (or not even a standard practice).
Next, the usual steps are performed like CV, fit, predict, etc.

I am using LinearSVC and StratifiedKFold (n_splits = 10, random_state=777, shuffle=True) able to achieve a prediction accuracy of 73%.

Question::

I want to pass inputs (CATEGORY, SERVICE and TITLE) to my model and it should predict the DEPARTMENT. How can I achieve this part?


texts: Similarity coloring of a self organizing map

texts:
I have implemented the algorithm to train self organizing maps in Python and it seems to be working well. I checked with some labeled data and the maps are learning the topology well. Here are some images:

I am calculating a Unified Distance Matrix for visualization which does not seem to be working well as seen in the 2nd image.

I believe that similarity coloring as seen in the world poverty map developed by Prof. Kaski and Prof. Kohonen would be apt for my application but I am unable to find a clear explanation of how it is to be calculated. The procedure has been outlined in the paper "Tips for Processing and Color-coding of Self-Organizing Maps ", however, it omits some technical details which are necessary to implement the similarity coloring. Here is the image that I am referring to:

Could someone help me with the exact steps to calculate the similarity coloring map? I have searched everywhere for a month and I am unable to find an explanation.


texts: Is it possible using tensorflow to create a neural network that maps a certain input to a certain output?

texts:
I am currently playing with tensorflow, but can't seem to get a hold whether it usefull for my problem?

I need to create a neural network, that is capable of mapping input to output.

The way things are progression now, i haven't a single example in which this has been done, all types of problem tensorflow seem to solve are classification task, and not this kind of mapping problem... Is tensorflow able to do so?, do I have to use a different neural network framework to do so?, And if it is available could somebody show some code (not the MNIST example)

My task:

I am currently trying to make an neural network that takes in samples of audio files, and generates MFCC features from the samples. The MFCC features can be calculated manually, which I have done, so I know what the I output i seek is.  MFCC feature is a feature vector is different real values numbers.  I cannot be classified as class A or Class B, or doing so would greatly reduce the accuracy of the output, as you are fitting the output to predetermined "bins"...


texts: Neural Network for Multiple Dependent Outputs

texts:
I have a dataset with approx 6 input features and 5 output values to be predicted. I am trying to understand what kind of neural network would be most suitable to assign probability across multiple dependent outputs, with the motivation to maximise the value of a single feature.
For example, suppose the inputs are:
company size
number of employees
turnover
average salary
country
years of operation

And the outputs to be predicted are:
% budget allocated to marketing
% budget allocated to sales
% budget allocated to R&D
% budget allocated to training
% budget allocated to shareholders

The training dataset also contains the column:
profit

Which is what I want to optimise against (i.e. the primary motivation of the model).
For a given set of inputs (company size, number of employees, turnover, average salary, country, years of operation), I want to be able to predict values for the 5 outputs which are most likely to achieve the highest 'profit'. The sum of the 5 outputs must equal 100%.
In other words, there is a finite budget to be allocated, and I want to create a model to predict the best budget allocation to maximise profit.
What neural network would be most suitable for this purpose? I have looked into multi-output regression, however I imagine this would assign independent probabilities for each of the 5 outputs (i.e. they won't add up to 100%).
Is it possible for probability to instead be assined to each output dependent of the other outputs (i.e. so they add up to 100%)? If so, is there a name for this type of approach?


texts: Can you choose a binary feature matrix for a binary classification model

texts:
This may be a stupid, but, I am new to deep learning (and machine learning for that matter) and I can't seem to find any literature to help with my question. All I can see when Googling many different questions (trying to change keywords to try get a hit on my question) is about binary classification. And also, binary classification where the feature matrix consists of real numbers.
I would like to know, is it possible to build a binary classifier with a binary feature matrix? And please can you point me to some literature.


texts: Which ML method for multiclass (non-binary) text classification should I choose (from SparkML)?

texts:
I am working on a quite big dataset that will be processed on the cluster, so this is why I am using PySpark for that purpose.

The presentable records of this dataset have a such structure:

+----------+------------+--------------------+--------------------+--------------------+
|         0|  07/29/2013|       Consumer Loan|        Vehicle loan|Managing the loan...|
|         1|  07/29/2013|Bank account or s...|    Checking account|Using a debit or ...|
|         2|  07/29/2013|Bank account or s...|    Checking account|Account opening, ...

After some preprocessing/data cleansing operations I would like to create and then obviously train a model that will classify issues (Issue) into some categories, that are still unknown. I have read some articles about TF-IDF, but not sure if this could be suitable for this case.


texts: Find columns with numeric values, but stored as string

texts:
I need to find the columns in data frame, which has numeric values and are stored as string.

data_set = pd.DataFrame({"Number":["1","2","3","4","5"], "Char":["A","B","C","D","E"]})
data_set.dtypes

In above code, column "Number" has numeric values, but stored as string. I have to iterate through columns and convert it to int or float.

I can do it by taking column name and convert it.

data_set["Number"] = data_set["Number"].astype(int)

What i need is to do it dynamically. Dynamically means, iterating through columns and changing it.


texts:
I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized images (for example 32x32) to input size, the content of the image is stretched horizontally too much, but for some medium size images it looks okay.

What is the proper method for resizing images while avoiding the content being destroyed?

(I am thinking about padding images with 0s to complete size after resizing them to some degree keeping ratio of width and height. Would it be okay with this method?)


texts: How to deal with variable number of permutation invariant features?

texts:
I want to learn from data where each record has a variable number of features that have no inherent order to them. Take as an example the task to predict whether a repair is worth it of some item. Each item can have a variable amount of features e.g. glass is broken, handle is missing etc. which are mostly categorical. There are up to +1000 different things to know about each item, but in reality there are only a handful present.
What ways are there to encode this kind of data or what models exist that can handle this natively?
I have a few things in mind

Binary encoding of set (resulting in a lot of dimensions)
Average features into single vector (dimensions become harder to interpret)

but I want to explore more options.


texts: Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

texts:
I have a preprocessing script (google cloud function) that generates files (stored in Google Drive). I want to load those files in my DW (Big Query).

What are the pros and cons of:

1) Running the preprocessing script, generate the files and then loadthose files

vs

2)  Loading data directly from the preprocessing script (skip the file generation, just do a direct insert in the DW from the preprocessing script)

?

I am interested in focusing the question not only in terms of technical stuff and costs, but also in terms of data processing methodology.

I think the question could lead to the dilemma loading online or more in a batch process.

I have added some conclusions of mine as answers.
Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks!


texts: Linear Regression error - InvalidArgumentError: assertion failed: [Labels must be <= n_classes - 1] [Condition x <= y did not hold element-wise:]

texts:
I am doing a linear regression model in TensorFlow. I have applied the code I saw on a course with my own dataset but I am getting an error I don't understand:

This is a link to the Google Colaboratory where the code is written:
Colaboratory
The problem must be on my dataset because with the dataset from the course it works, but I can't figure out what the problem is exactly.


texts: How to predict a None (not in the categories)?

texts:
Im using tensorflow.keras

I made a model that can predict cat and a dog but when I try to predict image without dog or a cat I still get an answer of a cat/dog.

I tried find the % value of the prediction to know how low/high the prediction was.
But I couldn't find much updated answers.

So how can I basically detect that thats not a cat or a dog?


texts: Is it possible to use a pretrained scikit learn model to make predictions on a dataset with different features (than those used during training)

texts: Say we have a model trained on dataset A, which has a number of features, as usual. We then persist that model to disk and use it when we need to run inference (make predictions). Usually we run inference against unseen data with the same features. Is it possible to run inference against unseen data with different features. Note that these new features are very similar to the ones used in training. So in a sense it would be a type of transfer learning (actually more like “transfer inference”). Can this be done in scikit learn, given that loaded models (from say Pickle) expect the column names to be the same?


texts: How do I run SMOTE on image data using the packages available?

texts: I need to balance some image datasets, how do I use SMOTE variants or the imblearn SMOTE method with images? I can't figure out how to, since they seem to be working only with numerical datasets.


texts: Finding an appropriate binary classification algorithm for time series data intervals

texts:
Maybe someone here has experience in this matter and can point me in the right direction. I want to classify parts of an interval of numerical movement data as either resting or not resting. I have training data of what resting intervals look like. And am looking for the right algorithm to tackle this problem. I don't need the code, just a friendly push in the right direction.
Here is a rundown of the expected process.

I have an array of values that represent measurement over 5 minutes.

testData = [0, 4, 3, 5, 2, 4, 3, 4, 6, 11, 9, 10, 3, 15, 21, 5]

I have training data

trainingDataResting = [[1, 2, 1, 6, 4, 2], [3, 2, 2, 0, 4, 1, 5, 2], [3, 6, 2, 1, 0, 4, 5, 2]]
trainingDataActive = [[10, 4, 5, 9, 19, 13], [12, 8, 20, 9, 14], [13, 22, 19, 21, 11, 7, 9]]

A resting interval has to be at least 6 measurements long, and the longest possible resting interval is of interest. Is there a way to classify portion of the testData as resting, based on the trainingData? Something like:
testData = [0, 4, 3, 5, 2, 4, 3, 4, 6, 11, 9, 10, 3, 15, 21, 5]
           '-------------------------' '----------------------'
                     Resting                 Not Resting

testResting = [0, 4, 3, 5, 2, 4, 3, 4, 6]
testActive = [11, 9, 10, 3, 15, 21, 5]

I get that there will be many solutions of possible resting intervals, but I am particularly interested in the longest and most resting one.
Of course I am working with a lot more training data than I provided here.
I was working on a decision tree -> Decision Tree, before I was informed that there is prelabeled data that I could use for training.
I was thinking about long short term memory networks, do you know whether this applies here?
Thank you for the help.
Best,
Jo


texts:
yes, DBSCAN parameters, and in particular the parameter eps (size of the epsilon neighborhood).
In the documentation we have a "Look for the knee in the plot".

Fine, but it requires a visual analysis. And it doesn't really work if we want to make things automatic.
So, I was wondering if it was possible to find a good eps in a few lines of code.

Let's imagine something like :

evaluate kNN distance
sort these values
scale them (so that
the values are always between 0 and 1)
evaluate the derivative
find the first point where derivative is higher than a certain
value, let's try with 1

In R, it would look like (using iris dataset as in the DBSCAN documentation) :

# evaluate kNN distance
dist <- dbscan::kNNdist(iris, 4)

# order result
dist <- dist[order(dist)]

# scale
dist <- dist / max(dist)

# derivative
ddist <- diff(dist) / ( 1 / length(dist))

# get first point where derivative is higher than 1
knee <- dist[length(ddist)- length(ddist[ddist > 1])]

and the result is 0.536 which looks quite good.

Is this approach relevant or totally nonsense ?


texts: How to add x and y error bars on a Bokeh python scatterplot?

texts: I have a bokeh scatterplot which I would like to add error bars in both the x and y direction. Is there any way to easily add this to each data point on the graph?


texts:
I am currently working on a Machine-Learning Model. In order to explain how it works, I have looked at Partial Dependence Plots, Feature Importance and all kinds of methods, but one thing still concerns me:
Should I evaluate these on the dataset I trained the model with, or the current dataset on which it performs predictions? This of course leads to different results depending on the distribution and the performance of the model.
How do you handle this?
Any opinions or input would be highly appreciated!


texts: What are the Objectives for Recommendation Systems and what Key Results should a Recommendation System focus on?

texts:
I know that a Recommendation system helps in the engagement of the users and helps users find more relevant content but I am in search of more complex objectives and key results with regards to Recommendation Systems.
Consider Youtube, their goal is to increase watch time with the system whereas some other company might have some other objective.


texts:
Which measurement(s) should one choose to compare two regression models?

After modifying a learning algorithm(specifically, a regression algorithm, let's call it M1) to generate another learning algorithm M2, how to validate if the above modification is efficient?

here is what I did(with 10-fold cross-validation)

I choose MSE as the only measurement, at each run, for M1 and M2, calculate the MSE of both the training and testing set.

And the result shows that:

average MSE of the training set of 10 runs: M2 < M1
average MSE of the testing set of 10 runs: M2 < M1

Question:

according to the above list, can we draw a conclusion that M2 is better than M1? thus, the modification of algorithm M1 is efficient(at least on this dataset)?

Or:

Did I miss some other important measurements? Is there a rule of thumb of comparing two regression models?


texts: Why is my LSTM, stateful network getting 0 accuracy

texts:
I'm building a NN to predict stock prices but I'm getting an accuracy of 0, I really don't know what hyper parameters to play with anymore but it seems to always get this accuracy

Build LSTM model

model.add(LSTM(
    50, activation="tanh", recurrent_activation="sigmoid",
    return_sequences=True, batch_input_shape=(batch, timesteps, features)))#stateful=True,
model.add(layers.Dropout(0.2))

model.add(LSTM(100, activation="tanh", recurrent_activation="sigmoid",
               stateful=True, return_sequences=True)) #stateful=True
model.add(layers.Dropout(0.4))

model.add(layers.LSTM(50, activation="tanh", recurrent_activation="sigmoid",
               stateful=True, return_sequences=False)) #stateful=True
model.add(layers.Dropout(0.2))

model.add(Dense(25))
model.add(Dense(1))

# Complie model
model.compile(optimizer="adam", loss="mean_squared_error",
              metrics=["accuracy", "mse"])

# Train the model
for i in trange(1, desc="Training the fuck out of this model"):
    model.fit(x_train, y_train,
              batch_size=batch, epochs=1,
              shuffle=False)#shuffle=False
    model.reset_states()```

#OUTPUT
```Epoch 1/1

   1/2918 [..............................] - ETA: 1:47:06 - loss: 0.0318 - acc: 0.0000e+00 - mean_squared_error: 0.0318
   2/2918 [..............................] - ETA: 56:57 - loss: 0.0246 - acc: 0.0000e+00 - mean_squared_error: 0.0246
   3/2918 [..............................] - ETA: 40:29 - loss: 0.0174 - acc: 0.0000e+00 - mean_squared_error: 0.0174
   4/2918 [..............................] - ETA: 32:15 - loss: 0.0135 - acc: 0.0000e+00 - mean_squared_error: 0.0135
   5/2918 [..............................] - ETA: 27:18 - loss: 0.0141 - acc: 0.0000e+00 - mean_squared_error: 0.0141
   6/2918 [..............................] - ETA: 24:00 - loss: 0.0150 - acc: 0.0000e+00 - mean_squared_error: 0.0150
   7/2918 [..............................] - ETA: 21:40 - loss: 0.0142 - acc: 0.0000e+00 - mean_squared_error: 0.0142
   8/2918 [..............................] - ETA: 19:48 - loss: 0.0125 - acc: 0.0000e+00 - mean_squared_error: 0.0125
   9/2918 [..............................] - ETA: 18:26 - loss: 0.0115 - acc: 0.0000e+00 - mean_squared_error: 0.0115
  10/2918 [..............................] - ETA: 17:20 - loss: 0.0108 - acc: 0.0000e+00 - mean_squared_error: 0.0108
  11/2918 [..............................] - ETA: 16:27 - loss: 0.0100 - acc: 0.0000e+00 - mean_squared_error: 0.0100
  12/2918 [..............................] - ETA: 15:42 - loss: 0.0094 - acc: 0.0000e+00 - mean_squared_error: 0.0094
  13/2918 [..............................] - ETA: 15:08 - loss: 0.0089 - acc: 0.0000e+00 - mean_squared_error: 0.0089
  14/2918 [..............................] - ETA: 14:32 - loss: 0.0083 - acc: 0.0000e+00 - mean_squared_error: 0.0083
  15/2918 [..............................] - ETA: 14:06 - loss: 0.0080 - acc: 0.0000e+00 - mean_squared_error: 0.0080
  16/2918 [..............................] - ETA: 13:39 - loss: 0.0076 - acc: 0.0000e+00 - mean_squared_error: 0.0076
  17/2918 [..............................] - ETA: 13:17 - loss: 0.0072 - acc: 0.0000e+00 - mean_squared_error: 0.0072
  18/2918 [..............................] - ETA: 12:58 - loss: 0.0068 - acc: 0.0000e+00 - mean_squared_error: 0.0068
  19/2918 [..............................] - ETA: 12:40 - loss: 0.0065 - acc: 0.0000e+00 - mean_squared_error: 0.0065```


texts: Avoid Load and re-Train of a Neural Network Model

texts:
I built a feedforward neural network, his goal is to classify a target value, the train has to be defined for n-days.

The input is constituted by 12 neurons (since I have 12 different data each of which for each day has around 25000 values [25000 x 1]). The network, for every single day, manages 12 * 25000 input values. (The data are all numeric)

Since I have to train for about 741 different days (This implies I have to perform a train with 741 * 12 * 25000 data), at the moment I'm able to train only for a single day, save the model and reload it to train for the subsequence day.

There is a way to avoid this save - load model procedure and train for all the 741 days at once? Should I have to use a particular data generator?

Can anyone give me some suggestions?

Thank you very much.


texts:
Suppose I have a data frame like this :
Hospital_name    State    Employees    ......
Fortis           Delhi    5000         ......
AIIMS            Delhi    1000000      ......
SuperSpeciality  Chennai  1000         ......

Now I want to use this data frame to build a machine learning model for predictive analysis. For that, I must convert the strings to float values. Also, some of these columns in Hospital_name and State contains 'NAN' values. In such a case, how should I prepare my data for building a model in Keras?


texts:
For the simple AND learning with a perceptron, it is required to have two inputs x1 and x2 and one target data y.

Most AND examples (such as in the book "Fundamentals of neural network-fausett") have this network topology in the figure above, that is 2 inputs(x1,x2) and 1 output(o). My question is, how do we give the network the target data (y) if there is only two inputs(x1,x2)?


texts: xgboost classifier predicted negative probabilities

texts:
I'm using XGBoost for a binary classification problem. There is no negative label, only 1 and 0.

I tunned the hyperparameters using Bayesian optimization then tried to train the final model with the optimized hyperparameters.

Mdl_XGB = xgb.train(OptimizedParams, dtrain)

scores_train = Mdl_XGB.predict(dtrain)

scores_test = Mdl_XGB.predict(dtest)

My problem is that the predicted scores for both train and test sets include both negative values and numbers greater than one. The scores are between -0.23 and 1.13.

Shouldn't these scores present the probability of belonging to class 1 (positive class)?


texts:
I want to build a neural network with a data input of 15-18 variables. I want to use the model for anomaly detection based on the reconstruction error. I've done some tutorials now but the data was always already given so I have to do the very first step on my own now.
Thus, I wonder, how do I handle the multivariate data?

I see three options:

1) Build a single neural network for each variable (univariate)
2) Build a neural network with a single outcome based on all multivariate input variables
3) Build a neural network with a multivariate output based on a multivariate input.

1) and 2) is quite cumbersome I'd say. But is 3) even possible? If so, what do I have to take into account?
Will the outcome variable then just be a data frame, or some other type like a certain array or so?


texts: what should be the Prediction or Target variable for Suicide data in India

texts:
I am very new to machine learning.I am studying Suicide data in India from kaggle dataset.
https://www.kaggle.com/rajanand/suicides-in-india/kernels
But I am unable to understand what should be the target variable for prediction or classification.
My immediate thought is that a socially important goal of such studies is a prevention of future suicides.For ex:
a number of suicides going to happen next year among students of a given high school. If a prediction value is high, then preventive measures need to be taken.
But I am unable to start which ML algorithm should i use or how to approach.I have completed data analysis.
Please help.


texts: How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

texts: If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?


texts: Math PhD (Nonlinear Programming) switching to Data Science?

texts:
I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is better understood:

Maths Coursework:

This has been mostly in pure maths: topology, functional analysis, etc, but also include more applied ones(on which I have specialized for the dissertation): convex optimization, nonlinear programming, numerical analysis, linear programming, multiobjective optimization. In addition, I have right now 0 knowledge of inferential stat, but I am confident in probability theory.

Programming:

I just took a year-long course in the Bachelor, but it was mostly Mathematica and some Java, of which I remember nothing honestly. In this course, the content did not include anything of data structures or design and analysis of algorithms, nor databases management systems. I also learned Matlab on my own for implementing algorithms in the bachelor thesis.

The above background was during the Bachelor and Master's program. Now, during the Ph.D. program, I discovered that Machine Learning is the perfect mix (for me) between Nonlinear Optimization, Programming and applications in the real world, i.e, it is both theoretically interesting and application-oriented. This is the reason why I became so excited to go to industry. Hence, I started learning things on my own(in my little free time) during the last 3 years.

Short summary of things learned:

Python: I am comfortable implementing optimization algorithms, work with jupyter notebooks and the numpy library (in fact, I had to do this for the dissertation), and doing basic data manipulations and cleaning tasks in pandas. This I learned online, in a platform called dataquest (https://app.dataquest.io). However, I don't think I have enough knowledge to pass an interview in data structures and algorithms(see above).
Machine Learning: I took a master level course in the topic at the uni (since I am in Germany, we don't have courses in the Ph.D., so this was all in my personal time), which I really enjoyed. Topics included: k-NN, PCA, SVM, NN, etc.
Taking a course in Databases this semester, which focuses on SQL.
Taking Deep Learning specialization on Coursera this semester.

Finally, I want to say that I feel totally capable of learning the topics. In fact, with time I intend to take more graduate-level courses available online(for example, Stanford CS231N, CS234, etc) because, in my opinion, online courses may not be rigorous enough. Hopefully, after the defense, I will be able to focus full time on this.

Hence the questions:

Can I still get hired at this point (I mean, after finishing this semester with the knowledge described above)? I honestly think I am not ready, but I feel confident that I  can get decent in a year.
Am I being too naive in thinking a company would give me a chance?
What should I do to become more hirable in any case?


texts:
I used XGBoost to predict company's bankruptcy, which is an extremely unbalanced dataset. Although I tried weighting method as well as parameter tuning, the best result which I could obtain is as follows:

Best Parameters: {'clf__gamma': 0.1, 'clf__scale_pos_weight': 30.736842105263158, 'clf__min_child_weight': 1, 'clf__max_depth': 9}
Best Score: 0.219278428798
Accuracy: 0.966850828729
AUC: 0.850038850039
F1 Measure: 0.4
Cohen Kappa: 0.383129792673
Precision: 0.444444444444
recall: 0.363636363636

Confusion Matrix:
[[346   5]
 [  7   4]]

As the confusion matrix shows my model can not identify bankrupted companies very well, which results in poor performance measures such as precision, recall, Cohen's kappa, F measure. Also, I tried  BlaggingClassifier, which is presented here. Really, it gives the following result:

Best Parameters: {'clf__n_estimators': 64}
Best Score: 0.133676613659
Accuracy: 0.809392265193
AUC: 0.819606319606
F1 Measure: 0.188235294118
Cohen Kappa: 0.142886555487
Precision: 0.108108108108
recall: 0.727272727273
Confusion Matrix: [[285  66]
 [  3   8]]

As it is shown, it predicts positive classes well but it does poor on negative class ( too many false positive).
Could you please let me know how it is possible to combine these two classifier results in order to obtain a better result? A simple way to combine two classifier is to use a convex linear combination for predicted probabilities: t * p1 + (1 - t) * p2, where 0 <= t <= 1 and p1, p2 are predictions of the two classifiers. Then , I should search for an optimal value of t over a grid but I don't know how to do it?

I read that anomaly detection for example,one class svm and Isolation Forest could be used for extremely unbalanced data set, so could you please let me know how to do it i.e., by an example code?
In general, I would appreciate if you could let me know how to deal with this issue.


texts:
CODE

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
# Importing the dataset
dataset = pd.read_csv('D:\\test.csv')
X = dataset.iloc[:, 0:2].values
y = dataset.iloc[:, 2].values
print (X)

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X_0 = LabelEncoder()
X[:, 0] = labelencoder_X_0.fit_transform(X[:, 0])
labelencoder_X_1 = LabelEncoder()
X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])
print (X)
onehotencoder = OneHotEncoder(categorical_features = [0,1])
X = onehotencoder.fit_transform(X).toarray()
X = X[:, 0:]
X = X[:, 1:]
print (X)

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
print (X_test)

import keras
from keras.models import Sequential
from keras.layers import Dense
import keras.utils

classifier = Sequential()
classifier.add(Dense(output_dim = 6, init = 'uniform', activation = 'relu', input_dim = 9))
classifier.add(Dense(output_dim = 6, init = 'uniform', activation = 'relu'))
# Adding the output layer
classifier.add(Dense(output_dim = 1, init = 'uniform', activation = 'sigmoid'))
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 100)

_, accuracy = classifier.evaluate(X, y)
print('Accuracy: %.2f' % (accuracy*100))

#predictions
dataset = pd.read_csv('D:\\test1.csv')
a = dataset.iloc[:, 0:2].values
labelencoder_a_0 = LabelEncoder()
a[:, 0] = labelencoder_a_0.fit_transform(a[:, 0])
labelencoder_a_1 = LabelEncoder()
a[:, 1] = labelencoder_a_1.fit_transform(a[:, 1])
onehotencoder = OneHotEncoder(categorical_features = [0,1])
a = onehotencoder.fit_transform(a).toarray()
a = a[:, 0:]
a = a[:, 1:]

predictions = classifier.predict_classes(a)
# summarize the first 2 cases
for i in range(2):
    print('%s => %d' % (a[i].tolist(), predictions[i]))

data after label encoders looks like

[[0 0]
 [0 1]
 [0 2]
 [0 3]
 [0 4]
 [0 5]
 [0 6]
 [0 7]
 [0 8]]

after one-hot looks like

[[1. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 1.]]

after [X_test = sc.transform(X_test)] looks like

[[ 0.          1.         -0.40824829 -0.40824829 -0.40824829 -0.40824829
  -0.40824829 -0.40824829 -0.40824829]
 [ 1.          0.         -0.40824829 -0.40824829 -0.40824829 -0.40824829
  -0.40824829 -0.40824829 -0.40824829]]

and X_train looks like

[[-0.40824829 -0.40824829 -0.40824829 -0.40824829  0.          2.44948974
   0.         -0.40824829 -0.40824829]
 [-0.40824829 -0.40824829 -0.40824829 -0.40824829  0.         -0.40824829
   0.         -0.40824829  2.44948974]
 [-0.40824829 -0.40824829 -0.40824829 -0.40824829  0.         -0.40824829
   0.          2.44948974 -0.40824829]
 [ 2.44948974 -0.40824829 -0.40824829 -0.40824829  0.         -0.40824829
   0.         -0.40824829 -0.40824829]
 [-0.40824829  2.44948974 -0.40824829 -0.40824829  0.         -0.40824829
   0.         -0.40824829 -0.40824829]
 [-0.40824829 -0.40824829 -0.40824829  2.44948974  0.         -0.40824829
   0.         -0.40824829 -0.40824829]
 [-0.40824829 -0.40824829  2.44948974 -0.40824829  0.         -0.40824829
   0.         -0.40824829 -0.40824829]]

It is trained successfully. But when i run prediction code
it gives error

Error when checking input: expected dense_205_input to have shape (9,) but got array with shape (6,)

variable (a) after one_hot looks like

[[1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 1.]]

I can't understand what is the problem with my prediction sample.


texts: Regression coefficient(s) when explanatory & response variables are time series

texts:
I have multiple VMs (Virtual Machines) on one physical server (hardware machine). I have CPU utilization information (every 5 minute) for all the virtual machines and the physical server. Now I want to understand how each VM is contributing to the overall load of the physical server. I am assuming that this is a time series problem. The CPU utilization of each VM as well the physical server are separate time series. One way to approach this problem is to apply linear regression considering CPU utilization of each VM as explanatory varaibles and CPU utilization of physical server as response variable.

  cpu_physical_server(t) = Function(cpu_vm_one(t) + cpu_vm_two(t) + cpu_vm_three(t) + .....)

As per my understanding, I can't directly apply regression modelling (lm() or glm()) as these are all time series (Link).

The question is how to calculate the regression coefficients when both explanatory and response variables are time series? What kind of correction(s) should I apply?


texts: How to train a text classifier for product search query to determine category

texts:
I am trying to train a product search query (e-commerce) classifier for deducing probable product categories from search query with a dataset of 700k queries with probable categories labelled
I tried first with multi class but then needed to change to multi-label as some of search queries in dataset had multiple probable categories. now issue with multi label is I'm getting very low accuracy of 30%.
For training I also trained a custom gensim word embedding on a dataset of 8 million products names and 3 million random search queries but results are more or less same.
The training set is in form of 30 categories (columns) with 0 or 1 labelled with search query 1 being true.
What would be the better way to solve this issue?


texts: Prediction approach on unique data or progressive data

texts:
In a employee attrition analysis with a table having rowwise data for a (employee like Id, name, Date_Join Date_Relieving Dept Role etc)

eID eName    Joining Releiving Dept Married Experience
123 John Doe 10Oct15 12Oct16   HR   No      12
234 Jen Doee 01jan16 -NA-      HR   No      11         (ie she is available)

I can run regression on this data to find the beta coefficients

eID eName    Joining Releiving Dept Married Experience
123 John Doe 10Oct15 12Oct16   HR   No      12
234 Jen Doee 01jan16 -NA-      HR   No      11

But I've seen other approach too..
where employee have multiple entries depending on their difference between joining date and current month or relieving month(say Employee A joined in Jan and Left in Dec so he'll have 12 entries updating corresponding columns like experience and marriage etc)

eID eName    Dept Married Experience
123 John Doe HR   No      0
123 John Doe HR   No      1
123 John Doe HR   Yes     2
123 John Doe HR   Yes     3

can someone tell what differentiate two approaches.. and what are the benefits of this second approach.


texts: When training NN, how do data loaders work on large datasets?

texts:
How do you normally organize large data-sets for easy loading when training Neural Networks? I have a largeish data-set which cannot fit into memory, it consists of 200,000 samples, with 10k samples being stored in grouped files. In total this is roughly 50GB of data.

I can separate the grouping to produce 200,000 individual files but I'm not sure if this is the correct course of action as the system will need to do many calls to the file system when training.

How do systems which train massive data-sets work? (For example image-net).


texts:
I don't understand how n-grams are language independent. I've read that by using character n-grams of a word than the word itself as dimensions of a vector space model, we can skip the language-dependent pre-processing such as stemming and stop word removal.

Can someone please provide reasoning for this?


texts: Associating processes with load on the system

texts:
I am trying to find out the processes on our linux servers that are responsible for load on the system.
We have an application server and an Oracle database server (both running Linux).

The application server runs processes which are mostly shell scripts, perl scripts and standard applications like Apache. Many of the processes open sessions in database.

I am sampling the load average on both servers once a minute using 'uptime' command.

I am also collecting the output of 'ps' command once a minute, to list the processes running.

Given the above information, what algorithms or machine learning techniques can I use to classify processes into 10 different buckets with 10 indicating the processes that cause heaviest load on the system and 1 the lightest load?


texts: Effect of NOT changing filter weights of CNN during backprop

texts: What is the effect of NOT changing filter weights of a CNN during backpropagation? I changed only the fully connected layer weights while training on the MNIST dataset and still achieved almost 99 percent accuracy.


texts: Tree-based algorithms and ordinal features

texts:
For tree-based methods (e.g., DT, Random Forest, Gradient boosting, etc.), does the conversion interval of an ordinal feature to continuous matter matters? (I can see why it matters for linear model, but I am not clear for tree-based methods)
For example:
Is there a difference between converting an ordinal feature from [‘Low’,’Medium’,’High’] to [1,2,3] compared to converting it to [1,99, 876]


texts: Using different timesteps for training data and target value

texts:
I would like to know whether it's wrong; when working with time series data; to use daily prices as features and the price after 3 days as target.

Is this correct or should I use the next-day price as target and after training; predict 3 times; each time for one more day ahead(using the predicted value as a new feature)

Will these 2 approaches give similar results?


texts: SVM model classifying into one class only, after standardization

texts:
I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are microarray expression values from two different microarray platforms performed by two different research groups and so their value ranges are very different. I am new to SVM and R, and so at first, I did a Z-score standardization on my Training set and then I did a Z-score standardization on my Test set separately. i.e.

(x_Train - mean_Train) / (stdev_Train)

(x_Test - mean_Test) / (stdev_Test)

As I explored more forum posts on this topic, I see that it is suggested to use the parameters from Training set on my Test set

(x_Test - mean_Train) / (stdev_Train)

So that my SVM model is more generalizable and can be applied to new samples.

Here is the problem,

When I standardized the sets separately, my best model (after 10-fold cross validation using the tune function) had around 75% accuracy classifying the test set. However, after standardizing as suggested with my Training parameters the accuracy of my model to classify the Test set dropped drastically to 25% using the same features.

I tried using different combinations of features to see if it makes a difference and I saw that no matter what I had changed, all samples from the Test set were classified into one class (all in normal, and zero in tumor), while my previous model with separate standardization classified samples into both classes.

Here is an example of one of the features

Training set (raw)
         Feature 1
Sample 1 25.82977
Sample 2 42.62437
Sample 3 28.91158
Sample 4 53.5708
Sample 5 30.92296
Sample 6 99.16994
Sample 7 40.75973

Test set (raw)
         Feature 1
Sample 1 2.885865028
Sample 2 2.572860413
Sample 3 2.809136715
Sample 4 2.259630716
Sample 5 2.797155715
Sample 6 2.439700763
Sample 7 2.197087754

Training set (standardized)
         Feature 1
Sample 1 -0.795137358
Sample 2 -0.132081654
Sample 3 -0.673466601
Sample 4  0.300086594
Sample 5 -0.594056732
Sample 6  2.100353934
Sample 7 -0.205698184

Test set (standardized)
         Feature 1
Sample 1 -1.700969415
Sample 2 -1.713326928
Sample 3 -1.703998671
Sample 4 -1.725693328
Sample 5 -1.704471684
Sample 6 -1.71858411
Sample 7 -1.728162542

I know SVM uses a threshold to classify samples into either normal or tumor. From what I showed above, their ranges are completely different, is that what is affecting its decision?

What is the problem here and how can I tackle it, thank you in advance. I am so lost and need every bit of advice!

*Edit: As you can see the values of Training set have a completely different range from the values of the Test set, can I still use the mean and standard deviation from my Training set to scale the corresponding features in the Test set?


texts: validation_curve differs from cross_val_score?

texts:
I'm trying to see how well a decision tree classifier performs on my input. For this I'm trying to use the validation and learning curves and SKLearn's cross-validation methods. However, they differ, and I don't know what to make of it.

The validation curve shows up as follows:

Based on varying the maximum depth parameter, I'm getting worse and worse cross-val scores. However, when I try the cross_val_score, I get ~72% accuracy reliably:

While I was using the default tree depth for clf here, it still puzzles me how the validation curve never reaches even 0.6, but the cross-val scores are all above 0.7. What does this mean? Why is there a discrepancy?

Code for reference below.

For the Validation curve:

import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import validation_curve

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()

param_range = np.arange(1, 50, 5)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", param_range=param_range,
    cv=None, scoring="accuracy", n_jobs=1)
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)

plt.title("Validation Curve with Decision Tree Classifier")
plt.xlabel("max_depth")
#plt.xticks(param_range)
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
lw = 2
plt.plot(param_range, train_scores_mean, label="Training score",
             color="darkorange", lw=lw)
plt.fill_between(param_range, train_scores_mean - train_scores_std,
                 train_scores_mean + train_scores_std, alpha=0.2,
                 color="darkorange", lw=lw)
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="navy", lw=lw)
plt.fill_between(param_range, test_scores_mean - test_scores_std,
                 test_scores_mean + test_scores_std, alpha=0.2,
                 color="navy", lw=lw)
plt.legend(loc="best")
plt.show()

For the cross-val scores:

clf = DecisionTreeClassifier(class_weight='balanced')
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
clf.score(X_test, y_test)

UPDATE
A comment has been asked about shuffling. When I shuffle the data by

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

I get:

Which makes even less sense to me. What does this mean?


texts: How to choose best model for Regression?

texts:
I'm building a model to predict the flight delay. My dataset contains the following columns:

FL_DATE (contains months(1-12)), OP_CARRIER (One hot encoded data of Carrier names), ORIGIN(One hot encoded data of Origin Airport), Dest(one-hot encoded data of Dest Airport), CRS_DEP_TIME(Intended time of departure ex: 1015), DEP_TIME(Actual time of departure ex: 1017),DEP_DELAY(the difference between crs-dep ex: -2), ARR_DELAY(arrival delay ex: -2)

My target variable is ARR_DELAY. After checking my data, I have decided it is a regression problem. However, I'm not sure what method do I need to use for selecting the appropriate columns. On the other hand, I was plotting each column with ARR_DELAY to check their relation and got something like this: FL_TIME vs ARR_DELAY

In such scenarios, if I have to build a model for such data which regression technique should I use?

PS: I'm new to Machine Learning. Please correct me If I'm heading in the wrong direction


texts: Decision Stumps with same value leaf nodes

texts: I'm doing some ADA boosting with Decision stumps and in inducing a binary classifying decision stump, i'm finding both leaf nodes to have a positive value. Can this be the case? Is this possible?


texts:
We have ~500 biomedical documents each of some 1-2 MB. We want to use a non query-based method to rank the documents in order of their unique content score. I'm calling it "unique content" because our researchers want to know from which document to start reading. All the documents are of the same topic, in the biomedical world we know that there is always a lot of content overlap. So all we want to do is to arrange the documents in the order of their unique content.

Most Information Retrieval literature suggest query-based ranking which does not fit our need.


texts:
I'm new here - I hope this kind of question is legit.

I have a background in mathematics - my field is probability theory, where I am quite strong (top 2% of University).

I have little experience in programming but I guess I get things done once I know what is out there (i.e. what program languages can do for me).
I want to learn more about data science and dive into the field particular when it comes to programming. I'd also like to get going on manageable yet interesting data science related tasks (building some "usefull" statistics skill might be desirable, too).

I know kaggle.com are there other places I could look into?
Any projects I might take on?

Thanks a lot. :)


texts: Determine the most important documents for supervised learning

texts:
I have somewhat of a general/high level question.

Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class classification). My supervised model performs fairly well on testing data but what I'm trying to do now is find a way to sort out which documents from the production data (not human labeled) if added to the training set would improve accuracy the most on that particular production data set.

The idea is once the initial predictions are made on production data, another algorithm would be run to determine the n most important documents (say 500) that if manually labeled by a human and added to the training set would improve accuracy the most, assuming the model does a 2nd round of prediction after the n documents have been added to the training set.

So the process would look like this:

Initial predictions on production data are made
Algorithm runs to determine the most important records to increase accuracy of the model (based on features most likely).
Human reviews these most important records and adds them to the training set
A 2nd round of prediction takes place, hopefully with a better accuracy metric

I'm thinking of using something like K-Means after doing some dimensionality reduction since it's NLP problem with lots of features.

Does anyone have any experience or suggestions in regard to this topic? Am I on the right track?


texts: Learning parameters when loss is a piecewise function

texts:
I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the loss has another value $L_2$; etc. The loss function resembles a piecewise function.
A concrete, simplified example of this problem is perhaps something like object classification. I have a set of objects, and their distances to a category $C$ that I want to classify these objects into. The distances are $[d_1, d_2, \dots, d_K]$. Assume without loss of generality that $d_1 \leq d_2 \leq \dots \leq d_K$. I want to learn a threshold $T$ for these objects that says, if the distance is near enough, then they belong to the category $C$; otherwise, they are not members of the category. For example, if $d_3 \leq T \leq d_4$, then objects $1, 2$ and $3$ (with distances $d_1, d_2, d_3$) belong to $C$.
What learning techniques I may use to learn weights of the network? Any help will be greatly appreciated.
I will next combine the above network with other differentiable learning components, so ideally it would be good if the approach is compatible with gradient descent.


texts: How to deal with categorical feature of very high cardinality?

texts:
I would like to train a binary classifier on feature vectors.
One of the features is categorical feature with string, it is the zip codes of a country.

Typically, there is thousands of zip codes, and in my case they are strings.
How can convert this feature into numerical?

I do not think that using one-hot-encoding is good as a solution for my case.
Am I right by saying that? If yes, what would be a suitable solution?


texts:
I'm learning some neural network by using tensorflow. I created a dataset of watermarked and not watermarked image, and I'm trying to create a model do classify both.
I started from InceptionResNetV2 but retraining all the layers ( otherwise results was poor )
After some test I managed to achieve this:

With this model.
Since it seems to me that was overfitting, I decided to do some augmentation in the dataset, by basically add some random shift to the images of the training set:
 ImageDataGenerator(rescale=.1/255,width_shift_range=0.1,height_shift_range=0.1)

But I obtained something really weird:

With the code you can find here.
As you can see, validation test is sometime promising, and sometimes drops to complete random, what is this symptom of?


texts: How to create a training set for sequence labelling

texts: I have a bunch of unlabeled text data that I would like to hand label. Are there any tools out there that you can use to produce your own labels? Ideally it would show a sentence at a time and give me the option of choosing a label for each word.


texts: How to choose between Tensorflow and Pytorch?

texts:
Recently I've been working on a pretty vanilla ANN model in Python with sklearn (and its preprocessing pipeline), mostly in jupyterhub notebooks if that matters.
I am considering changing the framework I work with as I need some more tools (drop out at least). but I am not so familiar with Python frameworks. The 2020 state of AI report point out (p. 14) that the two main frameworks are PyTorch and Tensorflow.
What are the practical differences between those two frameworks ?


texts: Target encoding before split on skewed data

texts:
Hi My data is distributed like follow:

And I only have categorical variables on many many levels.
As I need to make a regression task I thought about doing leave one out encoding on my categories. However, while my model has a very good performance on my validation (0.98 R2) on my test is kind of terrible (0.5 R2).
My colleague did the encoding before the split, reaching an R2 of 0.97 on test.
His reason is that this to cover unseen new categories and to cover all the possible targets during the encoding.
Isn't this a bad practice? Or for this highly skewed data is a possible behavior to adopt?
Thanks


texts: Better approach to assign values to determine potential fake sentences

texts:
I am trying to assign different values for each sentences based on information about the presence of hashtags, upper case letters/words (e.g. HATE) and some others.
I created a data frame which includes some binary values (1 or 0):
Sentence           Upper case   Hashtags

I HATE migrants       1             0
I like cooking        0             0
#trump said he is ok  0             1
#blacklives SUPPORT   1             1

I would like to assign a value based on the binary values above, if they are satisfied or not, for example:
- if Upper case = 1 and Hashtags = 1 then assign -10;
- if Upper case = 1 and Hashtags = 0 then assign -5;
- if Upper case = 0 and Hashtags = 1 then assign -5;
- if Upper case = 0 and Hashtags = 0 then assign 0;

This would be ok for a small number of requests and combinations, but with three variables to check, it would be a greater number of combination to consider manually!
Do you know if there is a way to take into account all these in an easy (and feasible) way?
Someone told me about using regression, but I have never used it before for similar task. The context is about fake tweets.


texts: Fuzzy rule based system: Should rules contain all inputs and outputs?

texts:
I am trying to design an FRBS using Matlab fuzzy tool box. The fuzzy system will be used to predict player's type based on the inputs and a set of rules defined by experts.
I have 6 inputs and 4 outputs (types of players). The given rules do not concern all inputs. Specific inputs are used for each player type.

Is it imperative to include all inputs and outputs in a rule?
Also is there a min/max of rules that an FRBS should include or does it depend on the experts expertise?

Thank you for your help.


texts: If I train a model using only my testing data, will the accuracy be 100%?

texts:
While I understand that if I train a model using the same data that I test on, then I'll certainly be overfitting, but will the accuracy of that model always be 100%?

In other words, is one method of testing whether I've created my model correctly, to test the model by training it on the testing data?


texts: receive value error decision tree classifier after one-hot encoding

texts:
I am trying to build a decision tree model. After one-hot encoding, it seems somehow the data still has a problem. When I run the following code, I receive this error:

ValueError: Number of labels=172 does not match number of samples=540

#Code:
import pandas as pd
import numpy as np

df = pd.read_csv("https://library.startlearninglabs.uw.edu/DATASCI420/Datasets/Bank%20Data.csv", sep=",")
df.columns=['age', 'sex', 'region', 'income', 'married', 'children', 'car',
       'save_act', 'current_act', 'mortgage', 'pep']
df.info()

(nrows, ncols) = df.shape
colnames = list(df.columns.values)
string_encoding = {}
df_encoded = df.copy()
for i in range(ncols):
    levels = list(set(df.iloc[:, i]))
    num_levels = len(levels)
    string_encoding_i = dict(zip(levels, range(num_levels)))
    string_encoding[colnames[i]] = string_encoding_i
    for j in range(nrows):
        df_encoded.iloc[j, i] = string_encoding_i[df.iloc[j, i]]

print(string_encoding)
print(df_encoded.head())

# One Hot Encoding Categorial Variables
from sklearn import preprocessing
enc = preprocessing.OneHotEncoder()

non_categorial_features = ['age',
                          'income',
                          'children',
                          'pep']

for categorical_feature in list(df.columns):
    if categorical_feature not in non_categorial_features:
        df[categorical_feature] = df[categorical_feature].astype('category')

df_with_dummies = pd.get_dummies(df, sparse=True)

df = pd.concat([df, df_with_dummies], axis=1)

df.head(5)

df = df.drop(['sex', 'region', 'married', 'car',
       'save_act', 'current_act', 'mortgage', 'pep_NO', 'pep_YES'], axis=1)

df.head()
df.info()

from sklearn import tree
import numpy as np
from sklearn.model_selection import train_test_split

# prepare for decision tree
np.random.seed(101)

title_names =['age', 'income', 'children', 'counts', 'age', 'income',
       'children', 'sex_FEMALE', 'sex_MALE', 'region_INNER_CITY',
       'region_RURAL', 'region_SUBURBAN', 'region_TOWN', 'married_NO',
       'married_YES', 'car_NO', 'car_YES', 'save_act_NO', 'save_act_YES',
       'current_act_NO', 'current_act_YES', 'mortgage_NO', 'mortgage_YES', 'pep']

df = df[title_names]

X = df.iloc[:,0:23]
Y = df.iloc[:, 23]
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size = 0.1, random_state = 99)

# decision tree
from sklearn.tree import DecisionTreeClassifier

# Use entropy = no limit on samples for split
model_ent = DecisionTreeClassifier(criterion='entropy').fit(X_train, y_train)
y_ent_pred = model_ent.predict(X_test)

# Use information gain (default) limit min_samples to 4
model_gini = DecisionTreeClassifier(min_samples_leaf=5).fit(X_train, y_train)
y_gini_pred = model_gini.predict(X_test)
```


texts:
I trying to create a model for determining whether a questioned hand written signature matches known signature samples, and predict if the signature is genuine or forgeries.

I'm guessing I'll have to use a CNN and probably some variation of PCA.

How would you tackle the "signature verification" problem?
Is there an already trained models for that? or datasets?

All I could find are research articles, but:

no GitHub projects.
no AWS/Azure/Gcloud service.
no Kaggle completions or datasets.

Maybe I not searching for the right thing!

Any dataset of signatures I could use, and ideas on how to approach that kind of problem would be very helpful. tnx :)


texts: Why such a big difference in number between training error and validation error?

texts:
Question

Why such a big difference between my 'Train loss' and 'Validation loss' as shown in the picture below? Is it a signal that my codes are wrong and my trained network is wrong as well?

Some of my codes are as follows:

DATA_SPLIT_PCT = 0.2

timesteps =  5
n_features =  20

epochs = 100
batch = 32
lr = 0.0001

lstm_autoencoder = Sequential([
    # Encoder
    LSTM(8, activation='relu', input_shape=(timesteps, n_features), return_sequences=True),
    LSTM(4, activation='relu', return_sequences=False),
    RepeatVector(timesteps),

    # Decoder
    LSTM(4, activation='relu', return_sequences=True),
    LSTM(8, activation='relu', return_sequences=True)
    TimeDistributed(Dense(n_features)),
])

adam = optimizers.Adam(lr)
lstm_autoencoder.compile(loss='mse', optimizer=adam)

for stock in stock_list:  # 500 stocks in stock_list
    lstm_autoencoder_history = lstm_autoencoder.fit(X_train_dict[ticker], X_train_dict[ticker],
                                                    epochs=epochs,
                                                    batch_size=batch,
                                                    validation_data=(X_valid_dict[ticker], X_valid_dict[ticker]),
                                                    verbose=False).history

plt.plot(lstm_autoencoder_history['loss'], linewidth=2, label='Train')
plt.plot(lstm_autoencoder_history['val_loss'], linewidth=2, label='Valid')
plt.show()

I used the for loop to feed my data into lstm_autoencoder network. In the dictionary variable stock_list, there are 500 stock names such as 'AAPL'.
I plotted lstm_autoencoder_history['loss'] and lstm_autoencoder_history['val_loss'] and it is weird because usually validation loss is higher than train loss.
I am curious to know why my plot has smaller amount of validation loss. For your information, I used Keras as my deep learning framework. And since I used Keras, I thought this library would handle the different proportion of training set size and validation set size by averaging the errors.


texts: How to infer which sequence of events are more likely to result in an event of interest?

texts:
Background
I have a sequence/series of events. Some sequences will end up with an event of interest $y$, while others won't.
$$S_1 = [a, b, c, d, ..., y]$$
$$S_2 = [a, b, b, e, ..., a]$$
$$...$$
$$S_n = [a, a, f, m, ..., y]$$
Sequences can be of varying lengths; all sequences are independent of one another; events are linearly spaced within a sequence. Within a sequence, memory of previous events is important (i.e., it's not just the previous event that is important), and the order of events is important, too.
My use-case is e-commerce (i.e., website navigation/browsing, where my event of interest is a transaction being made at the end of the customer journey, $y$). I guess this could generalise to many fields: words in a sentence, political events, component failure, personal development, etc.
I think I'm looking for an Association Rules Mining-type approach, but where the order in which items are added to the basket is important. If that makes sense.
Solution 1
Is there some method whereby I can find which sequence of consecutive events are more likely to end up with $y$ somewhere down the line? The event of interest, $y$ doesn't have to come immediately after the chain of events. For example, if $[a, b]$ is important, then I don't mind if the sequence is $[..., a, b, y]$ or $[a, b, ..., y]$, etc.
Solution 2
To make this even better, is there a method where the important sequence of events don't even have to be 'touching' each other? For example, the $[a, b, y]$ sequence might be equivalent to $[a, ..., b, y]$ or $[a, ..., b, ..., y]$, etc.


texts: Positive semidefinite kernel matrix from Gower distance

texts: I have a dataframe with continuous and categorical variables and I want to obtain a kernel matrix for classification. The kernel matrix must be symmetric and positive semidefinite, so that no eigenvalue is negative. I started with Gower distance matrix for mixed data, which is not positive semidefinite. I tried to transform the Gower distance matrix into a positive semidefinite and symmetric kernel with the function D2Ksof MiRVpackage in R, with no success. I tried also to apply the approach of page 799 in Zhao, Ni et al. “Testing in Microbiome-Profiling Studies with MiRKAT, the Microbiome Regression-Based Kernel Association Test” American journal of human genetics vol. 96,5 (2015): 797-807. with no success, as well. I always obtain a indefinite kernel matrix with positive and negative eigenvalues. Any suggestion?


texts: Bad Input Shape -- How to interpret and Diagnose; Also side ML question

texts:
I apologize I am a ML novice, but I am trying to learn. I am making a classifier based on this dataset to predict mental health disorders based on features. I wanted to run a very simple NB classifer model but I keep getting a bad input shape error (I want to feed in features such as age, ethnicity and gender to yield potential diagnoses). Unfortunately, I am having trouble diagnosing where my error is coming from and troubleshooting. Any guidance? (ignore the multiple input stuff at the top; I was trying different things but I am assuming that there is a problem with how I am inputting the data parameters)
Namely, for these labels (diagnoses) I want an output that will show the presence/lack thereof each [0 or 1] based on the features (which are numeric)
Feature Names
['YEAR', 'AGE', 'EDUC', 'ETHNIC', 'RACE']
Values
[ 9, -9,  4 , 2]
Labels:
['ADHDFLG', 'CONDUCTFLG', 'DELIRDEMFLG', 'BIPOLARFLG', 'DEPRESSFLG', 'ODDFLG', 'PDDFLG', 'PERSONFLG', 'SCHIZOFLG', 'ALCSUBFLG']
Corresponding Label values
[0, 1, 0, 0, 0, 1, 0, 0, 0, 0]
Also, side question -- does anyone have any recommendations for other Maching Learning tasks I can try with this? I am doing this for a class and am trying to push myself to learn new topics.
Thanks in advance!
import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder
import random
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.naive_bayes import GaussianNB

import scipy

from sklearn.model_selection import train_test_split

df = pd.read_csv("https://csprojectdatavisualizationsample50k.s3.us-east-2.amazonaws.com/sample_df.csv")
df_columns = df.columns
df_feature_names = (df_columns[1:6]).to_list()
df_features = df.iloc[:,2:6].values
df_label_names = (df_columns[26:36]).to_list()
df_labels = df.iloc[:, 26:36].values
#Input
print(df_label_names)

# Split our data
train, test, train_labels, test_labels = train_test_split(df_features,
                                                          df_labels,
                                                          test_size=0.50,
                                                          random_state=42)

print(train.shape)
print(test.shape)

# Initialize our classifier
gnb = GaussianNB()

# Train our classifier
model = gnb.fit(train, train_labels)

# Make predictions
preds = gnb.predict(test)
print(preds)


texts: Combine AdaBoost and Support Vector Regression?

texts: I have read several papers about using SVM instead of decision tree in AdaBoost, but I haven't seen any papers about using support vector regression (SVR) in AdaBoost. However, if using support vector regression in AdaBoost, I have to weaken SVR. So how do I weaken SVR when combining it with AdaBoost?


texts: How to do multitask learning using Caffe?

texts:
I wonder how to do multitask learning using Caffe. Should I simply use the output layer SigmoidCrossEntropyLoss or EuclideanLoss, and define more than one outputs?

E.g. is the following architecture valid (3 outputs, i.e. 3 tasks concurrently learnt)?

Corresponding prototxt file:

name: "IrisNet"
layer {
  name: "iris"
  type: "HDF5Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  hdf5_data_param {
    source: "iris_train_data.txt"
    batch_size: 1

  }
}

layer {
  name: "iris"
  type: "HDF5Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  hdf5_data_param {
    source: "iris_test_data.txt"
    batch_size: 1

  }
}

layer {
  name: "ip1"
  type: "InnerProduct"
  bottom: "data"
  top: "ip1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 50
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

layer {
  name: "ip2"
  type: "InnerProduct"
  bottom: "ip1"
  top: "ip2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 3
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

layer {
  name: "loss"
  type: "SigmoidCrossEntropyLoss"
  # type: "EuclideanLoss"
  # type: "HingeLoss"
  bottom: "ip2"
  bottom: "label"
  top: "loss"
}


texts: How to Deploy your trained ML model in client VM without they getting access to code

texts:
I am new to deployment and have a basic doubt about deploying my ML code on client's vm.

So I have built a python project which collects data from client site, processing, predicts and displays the result in dashboard. I have to use client VMs for deployment. Is there a way for me to hide the code or do something to it so that client cannot see my code and reuse my code for other purposes.

Might sound trivial but really stuck at this. Thanks in advance.


texts: Load numpy data from directory to keras image generator

texts:
I have two folders of hyperspectral data with five channels which are converted to numpy array. Each folder depicts the respective label.
Example :
dataset
----good_data
      ----good_image_01.npy
      ----good_image_02.npy
----bad_data
    ----bad_image_01.npy
    ----bad_image_02.npy

I have used keras image generator to feed the data to input pipeline previously with png images.
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)

But I am not sure how to achieve the same with numpy dataset. can anyone help?


texts: Percentage of missing values so that we can't perform imputation

texts:
(This is not a question on ways to handle missing data)

I have a dataset with around $80$ or so features and around $100000$ rows. Several of those features have missing (NULL) values for a "large proportion" (more than 20%) of the records.

Is there a general way to decide the % of the total rows for which a feature has to be missing, so that we can consider it useless and remove it from our analysis? Or does this depend on the context? Indirectly, I want to know the % of missing values for a feature so that imputation is meaningless.


texts:
I was using PIL library to open an image and then convert it into array later on for DL operations. However I found out that image opened was blurry as opposed to the original image. Why is the image coming out the way it is? and what can be done to fix it? If a possible solution is present in OpenCV that'd be welcome as well.

I tried to use OpenCV as well but I was getting an Assertion Error i.e

error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/resize.cpp:4044: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

Code:

from PIL import Image
import numpy as np
import pandas as pd
import os
import cv2
import keras
from keras.models import Sequential
from keras.layers import Conv2D,MaxPooling2D,Dense,Flatten,Dropout
import matplotlib.pyplot as plt

image=Image.open("../content/cell_images/Parasitized/"+i)
size_image = image.resize((50, 50))
print("image")
plot(image,count)
print("resize image")
plot(size_image,count)

Here is the code for plotting:

def plot(img, count):
    w=10
    h=10
    fig=plt.figure(figsize=(8, 8))
    img = np.random.randint(10, size=(h,w))
    fig.add_subplot(rows, columns, count)
    plt.imshow(img)
    plt.show()


texts: CNN model overfitting to a Color, not learning edges

texts:
I've been working on this binary classifier which predicts if a certain type of battery is in the live webcam feed or not. The problem I am facing is that my CNN model predicts any 'orange object' in the frame as a battery. It can't tell the difference between an orange pair of scissors and a cuboidal orange battery. Also, sometimes it detects a human as a battery too.

Battery Image:
https://robu.in/product/orange-5200mah-3s-40c80c-lithium-polymer-battery-pack-lipo/

I've tried several CNNs like,
A) Classic CNN with ~10 layers (3x(Conv2D, MaxPooling), 2xDense along with a few BatchNormalization and Dropout layers) --Validation Accuracy ~92%
B) Concatenation of two NN (1. using canny edge filter on my complete dataset 2. With the colored images)  --Validation accuracy <67.66%
C) MobileNet on Colored Images
D) ResNet
E) AlexNet
etc...

I've tried Data Augmentation(Using Grayscale images, Adding/Subtracting Pixel values, etc.)

All images are of size 150×150 Total image dataset of ~200 images

How should I tackle this problem of color overfitting?


texts:
The cost function given as $\hat{\beta} = (Y - \beta X)^T (Y-\beta X)$ is used to evaluate the weights $\beta$. Here $X$ is the data and $Y$ is the output. On taking the derivative, we get the estimates of the weights. This is a Least Squares formulation.

1) Can Least Squares (LS) be used when the observation (outputs) $y_i$, $i=1,2,..,N$ number of examples are categorical? I don't quite get the picture how classification problems using LS works in terms of derivative for categorical cases.

2) Can LS be used when the data $X$ is a one-hot encoding? Would the formulation and derivative be the same?


texts: CNN can't predict images outside the dataset

texts:
I am using celeba dataset to train my CNN face landmark detection model.
Here is my model
class LandmarkModel:
    def __init__(self,inp_shape):
        self.model = models.Sequential()
        self.model.add(layers.Conv2D(16, (3, 3), activation='relu', input_shape=inp_shape))#l1
        self.model.add(layers.Conv2D(32,(3, 3), activation='relu'))
        self.model.add(layers.MaxPooling2D((2, 2)))
        self.model.add(layers.Conv2D(64,(3, 3), activation='relu'))
        self.model.add(layers.Flatten())
        self.model.add(layers.Dense(512))
        self.model.add(layers.Dense(10))

    def getModel(self):
        return self.model

I have trained my model for around 5k-6k images with loss of 0.1.
When I use image from dataset that is outside of training sample I get correct prediction.
But when I use my own clicked images predictions are completely off. I have clicked photos exactly like in dataset.
I have also tried with downloaded celeb images still wrong predictions.
What is the reason of this behavior?


texts:
Given that I have a machine learning model.

I evaluated the model over several labeled datasets and acquired the accuracy (or any other metrics) for these datasets.

Now I receive a new dataset without labels. I run the model and got the prediction.

Is there any way to estimate the accuracy of my model on the new dataset?

I was thinking of measuring the similarity of the datasets (in the feature space), then based on the similarity to predict the accuracy of the new dataset. For instance, if the new dataset is not far from the old dataset, the accuracies might be similar.


texts: Simple Binary Classification Example in Python

texts:
I'm not sure the correct place to ask, but I'm trying to develop a simple function/algorithm that outputs a predicted number from a sequence of numbers (I have a background in Python, but little to no experience working with data prediction). More specifically, when there is only binary choice. I looked around online, but I'm really lost and couldn't quite find something discussing it:

Suppose there's a bunch of 1s and 0s in sequence, like:

0100010010111011010010001011010

Suppose there is some complex math pattern that is determining the sequence, even though it is unbeknownst to us observers. However we know for certain that this data is not random, and contains no anomalies.

What sort of function can be used to predict if the next digit is a 1 or 0?

For example:

01010101 -> the next digit predicted should be 0.

001100110011 -> the next digit predicted should be 0.

Is there some simple Python function (using numpy, pandas, etc.) that can predict the next 1 or 0, no matter the pattern?


texts: Why are autoencoders for dimension reduction symmetrical?

texts:
I'm not an expert in autoencoders or neural networks by any means, so forgive me if this is a silly question.

For the purpose of dimension reduction or visualizing clusters in high dimensional data, we can use an autoencoder to create a (lossy) 2 dimensional representation by inspecting the output of the network layer with 2 nodes. For example, with the following architecture, we would inspect the output of the third layer

$[X] \rightarrow N_1=100 \rightarrow N_2=25 \rightarrow (N_3=2) \rightarrow N_4=25 \rightarrow N_5=100 \rightarrow [X]$

where $X$ is the input data and $N_l$ is the number of nodes in the $l$th layer.

Now, my question is, why do we want a symmetrical architecture? Doesn't a mirror of the deep 'compression' phase mean we might have a similarly complex 'decompression' phase resulting in a 2 node output which is not forced to be very intuitive? In other words, wouldn't having a simpler decoding phase result in the output of the layer with 2 nodes necessarily being simpler too?

My thinking here is that the less complex the decompression phase, the simpler (more linear?) the 2D representation has to be. A more complex decompression phase would allow a more complex 2D representation.


texts: Determining common columns among many datasets without having to manually inspect them

texts: I have a bunch of datasets (10+) and I want to be able to join/ relate them to one another through columns that may contain similar values. For example, date and location are common columns that could exist in many of them. How could I detect those similar columns without having to manually inspect all of my datasets?


texts: How to reduce the resolution of a Image(276*276 --> 48*48) without affecting the features in it

texts: I have an image with resolution of $276*276$. I've created a Convolution Neural Network which accepts $48*48$ images. So, I want to resize that $276*276$ image to $48*48$ without reducing any features in the data. Can anyone help me solve this problem so that I will send the resized images as the input to my CNN model?


texts:
I have an SQL DB which has millions of records(10 years of data) and wanted to implement some ML Models. Most of the courses/tutorials explains using a local file or downloaded CSV. But in my case I can't download all the records, but I need to load the data incrementally and train my model.How do i proceed about this?
And also if i'm wrong about incremental loading of data, how do companies load data for their ML models?.
I use Python language and am flexible with using libraries (scikit, tensorflow etc.)
Thank you.


texts: How does Q-Learning deal with mixed strategies?

texts: I'm trying to understand how Q-learning deals with games where the optimal policy is a mixed strategy. The Bellman equation says that you should choose $max_a(Q(s,a))$ but this implies a single unique action for each $s$. Is Q-learning just not appropriate if you believe that the problem has a mixed strategy?


texts: Suppose you have a dataset with m = 50 examples and n = 15 features for each example. You want to use multivariate linear regression to fit the parameters theta to our data. Should you prefer gradient descent or the normal equation and why?


texts: Synthetic time series generation according to some distribution

texts: I'm trying to develop a change detection model that uses sliding windows. Given a time series with some features I've a sliding widows that analyses that time period and compares with a successive time window. If the distribution of data has changed than it is notified. Anyway, I don't have enough real data about my problem, so I need to generate some artificial time series but with different distributions, no matter what distributions, the important thing is that they are different, so that I can test when my model detects change in the data. How can I generate those data?


texts:
when we are scaling the data i needed some clarification. so for preventing data leakage we split the train and test sets and then perform the scaling on them separately, correct?
so while scaling or label encoding the data in the train and test datasets,

how do we ensure that the scaling on test set is according to the train set because fit_transform on the train and test sets scale the features differently. so do we just fit on the train set first and transform the train and test set later?
can we save the scaler so that the new data scaling is done as per the scaler used during training
the same goes for label encoders and binarizer

i mean what is the industry standard of feature transformation?


texts: Word embedding vectors for keyphrase extraction

texts:
I am just interested to know if we can use the generated word embedding vectors to extract keywords from a document or not.

If yes, how?


texts:
It seems as though most languages have some number of scientific computing libraries available.

Python has Scipy
Rust has SciRust
C++ has several including ViennaCL and Armadillo
Java has Java Numerics and Colt as well as several other

Not to mention languages like R and Julia designed explicitly for scientific computing.

With so many options how do you choose the best language for a task? Additionally which languages will be the most performant? Python and R seem to have the most traction in the space, but logically a compiled language seems like it would be a better choice. And will anything ever outperform Fortran? Additionally compiled languages tend to have GPU acceleration, while interpreted languages like R and Python don't. What should I take into account when choosing a language, and which languages provide the best balance of utility and performance? Also are there any languages with significant scientific computing resources that I've missed?