texts: Multiple sizes and types of Inputs for a NN

texts:
I have multiple inputs for a Neural Network and a regression problem.
The first input is a Adjacency matrix, the two other are one hot encodings(or binary). I am using the Flatten layer for each of the inputs and then the Concatenate layer from Keras to combine these inputs and then I use simple neural network to predict y. However my NN is stuck at a local minima after a while and doesn't provide correct estimates.
Do you maybe have any suggestions how can I improve this type of model?
Should I maybe provide something other then Flatten to the Adjacency Matrix?
Should I use other layer add, multiply or similar instead of concatenate layer


texts:
I am doing experiments on https://www.physionet.org/challenge/2017/sources/ submission.

I like one of the submission code, which use Xgboost to train the classifier. Training data is in .mat file which I converted to CSV file for training.

In the below code, I have a pre-trained model xgb.bin, with which I can test any input signal. But I want to train the model using different data and create my own training model.

Here, is the code which predicts the class name for a given input ECG file:

def predict(data):
    #data = io.loadmat(path)['val'][0]
    from numpy import genfromtxt
    data = genfromtxt('testdata/val.csv', delimiter=',')

    features_noise = np.zeros((5, ))

    snr, rr_num, var, fr, fr2 = find_noise_features(data)
    features_noise[0] = snr
    features_noise[1] = rr_num
    features_noise[2] = var
    features_noise[3] = fr
    features_noise[4] = fr2
    features = extract_basic_features(data, 30000)
    features = np.hstack((features, features_noise.reshape(1, -1)))

    mean_ = np.array([15.96300284066109753667, 0.00412371298595770857, 38811.34497233365254942328,
                      0.48050717744965593115, 0.14397582347542958736])
    scale_ = np.array([4.22917401559752281770, 0.00093664880988427878, 62350.76443798459513345733,
                       0.15396567666240373873, 0.07085474966801086349])
    features_noise -= mean_
    features_noise /= scale_

    prediction = 0
    if features_noise[0] < -2.9:
        prediction = 3
    if features_noise[2] > 6.0:
        prediction = 3
    if features_noise[3] > 3.0:
        prediction = 3
    if features_noise[4] < -2.0:
        prediction = 3

    bst = xgb.Booster({'nthread': 4})
    bst.load_model("xgb.bin")

    dfeatures = xgb.DMatrix(features)
    prediction_prob = bst.predict(dfeatures)
    prediction = np.argmax(prediction_prob)

    return prediction

def run(data):
    prediction = predict(data)
    print(prediction)

I could create dfeatures for all training data which is in the CSV file. (I am reading each CSV individually and calculating feature. Is it correct?)

Now once I have defaulters for all training ECG files, I want to create a xgboost model. But I don’t have any clue for that. Any suggestion highly appreciated.


texts:
What is Conjugate Gradient Descent of Neural Network? How is it different from Gradient Descent technique?

I came across a resource, but was unable to understand the difference between the two methods. It has mentioned in the procedure that:

  the next search direction is determined so that it is conjugate to previous search directions.

What does this mean? Also, what is line search mentioned in the web page?

Can anyone please explain it with the help of a diagram?


texts: How should I approach such classification problem where the input is an array of integers?

texts:
I am training a model for predicting a number between 0 to 10 in this case. These are the number of roots of a polynomial. The input array for the number of polynomials is the coefficients of that polynomial from $x^n$ to constant. Even though I prepared a balanced dataset in which all number of roots (from 0 to 10) are equally existed, my input is an array of array of integers.

For example:

23 43 -545 34 45 -34 234 -434 234 434 -2344334

And the output of this is the number of roots of that polynomial (as I said before).

I tried many combinations of layers, but the accuracy never gets higher than 50% percent. By accuracy, I mean the number of correct predictions (I count the biggest probability as the prediction).

My Keras code for modeling:

model = Sequential()
model.add(Dense(32, input_dim=degree+1, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(degree+1, activation='softmax'))

model.compile(loss="categorical_crossentropy",
              optimizer='adam', metrics=['accuracy'])

model.fit(X, y, epochs=500, batch_size=50, verbose=0)

Is there something that I am doing wrong? Is this a good model for this kind of problem? I am new to deep learing so if the answer is obvious, I am sorry.

Thanks.


texts: How close is close enough, with regression?

texts:
When exploring different techniques in machine learning (neural networks), I like to use binary classification problems as a test-bed, because it's very easy to understand how well the technique is working: 50% training/test accuracy is no better than chance, 70% is okay, 98% is very good, etc.
However, sometimes I need to use regression problems instead, and here I struggle to interpret my results. If I draw a scatter plot of training and test mean squared error before and after training, and before training I get around 0.1 loss, and afterwards I get 0.01 loss... is that... good? How do I recognize the difference between success, and barely doing any better than chance?
Typically I'm working with synthetic problems, by the way: fitting polynomials or other kinds of mathematical functions on random vector inputs.


texts: I know we can use SVMs probabilities after predicting validation data in order to build ROC curves. However, for CNNs, I have a binary classification problem and so the sigmoid activation function will give me probabilities of both classes. So, which probability should I use to build the ROC curve? what happens if the probabilities are close to each other?


texts:
I am confused with the process for calculating loss. My code is below:

logits = policy.predictions(states)
negative_likelihoods = tf.nn.softmax_cross_entropy_with_logits(labels=**actions**, logits=logits)

weighted_negative_likelihoods = tf.multiply(negative_likelihoods, q_values)

loss = tf.reduce_mean(weighted_negative_likelihoods)

gradients = loss.gradients(loss, variables)

logits is the output of policy network without softmax.

My question is :

What does actions mean ? Is it the action that agent has executed at t step or it should execute at t step ?
Thanks


texts: Why Do Neural Networks Prefer Dense Features

texts:
Reading through this tf-ranking notebook there is an interesting note that typically networks take in dense features, this is true in my general experience. But I am wondering why, if I were to pass in a categorical feature as a one hot encoded representation would it be worse?

  https://colab.research.google.com/github/tensorflow/ranking/blob/master/tensorflow_ranking/examples/handling_sparse_features.ipynb#scrollTo=aQ-VTA56sOTA


texts: Machine Learning: Classify Array of Numbers based on Patterns

texts:
I have experimented with various regression/classifier libraries, they accept training input like "5 is bad" and "10 is good" so they can tell you 7 is bad and 8 is good.

Now imagine a more complicated example: I have heart rate data and there are certain heart rate patterns that are good and other patterns that are bad. In this case it's nonsense to just say "heart rate 120" is good or bad and so forth.
How do you tackle a problem like this? Are there any known ML/AI algorithms that can intelligently recognize patterns?

A good start could be a classifier that I can at least train with a whole array of numbers instead of a single number?


texts: What will the required time to process 500GB of images using NVIDIA GEFORCE 930M GPU

texts:
I have an image dataset of size 500GiB, and my system specs are NVIDIA GEFORCE 930M, 12GB of RAM and Intel Core i5.

I have the following questions:

Is it possible such a large dataset to be used in my local machine?
If yes, How much time will be required for one epoch or equivalently on iteration? Any links or reference on how to compute the required processing time will be helpful.
If my system is not good, what are the other possible solutions I have?


texts:
hope you're all doing good !
I am working on Automatic Speech Recognition with Python with the LibriSpeech Dataset.
After preprocessing the audios data and applying an "MFCC featurizing" I append everything into a list and get a shape of (14174,). Knowing that each sample has a different length but the same number of features for example :
print(X[0].shape)
print(X[12000].shape)
>> (615, 13)
>> (301, 13)

Now when I feed the data into my network with an Input layer defined as
input_data = Input(name='the_input', shape=(None, input_dim)) # with input_dim = 13 MFCC features

I get the following error
ValueError: Error when checking input: expected the_input to have 3 dimensions, but got array with shape (14174, 1)

I tried reshaping with different shapes but I am still struggling.
This is the model
def final_model(input_dim, units, output_dim=29):
    """ Build a bidirectional recurrent network for speech
    """
    # Main acoustic input
    input_data = Input(name='the_input', shape=(None, input_dim))

    # =============== 1st Layer =============== #
    # Add bidirectional recurrent layer
    bidirectional_rnn = Bidirectional(GRU(units, activation=None,return_sequences=True, implementation=2, name='bidir_rnn'))(input_data)
    # Add batch normalization
    batch_normalization = BatchNormalization(name = "batch_normalization_bidirectional_rnn")(bidirectional_rnn)
    # Add activation function
    activation = Activation('relu')(batch_normalization)
    # Add dropout
    #drop = Dropout(rate = 0.1)(activation)

    # =============== 2nd Layer =============== #
    # Add bidirectional recurrent layer
    bidirectional_rnn = Bidirectional(GRU(units, activation=None,return_sequences=True, implementation=2, name='bidir_rnn'))(activation)
    # Add batch normalization
    batch_normalization = BatchNormalization(name = "bn_bidir_rnn_2")(bidirectional_rnn)
    # Add activation function
    activation = Activation('relu')(batch_normalization)
    # Add dropout
    #drop = Dropout(rate = 0.1)(activation)

    # =============== 3rd Layer =============== #
    # Add a TimeDistributed(Dense(output_dim)) layer
    time_dense = TimeDistributed(Dense(output_dim))(activation)
    # Add softmax activation layer
    y_pred = Activation('softmax', name='softmax')(time_dense)

    # Specify the model
    model = Model(inputs=input_data, outputs=y_pred)
    model.output_length = lambda x: x
    print(model.summary())
    return model

Thanks


texts: Representing cyclical features as sin/cos components

texts:
I'm working on a prediction project where we have a lot cyclical features such as hour of the day, weekday, month, day of year, etc etc.
After some searching I decided to follow the advice here.

Now I have the sin and cos component for every cyclical feature as a separate feature, so month becomes month_sin and month_cos.
However, I don't know for sure whether the model can deal with this correlation, as both components need to be equally weighted in order for the feature to make sense. The model assigns different weights to the sin and cos components after training though. My intuition tells me that this is bad, but I'm not sure what to do about it.

Currently gbm (R) gives the best results. For a gradient boosting model, is it better to force equal weights on the two correlated features, or is it better to let the model figure it out even if it results in different weights on the two components? Or would you suggest an entirely different approach?


texts: How do I use rnn to forecast to n periods with limited data?

texts:
So this is my 1st time trying to run a small time-series dataset through an RNN, but after a lot of searching, I haven't been able to find,

1. How I can use this to forecast to n periods ? (like in the model.predict(start, end) function in ARIMA.)

2. Is there a better way of doing this using NN ?

details on the data is given as comments in the code below. thank you.

'''
total timeseries data points = 39
frequency = Months
train = 36
test  = 3
Need to forecast 3 periods ahead upto 42, currently using forecast horizon as 1
Limited data so used all in a single batch
'''

periods    = 35
f_horizon  = 1
batch_size = 35

x_batches  = dfp1[ : (len(dfp1) - (len(dfp1) % batch_size))].reshape(-1, periods, 1)          # train data for t   periods
y_batches  = dfp1[1 : len(dfp1) - (len(dfp1) % batch_size) + f_horizon].reshape(-1, periods, 1) # train data for t+1 periods

#defining test data set
def test_data(testdata):
    test_x = testdata[ :(len(testdata)-1)]   # t
    test_y = testdata[1:len(testdata)]       # t+1
    return test_x, test_y

test_x, test_y = test_data(dfp3[-4: ]) #since 35 points were used in training, used the last 4 points for testing

outputs    = 1
inputs     = 1
hidden     = 100

with tf.device('/cpu:0'):
    with tf.variable_scope('var', reuse = tf.AUTO_REUSE):
        x  = tf.placeholder(tf.float32, [None, periods, inputs])
        y  = tf.placeholder(tf.float32, [None, periods, outputs])

        #creating a basic rnn cell
        basicrnn           = tf.nn.rnn_cell.BasicRNNCell(num_units = hidden, activation = tf.nn.relu)
        rnn_output, states = tf.nn.dynamic_rnn(basicrnn, x, dtype = tf.float32)

        stacked_rnn_output = tf.reshape(rnn_output, [-1, hidden])
        stacked_output     = tf.layers.dense(stacked_rnn_output, outputs)
        outputs_           = tf.reshape(stacked_output, [-1, periods, 1])

        #using mape for loss fn
        loss               = tf.reduce_mean(tf.abs(tf.divide(tf.subtract(outputs_, y), y))) * 100

        learningrate       = 0.001
        training_op        = tf.train.AdamOptimizer(learning_rate = learningrate).minimize(loss)
        init               = tf.global_variables_initializer()

        epochs = 1501

        with tf.Session() as sess:
            init.run()

            for i in range(epochs):
                sess.run(training_op, feed_dict = {x: x_batches, y: y_batches})

                if i%100 == 0:
                    mape   = loss.eval(feed_dict = {x: x_batches, y: y_batches})

                    print("ep %s: %s"%(i, mape))

            '''
            unable to pass the 4 test data points since input size is 35,
            can I reshape x & y before I pass test input, would it alter the model ?
            also why is the loss fluctuating, and not steadily decreasing.
            '''

            #y_pred = sess.run(outputs_, feed_dict = {x: test_x})
            #print(y_pred)


texts: What do the hidden layers of a neural network try to learn?

texts: Typically all the layers of an artificial neural network are trainable. But what are the hidden layers trying to learn?


texts: How to use Python's FastFM library (factorization machines) for recommendation tasks?

texts:
I have a dataset of <user, item> pairs where each entry records which user bought which item.
e.g.

<u1, i1>
<u1, i4>
<u2, i2>
<u3, i2>...

I created an encoded dataset with

no_of_features = no_of_users + no_of_items

and have set output variable y to 1, as each of those entries represented the user having bought that particular item.

Note: All y values are 1 in this case.

The encoded dataset looks like this:

user1 user2 user3 .... item1 item2 item3 item4 .... y

  1     0    0    ....   1     0     0     0   .... 1
  1     0    0    ....   0     0     0     1   .... 1
  0     1    0    ....   0     1     0     0   .... 1
  0     0    1    ....   0     1     0     0   .... 1

Now, I would like to know how to use fastFM to generate recommendations for cases <x, y> for any user x and any item y?

Clearly regression is out of the picture here. So should I use the classification or ranking approach of fastFM? Also how? e.g. If I use classification, do I need to generate instances with y=0? If I use ranking approach, do I arrange items by the output of ranking approach and recommend those with higher values?


texts: How to create A/B test segements for highly variable data

texts:
I have a data in which there is a high degree of variability.

My Objective is to do an AB test to check the behavior change due to new changes.

All samples has shown historically high and low performances. This means if I take any 2 cohorts randomly, they show vast historical comparison difference Following is the example for weekly comparison. Same behavior holds true for monthly and daily too.

W1: -10.04%

W2: 3.9%

W3: -4.2%

W4: -3.7%

W5: 5.4%

W6: -2.6%

W8: 0.8%

W9: -2.3%

W10: 1.92%
...
..
.

No matter I tried to segment users by their aggregate behavior and see the difference, its still this level of variability is observed.
I'm looking for clearly difference to be consistent no matter in which ever direction. Either its positive or negative. And, with small range of magnitude.

Can someone please suggest the alternative methods to minimize this difference?


texts: Multiple formulas for r squared eval metric for regression

texts:
I was came across different formulas for R squared on different articles.

  R Squared = 1 - RSS/TSS

  R Squared = ESS/TSS

RSS -Residual sum of squares.
TSS - Total sum of squares.
ESS - Explained sum of squares.

Can any one explain which one should be the correct one.


texts: Best approach for detecting human in a picture

texts:
I am trying to detect if an image contains human or not. I will use CNN for this. I am planning to make a dataset like following.

Train
     human (contains 2000 or more human image)
     other object (contains 2000 or more non human image)

Test
     human (contains 500)
     other object (contains 500 )

I am planning to use keras and I will train data from directory .

Is there any alternative approach ? Because I am having trouble with collecting human and non human object image.


texts: Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?

texts:
I am building an insurance recommendation engine. I have used some variables, like demographics, and built the model. Now I have claims data.

Is there a way to include the new data without restarting the process like this?

model1 = initial variables
model2 = new variables
model3 = x*initial variables+y*new variables


texts: Configure BRAT so that if no `.ann` file is found, then an empty `.ann` file is created

texts: I am annotating a new corpus with BRAT. I have a set of .txt files to annotate. Is it possible to configure BRAT so that if no .ann file is found, then an empty .ann file is created? Or am I supposed to myself provide the empty .ann files?


texts: In plain English, how to descibe i/o of the TensorFlow for language modelling?

texts:
I have followed the tutorial here about language modelling using Tensorflow to create LSTM and used PTB dataset. The code is here

I failed to understnad the exact specific input and the output of the model.

My question is: how many words do you feed to the model and how many words do you expect at the end? How the data processing happens?

For example: if I have the sentence "The quick brown fox jumps over the lazy dog.". This will be turned into vectors using word2vec. Then how are these given to the NN?


texts: Is data leakage in time series due to both I's of the IID principe or only one?

texts: I am sure that the Independent part of the IID principle gives you data leakage because of the correlation. But the identical part I am not so sure. Identical in time series means that your data is stationary (and not changing in distribution over time). If your data thus is not identical you would for example have a rising average, which you could detrend, but is this then also data leakage if you would not detrend this rising average?


texts: How to play with .cat.codes and get_dummies

texts:
I may be lost here and this might be a stupid question.
I know that get_dummies returns multiple columns depending upon the number of variables and .cat.code returns a single column with unique integers per unique value(Please correct me if I'm wrong).
I know fro .cat.codes is for categorical data and get_dummies is for object.

Can I use astype("Category") on an object and use .cat.codes on it?
Anything more that you know of to make analysis(and visualization) easier when it comes to these funtions?


texts: Enforce Floor limit when predicting values using Multioutput Regression with Gradient Booster

texts:
I have a very simple program below that builds a model using multi-output regression. Even though all the training data consists of positive float values I'm discovering that predictions made often yield negative values. How can I tell scikit to enforce a floor of 0 (or in other words not make negative predictions)? Or what other tools/libraries can I use to ensure this doesn't occur?
import csv
from sklearn.multioutput import MultiOutputRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

X = []
Y = []

results = []
with open("folder/training_data.csv") as csvfile:
    reader = csv.reader(csvfile, quoting=csv.QUOTE_NONNUMERIC)
    for row in reader:  # each row is a list
        x = row[:5]
        y = row[5:]
        X.append(x)
        Y.append(y)

xtrain, xtest, ytrain, ytest = train_test_split(X, Y, test_size=0.15)

gbr = GradientBoostingRegressor()
model = MultiOutputRegressor(estimator=gbr)

model.fit(xtrain, ytrain)

...

prediction = model.predict([[1.0,2.0,3.0,4.0,5.0]])

# I get e.g. [[-0.2, -0.1]] back where I'd rather have [[0,0]]

The structure of the X variables take form as float values from 0.0 to 1e7 and Y target variables from 0.0 to 1e7


texts: Rank advertisement and Score keywords used in advertisement based on its performance

texts:
I am trying to classify my advertisements and trying to understand which advertisement is performing better than others and why.
Step - 1: compare two ads based on CTR.
The problem here is I am basically comparing 2 advertisements with different impressions, for example, ad-1 is having 100 impressions and 25% CTR, whereas ad-2 is having 50 impressions but 10% CTR, here I cannot compare both directly because ad-1 has got more CTR because the impressions are more(which means we have shown this more number of times), we can use basic math and try to figure for ad-2 by multiplying by 2 and making it for 100 impressions but is there a better way that we could potentially forecast the ad-2 based on impressions and predict CTR.
Step - 2: score the keywords in each.
Here I am trying to score the keyword and tagline used in the image of ads(I have extracted them using AWS rekognition) but what is a good data science technique to score the taglines?
sample dataset -
ad-1
Date -  ad name -   audience type - ad type -   Impressions -   Clicks  - Conersions -  Taglines
02-12-2020  advertisement 1 sports  generic 10000   100 50  (Play like star, It's never late, Dont give up)
03-12-2020  advertisement 1 sports  generic 10000   130 60  (Play like star, It's never late, Dont give up)
ad -2
Date -  ad name -   audience type - ad type -   Impressions -   Clicks  - Conersions -  Taglines
02-12-2020  advertisement 2 Kids    generic 5000    100 40  (Whatever it takes, Play like a champion)
03-12-2020  advertisement 2 Kids    generic 4000    50  10  (Whatever it takes, Play like a champion)
Here is a screenshot of the data mentioned above

Can someone please guide me on this


texts: What is the main goal of using activation function in CNN?

texts:
What is the main goal of using an activation function in CNN?

I know the activation functions types and the purpose of each one. But here I am asking why to use them.


texts: Which non-training classification methods are available?

texts:
I am trying to find which classification methods, that do not use a training phase, are available.

The scenario is gene expression based classification, in which you have a matrix of gene expression of m genes (features) and n samples (observations).
A signature for each class is also provided (that is a list of the features to consider to define to which class belongs a sample).

An application (non-training) is the Nearest Template Prediction method. In this case it is computed the cosine distance between each sample and each signature (on the common set of features). Then each sample is assigned to the nearest class (the sample-class comparison resulting in a smaller distance). No already classified samples are needed in this case.

A different application (training) is the kNN method, in which we have a set of already labeled samples. Then, each new sample is labeled depending on how are labeled the k nearest samples.

Are there any other non-training methods?

Thanks


texts: Is there any good alternative for Keras library?

texts: I am currently using keras as a deep learning library on top of tensorflow. I just want to know is there any other library which is more efficient/easier than keras.


texts: Check Imputation Efficiency - How To Compare Data Frames?

texts:
I try to evaluate several NA imputation methods with supervised approach: I clone my original data frame with no NAs, artifically insert NAs into the resulting Data Frame and apply imputations to the latter.

Now, I'd like to evaluate the imputations by comparing the imputed new DFs with the original one. I wonder what would be the best metod; is there any distance method for instance that I could apply to the original/imputed DF pairs?

(My DF contains only numeric data, but a solution that handles factor variables as well would be especially handy).


texts: Where is my error in understanding gradient descent calculated two different ways?

texts:
The gradient descent algorithm is, most simply, w'(i) = w(i)-r*dC/dw(i) where w(i) are the old weights, w'(i) are the new weights, C is the cost, r is the learning rate. I'm aware of the graphical justification for this.

For one weight, this is w' = w - r*dC/dw.

Second, we also have this equation deltaC ~= sum(dC/dw(i) * deltaw(i) ), which is just the definition of linearity of C near the point that its derivative is calculated. For one weight, this is the same as deltaC/deltaw ~= dC/dw, e.g., the definition of derivative.

Let there be only one weight, and let s = -deltaC. Then we have -s = dC/dw * (w'-w), where we've split up deltaw into the original and perturbed value. Then w'-w = s * (1/ (dC/dw)), and w' = w -s * (1/dC/dw). (Since we want to reduce the cost, we want deltaC to be <= 0, so s is >= 0, and looks like a normal positive learning rate.)

What I haven't been able to understand is why I get two different answers for what appears to be the same operation, updating the weights to lower the cost. In one case I use dC/dw, and in the other, I use 1/(dC/dw.) In both cases, r and s are small positive numbers.

What am I missing?


texts: Extracting specific data from unstructured text - NER

texts:
I have a reasonably simple problem to solve. I need to extract reservations numbers from unstructured text. Based on my research, it seems to be an NER problem. Based on a visual analysis of the dataset, I could notice that the reservation number is frequently near specific keywords, such as 'confirmation', 'reservation', 'confirmation number', 'reservations number', etc.

First, I decided to try a Regex rule to extract the data, but some minimum variations might render this solution inefficient. The reservation number can have very different variations, such as:

ZXC51657856,
EA5FFD4,
45615177413515,
QT454545EF,

At this moment, I don't have a dataset available to train a classifier to solve this issue.

I would like to receive some ideas from the community to guide me towards an elegant solution to this problem, as I'm pretty new to ML in general and time is limited.


texts: Need explanation about output of R function neuralnet()

texts:
I am a beginner in R. I am a bit confused about the output of the neuralnet() function in R, from package neuralnet.

Here it shows steps equal to thirteen. I would like to inquire about the understanding of these steps. What is the total of all the steps that occur? The data I used as many as seven before the output as shown. Then my question is, how to display the details of each step that occurs.


texts: How can I implement tangent distance for k-nearest neighbor in python/scikit-learn?

texts:
My ultimate aim is to have a function which I can feed into scikit-learn's NearestNeighbor class as a custom metric parameter.

Existing packages

I have been researching existing libraries for a while.

The only thing I found was this KMeans package, for python 2 and based on implementing a C library.  I could neither load it in with ctypes nor convert it into an executable with gcc.

I also found this other C code and this Matlab script but with similar results.

Implementation

I also looked into a few papers, to see if I can implement it by myself. For instance, based on this I understand that the main thing I need to do is to calculate the tangent matrix. But, I do not understand

how do I define $s(p, \alpha)$ and especially
how do I calculate the derivatives in python.

I would be glad for any help, comment, whatsoever.

Update

As suggested, I raised the following related issues/requests:

ComeOnGetMe
Scikit-learn

Update 2

@ComeOnGetMe rewrote his code so it can be used along the scikit-learn specifications (example code). Many thanks for that! Nonetheless, when I tried to use it in scikit-learn it underperformed and was very slow, so there is further work needed with that.

Since then I also found a more detailed explanation for code implementation, although based on the C code already mentioned.


texts: Linear and non-linear dependence in a single DS model

texts:
I have a dataset with parameters (features) a,b,c, etc. We need to develop a model to predict a (our target).
b is correlated with a significantly (85%) and I suspect linear dependence. c is a measurement of b in another depth, so it has a high correlation with b and good correlation with a. Also, there are a bunch of other parameters (numerical features) whose dependence of a in non-linear (from background knowledge and box plot analysis). These features don't have a high correlation with a but have effect on a.
I am building a model to predict a based on the knowledge above. I think that a multi linear regression cannot be a good idea because of non-linear dependecies. On the other hand, b is a strong predictor of a and depend on a linearly. Therefore, I need to ensemble both linear regression and other methods (e.g., random forest) in a single model to have advantages of both.


texts: How can I check the correlation between features and target variable?

texts:
I am trying to build a Regression model and I am looking for a way to check whether there's any correlation between features and target variables?

This is my sample dataset

     Loan_ID    Gender  Married Dependents  Education Self_Employed ApplicantIncome\

0   LP001002    Male    No         0        Graduate      No            5849
1   LP001003    Male    Yes        1        Graduate      No            4583
2   LP001005    Male    Yes        0        Graduate     Yes            3000
3   LP001006    Male    Yes        0        Not Graduate  No            2583
4   LP001008    Male    No         0        Graduate      No            6000

CoapplicantIncome  LoanAmount   Loan_Amount_Term  Credit_History Area Loan_Status
      0.0               123          360.0            1.0        Urban     Y
      1508.0          128.0          360.0            1.0        Rural     N
      0.0              66.0          360.0            1.0        Urban     Y
      2358.0          120.0          360.0            1.0        Urban     Y
      0.0             141.0          360.0            1.0        Urban     Y

I am trying to predict LoanAmount column based on the features available above.

I just want to see if there's a correlation between the features and target variable. I tried LinearRegression, GradientBoostingRegressor and I'm hardly getting a accuracy of around 0.30 - 0.40%.

Any suggestions on algorithms, params etc that I should use for better prediction?


texts:
I am trying to compute the total number of parameters in Ilya Sutskever's seq2seq model.
In the paper, the authors mention a total number of 384M trainable parameters, or to be exact:

  The resulting LSTM has 384M parameters of which 64M are pure recurrent connections (32M for the “encoder” LSTM and 32M for the “decoder” LSTM)

In short, their model has:

one encoder LSTM and one decoder LSTM
each LSTM has four layers
each LSTM layer has 1000 units
input: 1000-dimensional word embedding of 160K possible input words
output: naive softmax over 80K possible words

Following this post and using the formula
$$
num~params = 4 (nm + n^2)
$$

I was able to compute:

For the encoder LSTM

n = 1000 for all layers
m = 1000 for the first hidden layer, because of 1000 dimensional embedding
m = 1000 for all other layers, because that's the number of units in each hidden layer

which yields:
input -> first hidden layer: 8M
first -> second            : 8M
second -> third            : 8M
third -> last              : 8M
total                      = 32M

which I am guessing is what they mean when they say 32M recurrent connections for the LSTM?

For the decoder LSTM

computations are exactly the same as above, with the caveat that the input size is 1000 because that's the output size of the encoder LSTM.

total                      = 32M

Where are the other 320M parameters?

The computations made so far account for 64M parameters.
We still haven't accounted for the final softmax layer, which admittedly could be pretty large since its output is 80K, and 320M can be suspiciously decomposed as 80K * 4K, but I'm not sure about this.

My questions are:

can the naive softmax layer alone account for the 320M missing parameters?
shouldn't we also be using the fact that this is a recurrent neural networks to compute additional weights, as in this post?

Thank you


texts: Which dissimilarity/similarity measure use after a dimension reduction ( PCA / AutoEncoder / ... )?

texts:
Each problem required its own similarity/dissimilarity measure. Imagine we are dealing with dataset composed with vector of real. I suppose that we mostly use the euclidean distance especially in low dimension. Unfortunately we often have to deal with huge dimensional dataset and we apply divers dimension reduction techniques to make the problem easier to solve.

I will be happy to know your opinion about different similarity/dissimilarity measure you use after made a dimensionality reduction and how they impact classification/regression/clustering metrics.

Is it better to use euclidean distance after a PCA or another distance measure ? Why ?

And if i prefer to use AutoEncoder to reduce my dataset dimensions, does it exists a special one that guaranty to offer better results with a specific similiarity/dissimilarity measure ?

Does it exist a dimensionality reduction technique that is optimal with a particular distance metrics an which is bounded in $O(n.log(n))$ in time ?


texts: Fast and/or memory efficient classification solutions

texts:
I'm working on a classification problem with a very large dataset (a little under 1 billion obs) and around 25 predictors.

I'm doing this analysis in R on a VM with 128GB of memory, but am still hitting memory issues when training certain types of models, not to mention having to wait for quite a while for runs to complete. I'm primarily using logistic regression and random forests, and keeping my training dataset to 10% of the overall sample.

What are some solutions (packages, platforms, techniques) I could use to address these memory and/or speed issues? I'm fairly comfortable with Python, so solutions do not have to be R-specific.


texts: Can you provide examples of business application of vector autoregressive model?

texts:
Vector Autoregressive models are exploited at Economics faculties all around the world. They are just another statistical model that solves problem of forecasting, although in a deeply complexity-uncovering manner.

Yet to my surprise, there is no evidence it has been used outside pure economics domain, namely, to solve business problems like we all - Data Scientists - do.

Can you share either your experience with application of VAR to solve business problem, a scenario in which it could hypothetically be used or state why it cannot be easily applied in business environment (if this is an objective, mathematically/empirically grounded reason, I am not looking for opinions on why this method has slow adoption)?

The question can apply as well to VECM models of course.


texts: Model Dump Parser (like XGBFI) for LightGBM and CatBoost

texts:
Currently my employer has multiple GLM in a live environment. I am interested in identifying new features and interactions to enhance the accuracy of these GLM; for now I am limited to the GLM structure so simply deploying a solution which automatically accounts for interactions is not possible.
I have in the past used XGBoost to identify powerful feature interactions through the use of XGBFI / XGBFIR. I am now looking in to using LightGBM and CatBoost to do the same but cannot seem to find a tool like XGBFI which works on those models.
Question: Does anything similar to XGBFI exist for LightGBM and CatBoost?


texts: Histogram indicating 20~25% in a certain range

texts:
This is a histogram of speeds of certain ships drawn to the density scale:

I was told that the percent of speeds in the [17, 18) range is between 20 and 25, but I believe it's between 30 and 50. Can anyone convince me wrong?


texts: How to determine x and y in 2 dimensional K-means clustering?

texts:
Recently, we were taught K-means clustering. I understood the basic idea of the algorithm and successfully implemented it for data with a single dimensional. Now we are told to implement it for two dimensional data. As far as I understood x and y can be two attributes of a dataset but our professor said otherwise. She said that the we have to determine x and y of an attribute in the data set to cluster the data. She used a simple 2D matrix as an example. This has got me confused. How can one determine x and y of an atttribute? Row number and column number seems silly to me to use it for distance calculation.

So, my question is how does one determine x and y for 2-D k means clustering?

As per this question the two attributes (weight and height) are used as x and y. Is this correct?


texts: Difference between sequence length and batch size in time series forecasting

texts:
I am using Keras for time series forecasting and I am trying to understand the tutorial on the offical site of keras about time series forecasting that you can find here (https://keras.io/examples/timeseries/timeseries_weather_forecasting/).
They use one keras-method called keras.preprocessing.timeseries_dataset_from_array and it has the following parameters (here is a documentation https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/timeseries_dataset_from_array):
    dataset_train = keras.preprocessing.timeseries_dataset_from_array(
    x_train,
    y_train,
    sequence_length=sequence_length,
    sampling_rate=step,
    batch_size=batch_size,
)

So my question is what is the difference between the sequence length and the batch size. I think the sequence is the size of the sliding window (x-features and one target y-value). But what is the batch size? Unfortunately I can't have a look at the output of this method as
   print(dataset_train) or
   print(dataset_train.head())

does not show me the data and I do not know any other function how I could have a look at the output of the method.
Has anyone of you had expercience with this method or generally with sequences and batches? I'd appreciate every comment.


texts: could not broadcast input array from shape (2,3) into shape (3) while using timestamp to build neural network in python

texts:
Here I want predict value every 60 minutes. So I have data 540 with three inputs. so I wrote an code with time steps and it gave me this error. Can anyone help me to solve this problem?

my code :

y=data['y1'].astype(int)
cols=['x1', 'x2', 'x3']
x=data[cols].astype(int)
n = x.shape[0]
p = x.shape[1]
x  = x.values
y = y.values
train_start = 0
train_end = int(np.floor(0.8*n))
test_start = train_end+1
test_end = n
x_train = x[np.arange(train_start, train_end), :]
x_test = x[np.arange(test_start, test_end), :]
y_train = y[np.arange(train_start, train_end), :]
y_test = y[np.arange(test_start, test_end), :]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))

num_time_steps = 9
num_features = x.shape[1]

x_train = np.zeros((x_train.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
x_test = np.zeros((x_test.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")

for i in range(len(x_train)):
for timestep in range(num_time_steps):
    x_train[i][timestep] = x_train[i + timestep]

for i in range(len(x_test)):
for timestep in range(num_time_steps):
    x_test[i][timestep] = x_test[i + timestep]

 y_train = y_train[num_time_steps - 1:]
 y_test = y_test[num_time_steps - 1:]

change code:

train_end = 80
x_train=x[0: train_end ,]
x_test=x[train_end +1: ,]
y_train=y[0: train_end]
y_test=y[train_end +1:]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))
num_time_steps = 9
num_features = x.shape[1]
x_train_n = np.zeros((x_train.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
x_test_n = np.zeros((x_test.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
for i in range(len(x_train_n)):
for timestep in range(num_time_steps):
    x_train_n[i][timestep] = x_train[i + timestep]
for i in range(len(x_test_n)):
for timestep in range(num_time_steps):
    x_test_n[i][timestep] = x_test[i+timestep]
y_train_n = y_train[num_time_steps - 1:]
y_test_n = y_test[num_time_steps - 1:]

Error,


texts: For a regression model, can you transform all your features to linear to make a better prediction?

texts:
I was thinking. Would it be a good approach to check your features one by one (assuming you have a manageable amount of them)  and see the relationship they have with your target variable, if they have a non linear relationship then transform each of those features using their appropriate function for each case to make them linear? In my mind if you do this your are guaranteed to have a better Linear model and also you are able to perform hypothesis testing on each feature to see the relevance of them, giving you the chance to perform some feature selection as well.

I know that the interpretability of model will be thrown out of the window, but the model will give a much better performance. Basically you could potentially end up with a model with only engineer features (assuming that all of them have a non linear relationship)

Would this approach be acceptable and it is worth exploring?


texts:
import numpy as np
import tensorflow as tf
import pandas as pd

from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import OrdinalEncoder
from sklearn.model_selection import train_test_split

def import_data():
    pulstars = pd.read_csv("pulsar_stars.csv", index_col=0)
    x = pulstars[["Standard deviation of the integrated profile", "Excess kurtosis of the integrated profile","Skewness of the integrated profile","Mean of the DM-SNR curve","Standard deviation of the DM-SNR curve","Excess kurtosis of the DM-SNR curve","Skewness of the DM-SNR curve"]]
    y = pulstars[["target_class"]]

    return x,y

inp , out = import_data()

out = out.dropna()

#normalize data
scaler = MinMaxScaler()
inp = scaler.fit_transform(inp)
out = scaler.fit_transform(out)

classConverter = OneHotEncoder(sparse=False)
out = classConverter.fit_transform(out)

layer = {
    "input": 8,
    "hidden": 8,
    "output": 2
}

i_h = {
    "weight": tf.Variable(tf.random_normal( [ layer["input"] , layer["hidden"] ] )),
    "bias": tf.Variable(tf.random_normal([layer["hidden"]]))
}

# weight = {
#     "i_h":tf.Variable(tf.random_normal( [ layer["input"] ], [ layer["hidden"] ] )),
#     "h_o":tf.Variable(tf.random_normal([[layer["hidden"]],[layer["output"]]]))
# }

# bias = {
#     "i_h":tf.Variable(tf.random_normal([layer["hidden"]])),
#     "h_o":tf.Variable(tf.random_normal([layer["output"]]))
# }

h_o = {
    "weight": tf.Variable(tf.random_normal([layer["hidden"],layer["output"]])),
    "bias": tf.Variable(tf.random_normal([layer["output"]]))
}

inp_dataset = tf.placeholder(tf.float32, [None, layer["input"]] )
target = tf.placeholder(tf.float32, [None, layer["output"]])

def feed_forward(inp_dataset):
    x1 = tf.matmul(inp_dataset,i_h["weight"]) + i_h["bias"]
    y1 = tf.nn.sigmoid(x1)

    x2 = tf.matmul(y1,h_o["weight"]) + h_o["bias"]
    y2 = tf.nn.sigmoid(x2)

    return y2 #output

output = feed_forward(inp_dataset)

error = tf.reduce_mean(0.5 * (target - output) ** 2)

#back propagation (update weight & bias)
update = tf.train.GradientDescentOptimizer(0.2).minimize(error)

#split dataset
x_train, x_test, y_train, y_test = train_test_split(inp, out, test_size = 0.2)

and this happen

ValueError                                Traceback (most recent call last)
<ipython-input-12-48df591e9215> in <module>()
     79
     80 #split dataset
---> 81 x_train, x_test, y_train, y_test = train_test_split(inp, out, test_size = 0.2)
     82
     83

2 frames
/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py in check_consistent_length(*arrays)
    210     if len(uniques) > 1:
    211         raise ValueError("Found input variables with inconsistent numbers of"
--> 212                          " samples: %r" % [int(l) for l in lengths])
    213
    214

ValueError: Found input variables with inconsistent numbers of samples: [13366, 1735]


texts: How to remove spikes from data with Python using signal.find_peaks


texts: Vector Arithmetic using WGAN-GP (Wasserstein GANs with Gradient Penalty)

texts: Vector arithmetic in the latent space has been demonstrated to produce meaningful output image samples from a trained DC-GAN in the paper by Chintala et al. In fact, the vector arithmetic they described was not directly over the individual vectors themselves in the latent space, which they report to give an unstable output image. Instead they had first taken an average of the vectors per images that looked the same before demonstrating that Man with Spectacles - Man + Woman = Woman with Spectacles. But how relevant is this feature in the new Wasserstein GANs with Gradient Penalty introduced by Gulrajani et al.? Since these are much improved versions of the DC-GANs, can the vector arithmetic be done now directly over the individual vectors in the latent space? Will appreciate if some relevant works can be pointed out on using vector arithmetic on trained WGAN-GP models.


texts: How can both generator and discriminator losses decrease?

texts:
In this paper there is a plot of how the loss of gans looks through epochs.

Figure 2:

These are of course averaged losses.

How can both the discriminator loss and generator loss decrease?

real_y = discriminator(real_sample)
fake_y = discriminator(generator(noise))
discriminator_loss = real_y-fake_y+1 # nicer, it is between 0 and 2

fake_y = discriminator(generator(noise))
generator_loss = fake_y

I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. One optimizer is making fake_y less and the other optimizer is making it more.

Maybe the loss functions aren't calculated like I said.

In the widely used analogy:

  In simple terms the generator is like a forger trying to produce some counterfeit material, and the discriminator is like the police trying to detect the forged items.

We can measure how good the police is by how many times out of 100 fakes he can identify the real and fake ones and the forger as deceiving the policeman out of 100 fakes.

Wouldn't that mean that if the police get better, the forger gets worse? (if using the aforementioned measure of how good they are)

Therefore we wouldn't be able to see both of them being good at the same time! But the graph from the paper indicates otherwise.

What am I missing?


texts: Is it possible to design a deep CNN model on a small size image dataset

texts: I have started a project of classifying a dataset using deep learning. I have tried transfer learning on pretrained models. Now, I want to design a CNN model which can do this work of classification but everywhere it says it needs large data. My question is: Is it possible to design a deep CNN model on an image dataset of size less than 1000 images?


texts: Creating a validation set on a benchmark dataset without a validation set

texts:
I am using a benchmark dataset, which is already split into 5 subsets = (a, b, c ,d , e) and, I am constrained by already defined train/test configurations. For example, if I am using the abc as train set and d as test set configuration, how should I select the validation set? do I take subset of abc, or, take subset of d?
the configurations are: abc/d, bcd/a, and abcd/e


texts:
Unfortunately trying to google or research null values in machine learning always brings up pages trying to teach you how to impute the values instead, but I'm trying to find models that can handle null values as input. The only one I've found currently is XGBoost, which is a gradient descent algorithm. I believe tree based algorithms should in theory handle null values as well, but I'm not sure if that generalizes to all tree based algorithms or if some work better than others.

Some background, I have created multiple datasets, including ones where the NaNs ARE replaced. Those can be used to train models that can't handle null values, like neural networks.

Due to the sparseness and nature of the data, imputing with mean or median is going to introduce a LOT of bias and doesn't make sense. Just want to get that out there since that's the first thing everyone suggests. Also, the missing data actually represents a situation where a patient didn't get a specific lab test done, which is useful information of its own.

So back to the original question, besides XGBoost, are there other models that handle null values as inputs? Does that generalize to ALL gradient descent algorithms or tree algorithms or just specific ones? Is there a preferred model for a situation where you want to keep null values?


texts:
I have a time series of data points. Then I am given a future timestamp and  I have to predict the value for the data point. For simplicity, you can assume that the timestamp is bounded i.e. for e.g. query can be of at most 1 hr in the future.

This is different from tradition train and predict models. Here you will be given the time as query input apart from past data.

Currently, I am training different models for each minute(yeah, 60 models: lots of waiting time ).

I am wondering if there is something available for this specific task?

EDIT: To get the view about the data, you can assume a simple time series of real numbers, and I have to use history to predict for any general time(within next 1 hr).


texts: Training model in Keras(TF backend) with GPU

texts:
I want to train my custom dataset for licence plate detection. I have 2000 image for this. First, I trained my labelling dataset in Keras (TF backend) with CPU.

  optimizer : Adam, learning rate : 0.01 , batch size = 64,
  iteration:100000

In this case, my training period exceeded 10 days, loss is 10. When I tested the model could detect licence plates. (acc : 80%)

Then, I decided to train with gpu because of the training period is too long.

I uninstall tensorflow and I install tensorflow-gpu(1.13), Cuda(10.0) and Cudnn(7.4) and I added these codes:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.InteractiveSession(config=config)

In this time, when I use the same values (Adam, lr=0.01, itr=100000, bs=64), loss value decreases very fast every iteration (loss begins with 210 and ends 0.000000). The training ends in four-five hours. But when I tested, my model could not detect any licence plate.

I try to change values. I did learning rate = 0.000001, iteration =3000 so that the loss is 0.7, training ends 10-20 minutes, but model detects very wrong. (and I know this number is very bad for learning rate)

My GPU : GeForce RTX 2080

How can I optimize these numbers, where is my mistake?

Thank you.


texts: building a boosting model for repeated measurments

texts:
I am working on an e-commerce data where the goal is to predict how will the user rate a movie from 1 to 5. We have a bunch of data from users but also from products. Some users have previously rated more than 10 (even 100s) and some less. Something like the following.

Imaging I am using a boosting model xgboost/catboost/lightgbm/... - I can feed the model with the data (excel the user_id) column and get sth that can predict the rate. but I believe this model is not at the personal level - meaning is not tailored to learn what each user like and is rather a generic model. So ideally there would be a tree per user.
I wonder how I can make this model to work at the personal level ? for example, learn that user_id 12 likes x and y but not z and w ?
How can I inform that model to build tree at each user ? I am sorry if I am not using the right terminology or the best title.


texts: Does adding a class to a model increase its performance?

texts: I am experimenting with object detection in images via CNN, to be more specific with yolov5 and faster RCNN. The models I use were pre-trained on the coco dataset before I finetune them. My dataset is relatively small with only about 1000 instances so I can't train the whole model from scratch. I am only detecting one class for now, but still get false positives sometimes, for objects that look quite similar to my class. My question is now, should I add this very similar class to the model and train on two classes instead of one to minimize false positives?


texts: Will my neural network get lazy if I give it an "easy" feature?

texts:
Let's say I start with a standard conv net architecture capable of 99% accuracy on MNIST such as this one, but let's say I merge in an "easy" feature to the fully-connecter layer such as a vector of length 10 that encodes the correct output digit 95% of the time and a random digit otherwise.

Will my network ever reach 99% accuracy
Will it take longer to get there?

I think the answer the question 1 is yes, because we will should always be able to find a downhill in the error before the conv net path has reached its full potential. In fact I think it may surpass the accuracy of the original architecture since we are leaking information about the correct output labels.

However I am not so sure about question 2. Does this make the shape of error function unfavorable in any way? I can't decide whether reaching the same accuracy of the original architecture will be faster, slower, or exactly the same.

Hopefully there is an easy answer, otherwise I will run an experiment and report back!


texts: What is the difference between "expected return" and "expected reward" in the context of RL?

texts:
The value of a state $s$ under a certain policy $\pi$, $V^\pi(s)$, is defined as the "expected return" starting from state $s$. More precisely, it is defined as

$$
V^\pi(s) = \mathbb{E}\left(R_t \mid s_t = s \right)
$$

where $R_t$ can be defined as

$$
\sum_{k=0}^\infty \gamma^k r_{t+k+1}
$$

which is a sum of "discounted" rewards after time $t$, i.e. starting from time $t+1$.

$V^\pi(s)$ can also be interpreted, even more precisely, as the expected cumulative future discounted reward. This denotation contains all the words which refer to specific parts of the formula above, where

"Expected" refers to the "expected value"
"Cumulative" refers to the summation
"Future" refers to the fact that it's an expected value of a future quantity with respect to the present quantity, i.e. $s_t = s$.
"Discounted" refers to the "gamma" factor, which is a way to adjust the importance of how much we value rewards at future time steps, i.e. starting from $t + 1$.
"Reward" refers to the main quantity of interested, i.e. the reward received from the environment.

Meanwhile, I've heard the term "expected reward", but I am not sure if it refers to the same concept or not, that is if "expected reward" or "expected return" are the same thing or not.

I know there's also the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as

$$
\mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1}  \mid s_t = s, a_t = a, s_{t+1} = s' \right)
$$

which, again, is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.

Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?


texts: Why do we use a softmax activation function in Convolutional Autoencoders?

texts:
I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras.

At the output layer, the author has used the softmax activation function. Shouldn't it be ReLU?

  According to me, this seems to be a regression problem, where we need
  to predict the continuous values for the segmented image pixels. If so, why are we using a softmax function, instead of ReLU or a linear function?


texts: How to build an overfitted network in order to increase performances

texts:
I am learning how to implement CNN, and searching on the internet I have found that a trick to design a good network is to first build it in such a way that it overfits, and then use regularization to elimnate overfitting and have a good performing network.

But how do I do this? I don't understand how do I build a network that overfits on purpose? And also in which way do I use regularization after?

Can someone helo me?

Thanks in advance.


texts: Change aliases of filter items in Tableau

texts:
Data sample contains a single feature: random integer number from 1 to 4.

Is it possble to change 1,2,3,4 representation on the filter card to some custom names, say: Type1,Type2,Type3,Type4? (not changing data set)


texts: Using feature importance to decet latent variables and grouping

texts:
Is it possible to use feature importance from Random Forests (e.g. based on gini impurity) or other models to determine which features I can use to group the rows of my dataset homogeneously?

For example, let's say I have a dataset with N rows and p columns, (one of the columns used as the label in my training task). I train the model and I get a rank of the importance of my features. Only 5 features are more important than a random feature "artificially" added to my dataset (this is used to understand if some features do not add any "predictable power" to my model). Can I assume that using these 5 features as dimensions to group my rows, provide "homogeneous group w.r.t. my output"?

Apologies for the confusing description of my problem. Rather than a strict answer I'd like to be pointed to the right direction to make some research into this. For now this is solely based on my intuition and I could be totally off-road.

Is there any area of ML/Stats/etc. where this is done?

For example, Latent Class Analysis seems something similar to this, am I wrong?

Example:

I want to predict the probability of an person to drink more than 0.5 liter of beer in a month. I have information about each individual in the population such as: age, sex, geographical area (US state), height, weight, etc.

My output is

0 = avg(liter of beer per mount) < 0.5
1 = avg(liter of beer per mount) >= 0.5

I train a model (let's say a Random forest) and the feature importance ranks says that "age", "weight" and "height" are useful in predicting the consumption of beer, while the other features do not add much to the predicting performance (or at least those are ranked as less important).

My assumption is that if I group the population if using the three "important" features these groups will share some similar characteristic, hence there's a latent variable which groups these individual together.

Groups:

1) Age: [21, 25], weight(Kg): [70, 75], height(cm): [170, 175]

2) Age: [21, 25], weight(Kg): [70, 75], height(cm): [175, 180]

3) etc.


texts: How to interpret Tensorboard Values for ObjectDetection

texts: I am training an object detection neural networt using tensorflow and the ssd mobilenetv3 model. I want to compare different training datasets. Which metric in tensorboard should I use for this? My first idea is comparing the mAP values. Is this the correct way? And how are the accuracy values calculated? From the testset oder from the training test?


texts:
I have a categorical variable, country which takes on values like India, US, Pakistan etc. I am currently using a linear SLM for a classification task.

So my country value varies from 1-20. How should this be a feature in the classification task. Should i have a one hot vector like (1,0,0..) for us and assign this vector 20 weights, or should i have integer from 1_20 and assign a single weight? I am using scikit learn. Does the answer depend on classifier?


texts: Sensorfusion: Generate virtual sensor based on analysis of sensorsdata

texts:
I have a steam engine which is equipped with the following sensors:

temperature sensor in the boiler room
temperature sensor in the heating room
pressure sensor in the boiler room
rotations-per-minute-sensor which measures rpm of the wheel, which is powered by the steam engine

What the steam engine does not measure is the current water level in the boiler. The time-series data is stored in a csv-file with the following structure:

| timestamp | temp_boiler | temp_heater | pressure_boiler | rpm |

With help of machine learning, I would like to predict the current water level in the boiler. As I am very new to Machine Learning, I do not really know how to accomplish this. I know that I cannot apply a linear regression, because it is a time-series data and there is a correlation between the current sensor values and the previous sensor values. Furthermore, I would need historic water level date in order to run the linear regression.

So what options are there?
Do I have to use a Neuronal Network because I do not have any water level values? (Keywords: Sensorfusion, Virtual Sensor, Softsensor)
Any help is greatly appreciated!

EDIT: As linear regression is not an option, I would like to know what the alternatives are. As I am a complete newbie to Machine Learning in general, I would like to know which algorithms or strategies I can use to solve this. I am not asking for code snippets or something like that. Just some input so that I can work this out!


texts: Why do decision trees have low accuracy?

texts:
It seems to be generally acknowledged that decision trees have low prediction accuracy. Is there a concise explanation for why they have low accuracy?

I've read this so much, I've accepted it to be true, but I realize I don't have any intuition as to why it's true.

As an example, here's an excerpt from Elements of Statistical Learning (page 352):

  Trees have one aspect that prevents them from being the ideal tool for predictive learning, namely inaccuracy. They seldom provide predictive ac- curacy comparable to the best that can be achieved with the data at hand.

Or on Wikipedia, under the heading Disadvantages of Decision Trees: "They are often relatively inaccurate. Many other predictors perform better with similar data. "


texts: How to automatically verify official documents?

texts:
I am new to machine learning and data science. I apologise if the question seems very basic. I have a requirement where I need to verify information submitted via a form with the corresponding official document. My approach till now has been to use google vision to extract text and use regex to extract necessary fields and compare with the form information. This is not always reliable because of the image quality and vision captures the noise as well.

Previously, I was thinking of just comparing each data file and search it in the text extracted and provide a metric of certainty.

I talked to one of my colleagues and he suggested using some for of supervised learning algorithm to process the documents, so that they automatically extract the key fields.

I want to ask would this method be simpler than my current approach. I am also worried about the scalability of my approach in event of minor changes in document formats.

I am looking for links for some articles or books related to this and answer to why using my own model will be better than just searching each word in a text.

Edit: The data can be imagined as a visiting card which contains the name of person, his office address, his contact number and the company's name. The problem here is different visiting cards can have different formats. Moreover the information is repeated twice: Once in English and the second in another language. For example , NAME : JANE DOE is followed by नाम : जेन डोए in next line. I only need the English name. The number of formats are constant but are high. Moreover, the address is not read together by Google Vision in most cases and using regex can become too complicated and case specific. I need to verify employee identity cards as proof that they are working for the company they claim to.


texts: Inception/ResNet doing worse than SIFT in feature extraction

texts:
We are doing our Thesis on multimodal retrieval. it's basically searching different modalities (multimedia ex: text, video, images ...) with other modalities. i.e. searching a database of images with a text query.

For any modality we need first to map it to a space where it has a constant number of features, and those feature must be somehow expressive of the data.

For images, papers we evaluate utilizes SIFT feature extraction, we use NUSWIDE for evaluating different methods, it already exists in SIFT format so most papers we evaluate uses these existent dataset.

We tried to improve on this feature extraction mechanism by using Inception or Resnet and taking layers exactly before softmax as our features. however, they perform way worse than SIFT. we used tensorflow and keras to extract features.

so any idea why would resnet/inception perfrom worse than SIFT?


texts: Gender Prediction from Offline Handwriting Using Convolutional Neural Networks

texts:
Starting from the fact that handwritten documents style are gender-dependent (male and female have different writing styles), I'm trying to predict writer's gender from its handwritten scripts using Convolutional Neural Network (CNN). I choose the IAM and KHATT datasets for English and Arabic respectively.  After I read research articles related to this problem, I realized that few of them used deep learning (with handwritten word and/or sentence as input to CNN) and the majority used classic methods like (LBP, HOG, GLCM, SFTA), the state of the art accuracy is 80.79% and 85% (for CNN and classic methods respectively), I also read this Article about identifying/verifying writer from its handwritten scripts which uses texture blocks from the written documents that give good results.
As data preprocessing I used the method of texture blocks, after line and word segmentation, i constructed texture image from handwriting words, and segment the texture image to texture blocks of size 100*100px.

I used 60 writers per gender for training (which give 42,000 texture blocks), and 7 writers per gender for validation (which give 4,000 texture blocks) and 7 writers per gender for testing (which give 4,000 texture blocks).

I'm using TensorFlow and Keras as framework, I started with simple (LeNet-like architectures) for base line, I got 60% test accuracy. As a second approach, I used different state of the art architecture in image classification like (VGG16, VGG19, RESNET34, RESNET50) and I got 64% test accuracy.

So, my questions are:

How to reduce overfitting though I tried regularization methods (Dropout, L1, L2, Batch Norm) to reduce overfitting ?
I observed high variance in accuracy, if I re-shuffle the data (e.g from 64% to 50%) ?
How to modify CNN architectures to (multi-input CNN) to make the decision on multiple texture blocks, instead of one?

texture blocks of females:

texture blocks of males:

Please point me in the right direction.


texts:
Is this the correct Estimator for Logistic Regression in TF 1.10?

There used to be a function called: LogisticRegressor which is deprecated

In README.md file it says:

LogisticRegressor: Not supported. Instead, use binary_classification_head with a custom model_fn, or with DNNEstimator

When I look into documentation in tensorflow.org I see an example on Census data:

"We will train a logistic regression model that, given an individual's information, outputs a number between 0 and 1—this can be interpreted as the probability that the individual has an annual income of over 50,000 dollars."
Which is using:

model = tf.estimator.LinearClassifier(
    model_dir=tempfile.mkdtemp(),
    feature_columns=base_columns + crossed_columns,
    optimizer=tf.train.FtrlOptimizer(learning_rate=0.1))

Questions:

1) Is there any sample with Logistic Regression for new TF estimator API.

2) Which one would be a good dataset to do Logistic Regression?


texts: How can I implement a deep/machine learning algorithm for this?

texts:
I have a known matrix, $\bf C$ of size $N\times B$. Each row has a fixed number of non-zero elements.

The demand vector is given by $d$ of size $1\times B$.

The supply vector $s$ of size $1\times B$ is given by

$s=\sum_{i=1}^T C_i\alpha_i$.

Here. $C_i$ is the $i$th row vector of known matrix, $\bf C$.

I have a large set of data for $d$ and corresponding $\alpha_i$s and $s$.

Now, I want a machine/deep learning algorithm that gives me $\alpha_i$s for a given $d$.

How can I implement a deep learning algorithm for this?


texts: What conclusion can I get when the variable is influenced by other but there isn't any correlation?

texts:
I am doing an analytic exploratory analysis.
If the target is a continuous variable and the attributes are all categorical (discrete values), in order to know if exist any influence on the target from the each attribute I am doing the ANOVA-test like this:
fvalue, pvalue = stats.f_oneway(df[y], df[x])
pvalue < 0.5

If that condition is true, there is a dependency between variables.
For all variables I get true dependency with ANOVA, but the values of the correlation are between -0.1 and 0.2.
What kind of conclusion can I do with this?
Can I say for example that if in the variables with negative correlation, I get new values in order to make that correlation a high positive value, then the target will get a higher value?
Any ideas?


texts: Proper evaluation method for recommendation system with implicit feedback?

texts:
I am trying to implement a recommendation system for a live-streaming website. Here "users" are simply the website users and "items" are streamers that they should watch. I collected data on which users follow which streamers -- however, following a streamer does not necessarily correspond to liking the streamer. Thus the setting is: I have binary implicit feedback (0 = no follow, 1 = follow) for 10k users, 2k items, and 200k interactions. I also have follow timestamps, but I have not tried to use this data yet. The models I'm using are primarily collaborative filtering, matrix factorization/latent factor models which are implemented in Implicit & LightFM (Python). My questions are: (i) how do I make a train/validation/test split, (ii) what metric is appropriate?
My understanding is that to make a train-test split we "mask" some of the observations in the interaction matrix $R$, call these observations $\{r_{ui}^t\}$ and then train our model on $R\setminus\{r_{ui}^t\}$. So we just treat the test observations as $0$ in training phase (?). Then for each user we score all of their items in the test set (or do we score all of their items in either the train or test set?), rank the items by their scores, and then calculate an evaluation metric. Please correct/validate me :) Here are my observations regarding my two main questions...
(i). In the paper [1], the authors make a train-test split based on time period. On the other hand, in [2], the authors use a leave-one-out strategy where they drop one positive user-item pair for each user. Finally, [3] generates a test set by using a random 10% sample of entries (rather than selecting random positive entries), although it isn't clear to me how this is favorable to just selecting random positive entries. I'm leaning towards the method in [2] for its simplicity and because there does not seem to be a consensus. Furthermore, none of these works mentioned using, or offered a method of making, an additional validation set to tune hyperparameters. Is it just a matter of repeating the split method twice?
(ii). For implicit feedback I understand that recall-based metrics are more important than precision. In [1,3] they use mean percentage ranking
$$\overline{rank} = \frac{\sum_{u,i} r_{ui}^t rank_{ui}}{\sum_{u,i} r_{ui}^t}$$
where $rank_{ui}$ is the percentile ranking for item $i$ in user $u$'s full ranking list and $r_{ui}^t$ is the implicit feedback rating for a test observation. This seems useful when $r_{ui}$ may take on many values and one would like to respect these ratings; however, for me $r_{ui} \in \{0,1\}$. Also, this uses the entire ranking list, while perhaps a cutoff-sensitive metric such as $MAR@K$ is better for my application (?). In [2] they only report $AUC$, probably because the BPR model optimizes for $AUC$. With their leave-one-out strategy they denote $S_\text{train}, S_\text{test}$ to be the sets of positive interactions and then compute
$$AUC = \frac{1}{|U|} \sum_u \frac{1}{|E(u)|} \sum_{i,j \in E(u)} \delta(\hat{r}_{ui} > \hat{r}_{uj})$$
where $|U|$ is the number of users and $E(u) = \{(i,j) | (u,i) \in S_{\text{test}} \wedge (u,j) \notin (S_\text{test} \cup S_\text{train})\}$ are the evaluation pairs for user $u$. That is, they calculate how often a positive interaction is rated higher than an unobserved interaction. This seems like it might be a favorable metric considering our interaction data is binary.
References
[1] Hu, Y., Koren, Y., & Volinsky, C. (2008, December). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263-272). IEEE.
[2] Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012). BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
[3] Johnson, C. C. (2014). Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, 27, 78.


texts: CNN accuracy and loss doesn't change over epochs for sentiment analysis

texts:
I am performing text classification as Good [1] or Bad [0]. The texts are preprocessed and converted to Vectors using Google Word2Vec. Further CNN architecture is used for training. I have roughly 13000 texts as Bad[0] and 5450 texts as Good[1] for training (making it a roughly 70:30%)
The issue starts when I realize I don't have enough compute power (2GB GPU). Hence I compromise and use 100 dimensions of word embeddings from Word2Vec (instead of the 300). After certain hyperparameter tuning in the CNN architecture, I am able to obtain a 30-35% precision, which I am happy with.
After months, I have an 8GB GPU in the server and I implemented the 300-dimensional word embeddings in Word2Vec and kept for training. Ideally, I should have obtained better results; instead, the loss and accuracy don't change with time for every epoch. Thus it predicts all texts as Bad[0].
Can you please help me identify the problem and If I am missing out anything here!

EDIT:

I would like to add some clarifications:
In the Linux server with GPU 1070Ti 8GB, I tried with three experiments in this order: a) 300-dimensional word embeddings b) 100-dimensional word-embeddings and c) 105-dimensional word embeddings
I have obtained no change in accuracy and loss for a) and c). However, for b), it is exactly the same results as I obtained with my local GPU(750Ti Nvidia 2GB). In short, its working fine for 100-dimensional in the server.

Now, since I can't assign 300d word vectors in my local GPU, I make up the same experiment as c) with 105d vectors in the local GPU to just check if there's any fault in code, and surprisingly it's giving around 30% precision much similar to earlier results.

I am having a hard time figuring out the issue in the server GPU as its working fine for 100d word vectors but fail to give proper predictions for other dimensional word-embeddings.

I am attaching some results which might make it a lot clear:

1.) Trained with 100d vectors in local(2GB) GPU
             precision    recall  f1-score   support

    0.0       0.70      0.83      0.76      1973
    1.0       **0.31**      0.17      0.22       850

avg / total       0.58      0.63      0.60      2823

2.)  Trained with 100d vectors in server (8GB) GPU
             precision    recall  f1-score   support

    0.0       0.70      0.77      0.73      1973
    1.0       **0.30**      0.23      0.26       850

avg / total       0.58      0.61      0.59      2823

3.) This is the same result for both 105d and 300d vectors trained in server GPU
             precision    recall  f1-score   support

    0.0       0.70      1.00      0.82      1973
    1.0       **0.00**      0.00      0.00       850

avg / total       0.49      0.70      0.58      2823

4.) Trained with 105d vectors in local(2GB) GPU
             precision    recall  f1-score   support

    0.0       0.70      0.83      0.76      1973
    1.0       **0.30**      0.18      0.23       850

avg / total       0.57      0.64      0.61      2823


texts: How do I deal with data that has only limited target values?

texts:
I'm currently working on a small project using the D1NAMO dataset (1). I want to predict the glucose level (that is given in the dataset) based on several features: accelerometer data, heartbeat (ECG) data, breathing data and some other features. I have several users (006, 007, 008 etc) that all have a few days of data.
The problem is that the target value (glucose) is measured only once in 5 minutes, whereas the other measurements are done every few milliseconds. That means that in my data I have many rows without the target value. For example, for one user (008) I have 214 rows where I have a glucose level and around 60.000 rows where glucose is NaN. I want to use a model without the notion of time, so that could be anything such as FFNN, CNN, Decision Trees (XGBoost), ensembles etc.
Snapshot of my data:

My questions:

How should I deal with this problem where I have only few target values? Should I summarize all values in a window of 5 minutes to match with the target value?
Should I make a model on the instance level or population level?

Thanks in advance for any replies.
(1) https://www-sciencedirect-com.vu-nl.idm.oclc.org/science/article/pii/S2352914818301059


texts: Implementation of BERT using Tensorflow vs PyTorch

texts: BERT is an NLP model developed by Google. The original BERT model is built by the TensorFlow team, there is also a version of BERT which is built using PyTorch. What is the main difference between these two models?


texts:
I have an assignment in which I have to provide a solution to measure the tread depth of a tire using a single rgb image.

I thought of two possible solutions:

Using CNN to measure the image similarity between a tire image and an image of a known depth.
Using a regression model to predict the tread depth.

Is there any other method I could try that can work over a large dataset?

Any suggestions would be helpful.


texts: Pre-trained models for finding similar word n-grams

texts:
Are there any pre-trained models for finding similar word n-grams, where n>1?
FastText, for instance, seems to work only on unigrams:
from pyfasttext import FastText
model = FastText('cc.en.300.bin')
model.nearest_neighbors('dog', k=2000)

[('dogs', 0.8463464975357056),
 ('puppy', 0.7873005270957947),
 ('pup', 0.7692237496376038),
 ('canine', 0.7435278296470642),
 ...

but it fails on longer n-grams:
model.nearest_neighbors('Gone with the Wind', k=2000)

[('DEky4M0BSpUOTPnSpkuL5I0GTSnRI4jMepcaFAoxIoFnX5kmJQk1aYvr2odGBAAIfkECQoABAAsCQAAABAAEgAACGcAARAYSLCgQQEABBokkFAhAQEQHQ4EMKCiQogRCVKsOOAiRocbLQ7EmJEhR4cfEWoUOTFhRIUNE44kGZOjSIQfG9rsyDCnzp0AaMYMyfNjS6JFZWpEKlDiUqALJ0KNatKmU4NDBwYEACH5BAUKAAQALAkAAAAQABIAAAhpAAEQGEiQIICDBAUgLEgAwICHAgkImBhxoMOHAyJOpGgQY8aBGxV2hJgwZMWLFTcCUIjwoEuLBym69PgxJMuDNAUqVDkz50qZLi',
  0.71047443151474),

or
model.nearest_neighbors('Star Wars', k=2000)
[('clockHauser', 0.5432934761047363),
 ('CrônicasEsdrasNeemiasEsterJóSalmosProvérbiosEclesiastesCânticosIsaíasJeremiasLamentaçõesEzequielDanielOséiasJoelAmósObadiasJonasMiquéiasNaumHabacuqueSofoniasAgeuZacariasMalaquiasNovo',
  0.5197194218635559),


texts:
Should the two sequential parts of a hurdle model (Response model and Expected value model) be tuned and evaluated independently, and then combines for the final results (Product of the probability to respond and the expected value)? Or should they be tuned and evaluated as a single model? What is the best approach to evaluated the overall/global performance of a two part hurdle model?
For example, if the two models are evaluated and tuned independently, then the hurdle model (i.e. probability to respond) will be evaluated based on classification metrics (e.g., ROC, F1, Recall, etc.) and the expected value model will be evaluated based on regression metrics (e.g., MSE, MAE, etc.). Once they combined, what is the best approach to evaluated the overall/global model performance?
Otherwise, if the two parts of the model should be combined in a single model to be tuned and evaluated, then can regression evaluated metrics be used to evaluated the overall/global model performance?


texts:
I’m trying to calculate probability of class 1. I’m using gradients boosting (catboost classifier)
Is it normal to have an equal rate of positive classes in every predict_proba() bucket? e.g.:

[Probability] : Positive rate

[0 - 0.25] : 17%

[0.26 - 0.50] : 17%

[0.51 - 0.75] : 17%

[0.76 - 1] : 17%

However when I 'm using logistic regression and convert WOE to the scores, the distribution of class 1 rate decreases with increasing score. e.g.:
[Score] : Positive rate

[100-150] : 17%

[151-200] : 15%

[200 - 250] : 13%

[250-300] : 10%


texts: Data science tools for easing the participation of a business into their scoring system

texts:
I'm a working in a small company. The company sells products on a website and they have a python script that runs everyday to attribute a score to each product based on a set of parameters (google analytics events, similar products popularity, price, etc).
The problem is that the scoring outcome is not satisfying, and requiring developers to edit this script arbitrarily, based on business people assumptions, is time consuming and not a proper way to achieve what the business needs.
The ideal solution in my point of view, would be to provide the business team a way to act on the scoring logic in a GUI, integrate what they could come up with, and display the immediate result of applied modifications so they can iterate until they find the right model.
In your experience, what kind of tool/service/library could easily allow non tech users to input formulas on a set of data, which could be synced and integrated in a python script ?


texts: I am new to CNN and want to try implementing YOLOv4 with a custom dataset for vehicles. As I understand it, k-means clustering is done to give labels to a set of data. I was going through some papers like Article 1 and I am confused by why they are using k-means clustering when they are using a public dataset that has already been labeled.


texts: How can I transpose a high dimensional dataset?

texts:
I have a (.csv) file with more than 35,000 rows and 100 columns, where the rows represent the attributes and columns represent the instances. In Excel the maximum number of columns is 16,384. Therefore, it is impossible to do this in one single Excel sheet. I need my dataset to be transposed in order to perform some machine learning algorithms in WEKA. Is there any tool allow such thing like this?

Also, is there a way to transpose (.arff) file in WEKA?


texts: TimeDistributed with different input / output sequence length

texts:
I'm looking into using TimeDistributed in my LSTM to see if it would improve the accuracy of my model.  I'll be honest, I'm still not 100% sure what the specific use case for TimeDistributed, I just thought I'd give it a try and see if it improved my accuracy. Aside from blind trial/error, why would someone use TimeDistributed?

Now, my next question is probably tied to my lack of understanding of exactly what TimeDistributed's specific use case is, but I understand that the output from TimeDistributed is a sequence, and that sequence is the same length as the input sequence because you use return_sequences=True on the final layer of the model (all of them in my case, as it's stateful).

In my case I'm inputting a sequence which is 100 length, and I'm then forecasting a sequence of length 3 from that input sequence, but I get this error:

  Error when checking target: expected time_distributed_1 to have shape
  (100, 3) but got array with shape (3, 1)

Am I right in assuming that you can only use TimeDistributed if your output sequence is the same length as the input sequence?

Many thanks


texts: Free exchange rate data API for all countries at at least daily frequency since at least 2000?

texts:
I thought it would be a very common dataset: world exchange rates at daily frequency for every currency pair (or each currency's rate vs USD so triangulation can fill in for every other rate)
But apparently it's not so easy to find. I tried the World Bank API (it's massive, and it may be there, but I can't find a specific endpoint that provides it), and the IMF (they have data explorers but no forex API AFAIK).
As this must be a very interesting time series dataset, I figured it would be easy to find, but I cannot locate it.
strong preference for a completely open/public API (one that doesn't require keys)


texts: Remove correlated features before or after splitting test and training set?

texts: I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it before would leak test information into the training set, but I could very well be missing something. I'm also concerned that the split of the data will impact the analysis of correlation between features, though maybe in practice the likelihood of this is low. Anyone have any insight? Thanks!


texts: Predicting on real test set gives only very high probability for 1 for a very unbalanced data

texts:
Excuse me for this brief description of the problem, as I'm very bound on time, I'll try to sum up as much as I can.

I have a multivariate time-series, that I trained using an RNN, there are periods and repeating time indexes, from 2013-01 to 2016-09, steps are months, by repeating, I mean various subsets ordered from January to December, many times for the same year, for hundreds of times, and I am predicting the next year knowing other features.
Predicting using Keras on real test data expecting the same shape as train data
I trained using LSTM, on 3 years, and trying to predict also repeating time-series for the year 2017. I used fixed batch size, and one last layer for binary target value so I used such a basic neural network:

model = Sequential()
model.add(LSTM(10, batch_input_shape=(12, train_X.shape[1], train_X.shape[2]), return_sequences=True, stateful=True, activation='sigmoid', inner_activation='hard_sigmoid'))
model.add(Dropout(0.2))
# model.add(LSTM(70))
# model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])

The batch size is 12 ( I chose) for 12 months, the target in train is very unbalanced with

pd.value_counts(test_y)
1.0 163781
0.0 5551
dtype: int64

Yet, I waited for some low probability for one in other words predicts of zeros.

res = model.predict(t_e_s_t, batch_size=12)

res
array([[0.9633749 ],
[0.79078996],
[0.99266464],
...,
[0.9891131 ],
[0.7582535 ],
[0.95778626]], dtype=float32)

All values of probability are above 0.5 and near 1 that means no probability for any entry to be zero. What could be wrong?

[Edit]

I added

from sklearn.utils import class_weight

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(train_y),
                                                 train_y)

and passed class_weights to fit method, still zero values under 0.5.


texts: I have no access to gpu due to usage limits?

texts:
I start running my code using google colab I first set the execution to GPU
and then I run my code for a training task using keras !after 1 hour I got a message saying I can't use GPU due to usage limitation !
Could anyone help understand the problem ??


texts: How to merge two csv files using multiprocessing with python pandas

texts:
I want to merge two CSV files with a common column using python pandas. With a 32 bit processor, after 2 gb memory, it will throw memory error. How can I do the same with multi processing or any other methods?

import gc
import pandas as pd
csv1_chunk = pd.read_csv('/home/subin/Desktop/a.txt',dtype=str, iterator=True, chunksize=1000)
csv1 = pd.concat(csv1_chunk, axis=1, ignore_index=True)
csv2_chunk = pd.read_csv('/home/subin/Desktop/b.txt',dtype=str, iterator=True, chunksize=1000)
csv2 = pd.concat(csv2_chunk, axis=1, ignore_index=True)
new_df = csv1[csv1["PROFILE_MSISDN"].isin(csv2["L_MSISDN"])]
new_df.to_csv("/home/subin/Desktop/apyb.txt", index=False)
gc.collect()


texts: Loss for CNN decreases and settles but training accuracy does not improve

texts: I am training a CNN with 2 conv layers 2 Relu and max pooling and 2 FC layers the last of which has only 2 units since it's a binary classification problem. The images are spatio-temporal continuous, basically those of stream functions of the atmosphere, obtained as contour plots in MATLAB from data generated out of a PDE solver. So, it is quite difficult to understand the difference between the two labels by just looking at the respective images. The labels are generated through some statistical algorithm that operates on the intensity of the pixels or simply the value of the matrices that make the contour plot. While training the CNN, I see that with a learning rate of .001, the loss decreases gradually and monotonically at all time where it goes down to 0.6 in the first 200 epochs (not suddenly, quite gradually, the slope decreasing as the value goes down) and settles there for the next 500 epochs. However, the training accuracy fluctuates around 50% and my testing accuracy is 50% too. Moreover, I see while looking at the false +ve and -ves that the CNN has got one label right all the time and the other label wrong all the time. I have used dropout of 0.5 and L2 regularization. I am using TensorFlow for all the programming. There is no imbalance in the training set between the TRUE and FALSE labels. I have 4500 TRUE example and 4500 False examples. I have randomly shuffled the training set and normalized them with 255 since I'm using RGB images. Testing is done on 200 TRUE examples and 200 False examples. Kernel size 5x5 in both layers , first one having 32 and second one having 64 filters and FC layer has 1024 neurons. I have played around with these parameters with no real improvement. Any suggestions would be really appreciated.


texts: Keras cNN Transfer Model: Reduce Final Model Size?

texts:
I'm working with multiple cNNs to be ran on mobile devices.  If I create these cNNs from scratch (black n white, 256x256), I'm able to produce a binary classification model of about 10mb, which is a size that would work great on a model device.

However, using transfer learning such as VG16 Imagenet gives my much better classification accuracy.  But, the model size is closer 100mb!  That is 10x as big as from scratch.  I have to take those grayscale images from scratch and convert them to 3 channel, which accounts for some of the size.  Also, freezing less layers, like 10 instead of 14 reduces the size, but very slightly.

Any suggestion on a way I can still leverage Keras Transfer learning and produce a much smaller model (like a way there a transfer learning model that is B&W)?

Here is my Transfer baseline model:

def baseline_model_func():
vgg_model = VGG16(weights='imagenet', include_top=False, input_shape=(input_dim, input_dim, 3))

for layer in vgg_model.layers[:10]:
    layer.trainable = False

x = vgg_model.output
x = Flatten()(x) # use global pooling
x = Dense(1024, activation="relu")(x)
x = Dropout(0.5)(x)
x = Dense(1024, activation="relu")(x)
predictions = Dense(1, activation="sigmoid")(x)
model_final = Model(inputs = vgg_model.input, outputs = predictions)
model_final.compile(loss = "binary_crossentropy", optimizer = optimizers.SGD(lr=0.0001, momentum=0.9),
                    metrics=["accuracy"])
return model_final

`

Here is my cNN from scratch baseline model:

def baseline_model_func():
model = Sequential()
model.add(Conv2D(filters=16, kernel_size=2, padding='same', activation='relu',
                        input_shape=(input_dim, input_dim, channel_numbers)))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=128, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=256, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Conv2D(filters=512, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))
model.add(Flatten())
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss = "binary_crossentropy", optimizer = "adam", metrics=["accuracy"])
return model

```


texts: To detect unauthorized access using outlier detection

texts:
I am working on project where my task is to find unauthorized access using any machine learning technique.

Let me clear my problem definition.

UserA access website using chrome browser from windows PC.
UserB access website using Internet Explorer from windows PC.
UserA never use safari to access website but all of a sudden it uses safari to access website from mac system.

Point 1 and 2 are normal logins but point 3 may be used by attackers because it is not matching with UserA behavior.

Thus i am am trying to implement an unsupervised model in python to first learn normal behaviors of all users for website access with other features such as -

username
country
service_key (unique identifier)
system
app_type (client, server, enterprise etc.)

Below I am sharing sample data-set of mine.

+==========+=========+=============+=========+==========+
| username | country | service_key | system  | app_type |
+==========+=========+=============+=========+==========+
| userA    | india   | e08fe2d     | windows | 2        |
| userA    | india   | e08fe2d     | android | 2        |
| usreB    | china   | bb15d36     | windows | 3        |
| userB    | india   | bb15d36     | windows | 3        |
| userB    | russia  | bb15d36     | mac     | 3        |

| userA    | usa     | e08fxxx     | mac     | 3        |
| userB    | china   | bb1xxxx     | ios     | 1        |
+==========+=========+=============+=========+==========+

Above 5 records are manually analyzed and marked as normal logs and collected from different users system. But the last 2 records are abnormal and should be detected as outlier because their is no match in features. Here to be note is that logins can have different country and system but it cannot have different username, service_key and app_type.

Can anyone suggest me to model this or can share blog or samples in python?


texts: Training Examples used in Stochastic Gradient Descent

texts:
Hi I was reading the difference between GD and SGD and found the below link.

[What is the difference between Gradient Descent and Stochastic Gradient Descent?

Based on this information I wanted to understand how would SGD train in the below scenario :

Say we have a dataset having 10000 rows and 45 predictors. Now, since SGD trains each predictor on one example(1 from 10000) , does that mean it uses only 45 examples in total to train the 45 predictors?

I'd really appreciate a clear explanation of this scenario.

Thanks!


texts: Handling Concat and Shift Feature in Pandas for Data Science

texts:
I am trying to USE Lag features AND Concat() and Shift() function,

    seies = Series.from_csv('sugar_price_2.csv', header=0, sep=';')

    In [25]: prices = DataFrame(series.values)

    In [26]: dataframe = concat([prices.shift(3), prices.shift(2), prices.shift(1), prices], axis=1)

    In [27]: dataframe.coloumns = ['t-2', 't-1', 't', 't+1']

    In [28]: print(dataframe.head(20))

       0       0       0     0
0      NaN     NaN     NaN  2800
1      NaN     NaN  2800.0  2800
2      NaN  2800.0  2800.0  2800
3   2800.0  2800.0  2800.0  2800
4   2800.0  2800.0  2800.0  2800
5   2800.0  2800.0  2800.0  2800

But The 't-2', 't-1', 't' Coloumn names aren't showing up .

Can anyone say what's wrong in my code...


texts: How to create a new column based on two other columns in Pandas?

texts:
I am searching for a way to create a new column in my data. I have tried using iterows() but found it extremely time consuming in my dataset containing 40 lakh rows. So here is what I want.
Consider I have 2 columns:
Event ID, TeamID ,I want to find the no. of unique TeamID under each EventID as a new column. In other words, I want to find the number of teams participating in each event as a new column.