texts: Implementation of RMS prop for linear regression

texts:
I'm trying to implement linear regression using Rms Prop optimizer from scratch.
Code:
EPOCHS = 100

w3 = tf.Variable(w_vector, dtype = tf.float32)
w4 = tf.Variable(0, dtype = tf.float32)
lr = 1e-5
beta = 0.9
epilson = 1e-7
momentum = 0.0
for epoch in range(1,EPOCHS+1):
    mom_w = 0
    mom_b = 0
    mean_square_w = 0
    mean_gradient_w = 0
    mean_square_b = 0
    mean_gradient_b = 0

    y_pred1 = tf.squeeze(tf.matmul(w3,x, transpose_a = True, transpose_b = True) + w4)
    dw3, dw4 = gradients_mse(x, y, y_pred1)

    # My eqautions for RMS prop
    mean_square_w = beta * mean_square_w + (1-beta) * dw3 ** 2
    mean_gradient_w = beta * mean_gradient_w + (1-beta) * dw3

    mom_w = momentum * mom_w + lr * (dw3/(tf.sqrt(mean_square_w + epilson - mean_gradient_w ** 2)))

    mean_square_b = beta * mean_square_b + (1-beta) * dw4 ** 2
    mean_gradient_b = beta * mean_gradient_b + (1-beta) * dw4

    mom_b = momentum * mom_b + lr * (dw4/(tf.sqrt(mean_square_b + epilson - mean_gradient_b ** 2)))

    w3.assign_sub(mom_w)
    w4.assign_sub(mom_b)

print('w3 : {}'.format(w3.numpy()))
print('w4 : {}'.format(w4.numpy()))

Output:
w3 : [[-1.2507935]]
w4 : 0.0033333366736769676

Now I create a single layer and single neuron neural network with no activation function. Assign the same weights in its neuron and use RMS prop as optimizer I get different final weights. However, this was not the case for sgd optimizer.
Code:
# using keras to get same results
def create_model():
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(units = 1,  name = 'd1', input_shape = (x.shape[1],)))

    model.compile(optimizer=tf.keras.optimizers.RMSprop(
    learning_rate=1e-5, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False),
                loss="mse")

    return model
model = create_model()

d1 = model.get_layer('d1')
d1_weights = [tf.constant(w_vector, dtype = tf.float32), tf.constant(np.array([0]), dtype = tf.float32)]
d1.set_weights(d1_weights)

model.fit(x, y, epochs = 100)

d1 = model.get_layer('d1')
print('w3 = {}'.format(d1.weights[0].numpy()))
print('w4 = {}'.format(d1.weights[1].numpy()[0]))

Output:
w3 = [[-1.2530397]]
w4 = 0.0010913893347606063

My gradients are calculate correctly for mse loss function. I have crosschecked them with tensorflows inbuilt gradient computation function gradient tape.
Code:
# Computing gradients
def gradients_mse(X, Y, Y_PREDS):
    DW1 = tf.matmul(X, tf.reshape(Y-Y_PREDS, (X.shape[0],1)), transpose_a = True) * (-2/X.shape[0])
    DW0 = (-2 / X.shape[0]) * tf.reduce_sum(Y - Y_PREDS)
    return DW1, DW0

The only thing that can go wrong in this implementation is I think calculation of mom_w and mom_b using incorrect equations.
x.shape = [10,1]
The default batch size is 32 so it will have no effects on weight updates. The same code gives perfectly matching output when I try to use simple gradient descent instead of RMS prop.


texts: Using Machine Learning techniques for text-analysis

texts:
I am analysing a bunch of tweets and I want to understand which political party the authors support.

I am using Mathematica but I am thinking to re-write my code in Python. If you have any suggestion, do not esitate.

Let say, for example that I have five parties labelled by a number: Party1, Party2,...,Party5 and I provide a list of tweets which are for sure associated to the parties. This list is roughly trainingList = {tweet1-> "Party1",tweet2->"Party2",...,tweet5->"Party5",tweetK->"Neutral"}. Moreover, I added a class "Neutral" because some tweets are not classifiable. Actually, I provide more than one tweet par party. In fact, the trainingList is much longer.

Notice that I remove stop-words from the tweets and I apply a stemming-words algorithm to simplify the expressions. Then, every tweets is a list of potentially-significant words.

I create the classifier function, providing a prior (the explicit for of the prior is not important)

c = Classify[trainingList, ClassPriors-> prior]

Then, I want to do some checks. I take a bunch of tweets belonging to other party-supporters and try

c[{word1,word2,word3,word4,word5,word7},"Probabilities"]

which gives me a list of probabilities $p_i$. For example,

<| 'Party1' -> p1, 'Party2' -> p2, 'Party3' -> p3, 'Party4' -> p4, 'Party5' -> p5, 'Neutral' -> p6 |>

Question1 Is there a way to understand how the algorithm associate these probabilites? More specifically, how each word is associated to a party

Question2 Is there a way to see which are the most frequent n-grams of words associated to a given party (class)?


texts: Object detection model's performance jumping up and down

texts:
I am training a model to detect buildings from satellite images in rural Africa. For labels, I use OpenStreetMap geometries. I use the Tensorflow Object Detection API and SSD Inception V2 as a model. I trained separate models on two different datasets (in different geographical regions). In one area, the model learns and quiet quickly converges:

When training the model in the other area, however, this happens:

Note that I use the exact same model, configuration, batch size, the training area is of the same size, etc. In the second case, the model's prediction change extremely rapidly and I am not able to see why. For example, here is a comparison of the predictions the model makes at 107k and 108k global steps:

I am quite new to deep learning and cannot understand why this might happen. I would be very grateful for any tips where to look, etc. It might be something simple that I am overlooking.
Let me know if I should provide more information


texts: Is there mathematical verification for TBPTT (truncated back propagation through time)

texts:
My question is in the title. I'm currently looking for a paper or academic reference to that algorithm, even in a novel framework.

Thank you for all possible replies,


texts:
I am a 35 year old IT professional who is purely technical. I am good at programming, learning new technologies, understanding them and implementing. I did not like mathematics at school, so I didn't score well in mathematics. I am very much interested in pursuing a career in Big Data analytics. I am more interested in Analytics rather than Big Data technologies (Hadoop etc.), though I do not dislike it.  However, when I look around in the internet, I see that, people who are good in analytics (Data Scientists) are mainly Mathematics graduates  who have done their PHds and sound like intelligent creatures, who are far far ahead of  me. I get scared sometimes to think whether my decision is correct, because learning advance statistics on your own is very tough and requires a of hard work and time investment.

I would like to know whether my decision is correct, or should I leave this piece of work to only intellectuals who have spend their life in studying in prestigious colleges and earned their degrees and PHDs.


texts: Create new variable in python for each iteration

texts:
I have two different dataframes like below.

And I need to make a dataframe like the following one.

Can you please help on how to that in python?

Thanks


texts: Why does an attention layer in a transformer learn context?

texts:
I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the output of an attention layer is a context vector. That is to say: what is it about the way that a transformer is trained causes the attention layers to learn context? What I would expect to see in the paper is a justification along the lines of "when you train a transformer using attention on sequence-to-sequence tasks, the attention layers learn context and here's why...". I believe it because I've seen the heatmaps that show that attention between related words, but I want to understand why that is necessarily the result of training a transformer.
Why couldn't it be the case that the attention layers learn some other features that happen to also be beneficial in sequence to sequence tasks? How do we know that they learn context, other than that's what we observe?
Again, I get the math and I know there are several posts about it. What I want to know is what about the math or the training process implies that the attention layers learn context.


texts: Multivariate - Time series data pattern changes


texts: How to reverse ReLU activation in deconvolution

texts: I recently came across Matt Zeiler's deconvolution (reversing convolution) paper . How is deconvolution able to reverse the rectified scalar output? From what I understand it sounds analogous to reversing the output of a linear / logistic regression model. How is it possible to go from a scalar to a vector?


texts:
if Word2Vec is nothing but a transformation of one-hot into a dense vector, why can't I just feed one-hot into LSTM (or for that matter sacrifice first dense layer, in any network that will end up using the embedding) and call it a day?

Why would I actually spend time pre-computing Word2Vec embeddings?
Yes, the resulting embedings are vectors that are clustered together if the words have similar meaning. But I presume a feed-forward classifier would figure out a good internal model anyway?


texts: Multicollinearity and impact of individual features

texts:
Assume the following scenario:

I have four features: $x_1$, $x_2$, $x_3$, and $x_4$
There are non-negligible multi-collinearity among the features.
I want to predict $y$ (response variable) with those 4 features.
I use simple multiple linear regression model: $y = a_1x_1 + a_2x_2 + a_3x_3 + a_4x_4$

Let's say that I want to understand the impact that different components of a chair have on the chair's retail price. For example:

$y\,\,\,$ = chair's retail price

$x_1$ = color of cushion used

$x_2$ = overall design of a chair

$x_3$ = strength of a chair

$x_4$ = softness of a chair

$x_1$ is completely independent, but other features are all somewhat impacted by the other features due to multicollinearity. For example, changing the color of cushion changes the design of a chair. Changing the design (structure) of a chair changes the strength of a chair.

I've heard that the analysis of regression coefficients are unreliable under severe multi-collinearity.

Assuming that the multiple regression model fits the chair price well, can I naively use each feature's regression coefficient to understand the impact of each feature on the response variable? If not, what technique should I use?

Ex 1: If I use a red cushion ($x_1$), I can increased the retail price by 3 dollars

Ex 2: If I use conference room style chair ($x_2$), I can increase the retail price by 12 dollars


texts: Neural network outputting same result for all inputs

texts:
I'm building an encoder-decoder neural network in Keras for sequence generation. The specific task is to try and change the styles of the text.

Both my encoder and decoders are LSTMs with latent dimensions of size 50. Also, the inputs to the network are one-hot encodings. I'm training the network using a couple of thousand data points and for over 100 epochs. The optimiser I use is rmsprop.

The accuracy and loss seem to decrease over time but during the inference stage, I find that every input sentence results in the same output. This is with a greedy sampling strategy.

I've played with different hyperparameters e.g. changing latent dims, batch-size, different optimisers etc. but to no avail. What strategies can I employ to assess whether this is a coding bug or an issue with the network?


texts: Different results every time I train a reinforcement learning agent

texts:
I am training an RL agent for a control problem using PPO algorithm. I am using stable-baselines library for it.

The objective of an agent is to maintain a temperature of 24 deg in a zone and it takes actions every 15 mins.The length of episode is 9 hrs. I have trained the model for 1  million steps and the rewards have converged. I assume that the agent is trained enough. I have done some experiments and have few questions regarding the training

I test an agent by letting it take actions from a fixed initial state, and monitor the actions taken by actions and states for an episode. When I test the agent multiple times, actions taken and states resulted are different every time. Why is this happening when the agent is trained enough?
I train an agent for 1 million steps. I train another agent for 1 million steps on the same environment with same step of hyperparameters and every thing else same. Both these agents converge. Now when I test these agents actions taken by these agents are not identical/similar. Why is this so?

Can someone help me with these.?

Thank you


texts: Maximum number of classes YOLO net can recognize

texts:
I'm trying to make a mobile app on image recognition(Computer Vision Application) . Does anyone know whether modern day smartphones have enough processing power/memory to recognize, say about 1 million classes from their real-time camera feed (30 fps) ?

(On-device inference or YOLO neural net concepts needed.)

What is the maximum number of classes it can reliably classify on a mobile device?

Some insight into figuring out computational loads/times in general would be helpful.


texts:
Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it.
Here the match is to find a value in a series that is made up of given set of keywords and one mandatory keyword.
Example :
Here’s the keywords table :
KW:
case,kws,mand_kw,Friendly name
1,full|name|nm|txt|dsc[0-9],full,Full Name
2,first|name|nm|txt|dsc|[0-9],first,First Name
3,last|name|nm|txt|dsc|[0-9],last,Last Name
4,legal|name|nm|txt|dsc|[0-9],legal,Legal Name
5,nick|name|nm|txt|dsc|[0-9],nick,Nick Name

DF
_id,field_name,field_friendly_name,purpose_of_use,is_included_in_report
1,FullName,,,
2,xyz,,,
3,custname,,,
4,FullNm,,,
5,FirstName,,,
6,abc,,,
7,FirstNm,,,
8,LastName,,,
9,LastNm,,,
10,LegalName,,,
11,LegalNm,,,
12,NickName,,,
13,pqr,,,
14,NickNm,,,

For each of kws from KW find match in field_name in DF that it is made up of only keywords specified in kws and not any other words and a mandatorily should have a mand_kw keyword.
For example :
Case1:
Find all field_name that is only made up of these set of keywords “Full|name|nm|txt|dsc|[0-9]” and mandatory contains keyword "full" if so, assign field_friendly_name of KW - “Full Name” to field_friendly_name in DF and update is_included_in_report as true.
If the value is not made of these keywords, then skip.
_id,field_name,field_friendly_name,purpose_of_use,is_included_in_report
1,fullname,Full Name,,TRUE
4,fullnm,Full Name,,TRUE
8,fullname74,Full Name,,TRUE
9,fulltxt,Full Name,,TRUE


texts:
I have a dataset consisting of sensor recordings about human movement. There are 22 classes of different movement like sitting or walking and 19 sensor values.

Each recording of a movement has about 1000 lines contained in a csv file.

My problem: I don't know how to present those recordings to a neural network (TensorFlow) so that it can be trained on the movement classes and even predict what was done in recording by getting those 19000 values.
I don't even know which Neural Network Model I should use and therefore need your help.


texts: CSR scipy matrix does not update after updating its values

texts:
I have the following code in python:

import numpy as np
from scipy.sparse import csr_matrix
M = csr_matrix(np.ones([2, 2],dtype=np.int32))
print(M)
print(M.data.shape)
for i in range(np.shape(M)[0]):
    for j in range(np.shape(M)[1]):
        if i==j:
            M[i,j] = 0
print(M)
print(M.data.shape)

The output of the first 2 prints is:

  (0, 0)    1
  (0, 1)    1
  (1, 0)    1
  (1, 1)    1
(4,)

The code is changing the value of the same index (i==j) and setting the value to zero.
After executing the loops then the output of the last 2 prints is:

  (0, 0)    0
  (0, 1)    1
  (1, 0)    1
  (1, 1)    0
(4,)

If I understand the concept of sparse matrices correctly, it should not be the case. It should not show me the zero values and the output of last 2 prints should be like this:

  (0, 1)    1
  (1, 0)    1
(2,)

Does anyone have explanation for this? Am I doing something wrong?


texts: How to incorporate features available only at training time into my model

texts:
I'm building classifier with a binary label.
In my particular dateset the training data has many more features than the test set, in fact most of them are not available in the test set (beyond my control).
How can I still utilize those features when training my model? It'll be a shame to just throw them away.
I thought of maybe trying to predict the training-only features as well as the labels, sort of like semi supervised learning.
Most of the missing features are categorical.


texts:
Let's say that I have two recommendation system models built,  Model A and Model B.
Now I track the performance of both the models for 5 days from 1st Jan to 5th Jan.
Each model has been assigned a different number of users to test the performance each day.
We capture metrics like as shown below.
Now, how can I know which model performs better?


texts: Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce

texts:
I understand Hadoop MapReduce and its features but I am confused about R MapReduce.

One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with Hadoop.

My doubt is:

R can do all stats, math and data science related stuff, but why R MapReduce?
Is there any new task I can achieve by using R MapReduce instead of Hadoop MapReduce? If yes, please specify.
We can achieve the task by using R with Hadoop (directly) but what is the importance of MapReduce in R and how it is different from normal MapReduce?


texts: Why modin.pandas takes more wall time than pandas?

texts:
All data scientist know that to scale the pandas to large dataset, we use modin.pandas instead of pandas. But when I tried to load a 500 MB csv dataset in my jupyter notebook in Ubuntu 16.04 to compare the performance of modin.pandas and simple pandas , it gives unexpected time duration of execution.

Simple pandas is taking less time, while modin.pandas is taking more time. Why?

Code:

import time
import pandas
%%time
# a=time.time()
pandas_csv_data = pandas.read_csv("order_products__prior.csv")
# print(time.time()-a)

import modin.pandas as pd
%%time
# a=time.time()
modin_csv_data =pd.read_csv("order_products__prior.csv")
# print(time.time()-a)

Below is the screenshot of code and its corresponding output.


texts: Why do neural networks use cost minimization of loss function and not profit maximization of profit function?

texts:
In neural networks, gradient descent is used to find optimum minimum value of cost function. Why this preference instead of finding maximum value of profit function?
What are the pros and cons of minimising cost function versus maximizing profit function?


texts: Are there any differences between Recurrent Neural Networks and Residual Neural Networks?

texts: I'm currently studying the former and have heard of the latter, and right now I'm thinking that they're the same. Are they?


texts: NARX Network Multi-Step Forecasting Question

texts:
I've been attempting to do multi-step ahead prediction with the NARX (Non-Linear with Exogenous Inputs) Neural Network. As I understand it, this network can be defined mathematically as follow:

$y(t+1) = F(y(n), ..., y(n-q+1), u(n), ..., u(n-q+1))$

Where the size of the tapped delay line memory is represented by $q$. So in order to do a 1-step ahead prediction, the network simply takes as input the current and past values of the exogenous function $u(n)$ as well as the current and past values of the time-series I am interested on $y(t)$.

Now suppose that I want to predict all the values from $y(t+1)$ to $y(t+10)$, how would I go about doing this? It seems that I need the current value of the exogenous time-series $u(n)$ at each time-step however this is not available to me. Am I missing something here? I would greatly appreciate if anyone can point me in the right direction.


texts: Comparison between cost functions to determine the "best" model?

texts:
I'm building an LSTM neural net for time series prediction (regression) and I am incorporating custom loss functions into training. I'm trying to determine which cost function (of 3 cost functions) gives the "best" model, in other words, trying to define what "best" means.
The 3 cost functions yield results that are all on different scales, and in addition 2 produce positive numbers while the last produces negative numbers.
I have 5 datasets and I am training one model per dataset. With 3 losses, this gives me 3 * 5 = 15 losses in total. On validation data, the results look something like the below.
         | loss1 | loss2 | loss3
==================================
data1    | 1.106 | 5.074 | -1.872
==================================
data2    | 1.067 | 2.390 | -1.903
==================================
data3    | 0.823 | 4.724 | -1.892
==================================
data4    | 1.157 | 4.809 | -2.233
==================================
data5    | 0.583 | 2.854 | -2.120
==================================
Average  |   x   |   x   |   x
==================================

My goal is to somehow compare how effective a model trained with loss1 is compared to a model trained with loss2 on predicting out of sample data.
One procedure I have tried is to standardise each loss: (loss - mean(loss)) / std(loss) across datasets, then take an average across datasets for each loss and check which is smallest. However with a small sample size of 5, I don't know if this is valid. The average, after standardizing, could be a simple average, or a geometric average, or a harmonic mean. If I apply this method to the table above, I get these results
                   |  loss1   |  loss2   |  loss3
========================================================
Simple Average     | -3.9e-16 | -3.9e-16 | -9.7e-16
========================================================
Geometric Average  | 0.82     | 0.93     | 0.90
========================================================
Harmonic Average   | 2.69     | 2.36     | 2.55
========================================================

We can see that depending on the type of average chosen, a different loss is considered best. For simple average loss3 is best, for geometric loss1 is best, and for harmonic loss2 is best... sort of confusing.
Is this a valid way to compare model performance across different loss functions, or is there another method that would be more suitable to determine the "best" model?
An alternative would be to look at an accuracy metric across models (the same accuracy metric for all losses and datasets) and check which loss yields the highest accuracy.


texts: Do I use actual data or data difference to train machine learning model?

texts:
I would like to predict tomorrows temperature :-).

But I'm unsure of the best approach. Do I simply drop data from the last x days, or do I try do drop data from the last x days in difference?

Last three days could look like: 21.3C, 22C, 21.9C

Where the difference is whatever previous, 0.7C, -0.1C

You would need more than 3 days, I'm just setting an data example. I think I also would need to insert season among other variables to complete the experiment. But I'm just asking for temperature here.

Have anyone checked this? While the temperature can be anything from -20 to 40 C with decimals (1, 0.x) makes up a lot of numbers! But with difference the range is maybe -5C to +5C, also with decimals (1, 0.x) but it beats the other range making the range smaller and therefor easier for ML to predict?

If difference is best, should I also do percentage difference since then I would take to account different climate has different swings?


texts: What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem?

texts:
I have a multi-class classification problem with class imbalance. I searched for the best metric to evaluate my model. Scikit-learn has multiple ways of calculating the F1 score. I would like to understand the differences.
What do you recommending when there is a class imbalance?


texts:
I have some issue, during package installation in R version 3.0.2 (2013-09-25) ubuntu machine:

install.packages("randomForest")

  Installing package into ‘/usr/local/lib/R/site-library’
       (as ‘lib’ is unspecified)
       Warning in install.packages :
       'lib = "/usr/local/lib/R/site-library"' is not writable

How to resolve it?


texts: Why you shouldn't upsample before cross validation

texts:
I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross validate when oversampling data using SMOTE technique.
I have created a model using AdaBoost algorithm and set the following parametres to be used in Grid Search:
    ada = AdaBoostClassifier(n_estimators=100, random_state=42)
    params = {
        'n_estimators': [50, 100, 200],
        'random_state': [42]
    }

According to the article, this is the wrong way to oversample:
    X_train_upsample, y_train_upsample = SMOTE(random_state=42).fit_sample(X_train, y_train)

    # cross-validate using grid search

    grid_naive_up = GridSearchCV(ada, param_grid=params, cv=kf,
                                 scoring='recall').fit(X_train_upsample,
                                                       y_train_upsample)
    grid_naive_up.best_score_

0.6715940782827282
    # test set
    recall_score(y_test, grid_naive_up.predict(X_test))

0.2824858757062147
Whereas the correct way to oversample is like so:
    from imblearn.pipeline import Pipeline, make_pipeline

    imba_pipeline = make_pipeline(SMOTE(random_state=42),
                                  AdaBoostClassifier(n_estimators=100, random_state=42))
    cross_val_score(imba_pipeline, X_train, y_train, scoring='recall', cv=kf)
    new_params = {'adaboostclassifier__' + key: params[key] for key in params}
    grid_imba = GridSearchCV(imba_pipeline, param_grid=new_params, cv=kf, scoring='recall',
                            return_train_score=True)
    grid_imba.fit(X_train, y_train);

    # How well do we do on our validation set?
    grid_imba.best_score_

0.29015614186873506
    # compare this to the test set:
    y_test_predict = grid_imba.predict(X_test)

0.2824858757062147
So, according to the article, the first method is wrong because when upsampling before cross validation, the validation recall isn't a good measure of the test recall (28.2%).
However when using the imblearn pipeline for upsampling as part of the cross validation, the validation set recall (29%) was a good estimate of the test set recall (28.3%). According to the article, the reason for this is:

When upsampling before cross validation, you will be picking the most
oversampled model, because the oversampling is allowing data to leak
from the validation folds into the training folds.

Can anyone explain to me simply how the oversampling allows data to leak into the validation and causes the overfitting? And why does this problem not occur in the imblearn pipeline?


texts: Implementation of a distributed data mining paper

texts:
I have a project about distributed data mining and I need to do some implementations, So I've searched and found this paper. The address of dataset is mentioned in the paper and I've downloaded it. For the process I should split the dataset into 10 smaller datasets. And the other task is using Weka4WS (weka for web services) for the process (For clustering part).

So my questions:
1. How can I split the dataset using python code?
2. What is Weka4WS and how should I use it, I mean the implementation (because it's not something like apps like the Weka itself)?


texts: Best clustering algorithm to identify clusters and determine the closet cluster each individual response is near?

texts:
I have a survey where each question is related to a different 'shopper' type (there are 5 types so 5 questions). Each question is either binary (True/False) or scale based.

IE:

1. Do you like to shop at our physical location store ? (True/False)

2. Do our discounts entice you to shop more? a. no b. maybe c. yes

For each response I convert the answer choice to a numerical value. So True becomes 1, answer choice 2C becomes 3 etc.

At this point, I am clueless as too what clustering algorithm to use so I can create clusters for each of the 'shopper' types and measures each individual survey response submitted to determine a single cluster closet to the responses given and label the response as that cluster.

IE. This individual that submitted the survey response is 'location conscience shopper type'

Open to any new method of analysis not just clustering


texts: Examples of "unusual"/non-trivial features that actually worked for improving model score

texts:
I have been working for a while in credit problems for classification and regression and on these problems I have had the necessity of improving already good performing models, for this when apparently almost domain knowledge features have been included, I have used some non-common sense features that actually worked for example:
Entropy of the binned distribution of payment history
Linear/quadratic trend of balance over time
Cosine of due balance

etc.
Do you have other examples of features that seemed "totally crazy" but that at the end improved your model's performance?
If so, Is there any theoretical reason that/those feature(s) worked in your problem?


texts: Different types of analysis which can be done with Webdata of users

texts:
I have rich customer data for site usage, particularly the web analytics data. Some of the data fields I have are:

Product Usage details
Geographic details
Session data: Capture what all they did during a session onsite, unique to timestamp
User details
Device used to login
Product looked at/ downloaded
Campaign data for promotion of a product
Search queries made on the on-site search bar
Results of those search queries

and so on.

I wanted to know what are the different types of analysis I can perform using this data, more inclined towards the Data science aspect of it. I have already thought of the following:

Churn Analysis
Campaign effectiveness
Find out which product is being sold the most and which at which geography
User behavior analysis
and such.

Could the community help me come up with more ideas which i can use to perform analysis on this data?

Thanks


texts: Difference of Activation Functions in Neural Networks in general

texts:
I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear.

It's reasonable that one differentiates between logical and linear type functions, depending on the desired binary/continuous output but what is the advantage of sigmoid function over the simple linear one?

ReLU is especially difficult to understand for me, for instance: what is the point to use a function that behaves like linear in case of positive inputs but is "flat" in case of negatives? What is the intuition behind this? Or is it just a simple trial-error thing, nothing more?


texts: LOF gives same number of outliers irrespective of parameters

texts: I am running lof algorithm for around 100k 2d points. Each time, I run the lof algorithm with different n_neighbours parameter, I get the same number of points as outliers. It's always 10% of the points as outliers. Is this how this algorithm is supposed to work? Why does it occur this way?


texts:
The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).

Why do we make use of GRU when we clearly have more control on the network through the LSTM model (as we have three gates)? In which scenario GRU is preferred over LSTM?


texts: Forward propagation in complicated neural networks

texts:
In a typical neural network you have a certain number of neurons in each layer, all neurons have incoming connections from each neuron in the previous layer (and visa-versa). This I have no problem figuring out how to propagate.

However when you have a network that doesn't have the neurons in layers (I'm looking into the NEAT algorithm); where neurons are just connected based on genetic algorithm (random creation - then testing of fitness) there isn't a simple way of figuring how to forward propagate, for example;

1 input neuron - connected to neurons 2 and 3
Neuron 2 is also connected to neuron 3
Neuron 3 is connected to the output.

Now in this case, if I just go through numerically then everything works out just fine. However with this subtle change;

1 input neuron - connected to neurons 2 and 3
Neuron **3** is also connected to neuron **2**
Neuron **2** is connected to the output.

The going through; 1, 2, then 3 would mean neuron 2 doesn't include the input from neuron 3. I can see that there's a brute force way through this, but is there a better, more efficient way?


texts: how to represent feature importance in xgboost in percentage?

texts:
I am looking for a way to represent the feature importance numbers in percentage. I read through articles and API documentation for XGboost in python and it gives me the feature importance score, which represents how significant a feature is in predicting the dependent variable. for example, these are the feature scores, that I get for my model.
{'f1': 63,
 'f2': 32,
 'f3': 79,
 'f4': 297,
 'f5': 371,
 'f6': 171,
 'f7': 308,
 'f8': 416,
 'f9': 90,
 'f10': 197,
 'f11': 141,
 'f12': 167}

Now, is there a way I can represent this in percentages, for example, for feature f1, 63 divided by something could give me these values in percentages?


texts:
I have a Keras Xception based model for gesture recognition. The accuracy of the model is around 60-70% for classifying 7 different gestures. The training dataset consists of 320x240 and 640x480 pixel images. Currently, I'm leaving the input_shape parameter of the model equal to default value of the Xception model in Keras, which is (299, 299, 3). I assume under the hood the network is rescaling all inputs to 299x299 pixels, which probably isn't a good approach.
My questions are:

Is the Xception model somehow optimized for the 299x299 image size? That is to say, should I aim to crop/pad the input to 299x299 pixels rather than change the model's configuration?
All usage examples I've seen so far have input width = height. Is there a reason to prefer square images?
If I don't use cropping/padding, I have two options for the input shape: rescale all images to 640x480 in a preprocessing step, or rescale all to 320x240. Is the 640x480 option likely to result in much better accuracy?


texts:
I am a newbie in machine learning but I have a coursework to create program that can extract some concrete features from the given text.

For example:
If I want to extract number of red apples and green apples, I will extract 3, 5 from "On that tree 3 red apples and 5 green apples grow" and 10, 1000 from "In the box there are 1000 green apples and 10 of red".
Actually the real examples can be more complicated like a extracting a car specification from the ad.

The concrete features are known before the data set processing.

Could you suggest what kind of algorithms I should use? Previously, I used only linear regression.


texts: How to cluster categorical and numerical data in the same dataset?

texts:
I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some unsupervised learning algorithm (K-Means clustering algorithm is currently used).

For example the Gender,None,Low,Medium,High,Breakfast,Lunch and Dinner columns are represented in binaries. eg: Gender represents 1 for male and 0 for female. While the rest of the above mentioned columns represents 0 for unavailibilty and 1 for availability. The Meal and the Exercise columns are also categorical but are not binary. For example in the Meal if it is breakfast then 1, lunch = 2, and dinner = 3. So how can we use this type of mixed dataset for clustering? Please ignore the Event column since it is the target column.

Further should we need to normalize the rest of the numerical data before adding to any sort of unsupervised learning algorithm? And how can we deal with those different types of categorical data? Your guidance is greatly appreciated.

Thanks.


texts: How to download BibTex raw dataset for deep learning (LSTM )?

texts:
I am trying to test my model on the benchmark text classification dataset "Bibtex". The Bibtex dataset is available in tf-idf vectors but LSTM works on sequences. Bag of Words is a vectorized representation of text as a vector of counts. Therefore, it is no longer a sequence of words. RNN takes sequences as an input.

How can I download the raw data of Bibtex for custom feature extraction?


texts: Why is performance worse when my time-series data is not shuffled prior to a train/test split vs. when it is shuffled prior to the split?

texts:
We are running RandomForest model on a time-series data. The model is run in real time and is refit every time a new row is added. Since it is a timeseries data, we set shuffle to false while splitting into train and test dataset.

We observed that there is a drastic change in scores when shuffle is True and when shuffle is false

The code being used is as follows

# Set shuffle = 'True' or 'False'
df = pandas.read_csv('data.csv', index_col=0)
X = df.drop(columns=['label'])
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05, shuffle=True)

count = 0
predictions = []

for idx in X_test.index.values:
    # Train the model on training data
    # print(count, X_train.shape, y_train.shape)
    rf = RandomForestRegressor(n_estimators = 600, max_depth = 7, random_state = 12345)
    rf.fit(X_train, y_train)

    predictions.append(rf.predict(X_test.loc[X_test.index == idx]))
    # print(len(predictions))

    X_train.loc[len(X_train)] = X_test.loc[idx]
    y_train.loc[len(y_train)] = y_test.loc[idx]
    count+=1

Initially, we thought the difference is due to covariance shift in the data. But that shouldn't affect this much for continuous fit

Here are screenshots for the prediction plot

With shuffle = False

With shuffle = True


texts: Converting a column to a DataTimeIndex and query by day


texts: How to prepare colored images for neural networks?

texts:
I have seen many examples online regarding the MNIST dataset, but it's all in black and white. In that case, a 2D array can be constructed where the values at each array element represent the intensity of the corresponding pixel. However, what if I want to do colored images? What's the best way to represent the RGB data?

There's a very brief discussion of it here, which I quote below. However, I still don't get how the RGB data should be organized. Additionally, is there some OpenCV library/command we should use to preprocess the colored images?

  the feature detectors in the second convolutional-pooling layer have
  access to all the features from the previous layer, but only within
  their particular local receptive field*

  *This issue would have arisen in the first layer if the input images were in color. In that case we'd have 3 input features for each pixel,
  corresponding to red, green and blue channels in the input image. So
  we'd allow the feature detectors to have access to all color
  information, but only within a given local receptive field.


texts: Why is the F-measure preferred for classification tasks?

texts:
Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is  generally used for (unsupervised) clustering tasks?

The F-measure is the harmonic mean of the precision and recall.

The G-measure (or Fowlkes–Mallows index) is the geometric mean of the precision and recall.

Below is a plot of the different means.

F1 (harmonic) $= 2\cdot\frac{precision\cdot recall}{precision + recall}$

Geometric $= \sqrt{precision\cdot recall}$

Arithmetic $= \frac{precision + recall}{2}$

The reason I ask is that I need to decide which average to use in a NLG task, where I measured BLEU and ROUGE ( where BLEU is equivalent to precision and ROUGE to recall). How should I calculate the mean of these scores?


texts: One Hot encoding for large number of values

texts:
How do we use one hot encoding if the number of values which a categorical variable can take is large ?

In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary features) in the training dataset which will immensely increase the complexity and hence the training time.

So how do we deal with such cases ?


texts: Tips on how to preprocess data and outliers for churn analysis

texts:
I am doing analysis on telecom churn dataset. I have 4617 observations and 17 variables. I am using Python. I have the following questions,

1) When I skewness and kurtosis for the normality test, two variables are not normally distributed (value exceeds 1). Should I do log transformation for those two variables or else for the entire dataset?

2) When I check the outliers using IQR method, close to 700 observations are outliers. I do not want to remove the outliers. Should I apply log transformation as well to address the outliers? Is it the correct way?

3) I checked google, they say you can cap the outliers by taking percentile values. Is it a good practice to address the outliers?

4) My final aim is to apply all classification algorithms. Can I do scaling after log transformation? Can we need to do scaling after log transformation? Tree models are ok without outliers but want to do for other models. My question is can we build the model after log transformation or we need to do scaling?

Please advise.


texts: How synthetic sampling of data helps classifier to predict better result?

texts:
I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem.

I have the idea how to create the synthetic example to bring noticeable accuracy on a given dataset.

I want to go into deep and understand how a classifier, especially SVM handle the data with the synthetic example to classify more accurately. It would also helpful to know for other techniques like boosting algorithms, random forest etc.

Any kind of guidance on above question will be very much helpful.

I have already asked this question on StackExchange here but didn't get any answer.


texts: Which supervised ML model to use for exam/grade prediction?

texts:
So I plan on making a mobile app that will let students predict their final grades based on their mock exam results. I can train my model with previous years results.
X: 5 mock results
Y: Final grade obtained
However, I have the issue that sometimes, or most the times, the user may be using the app whilst not having taken ALL the mock exams yet, they may want to see if they are on track and use it once they have done maybe 3 mocks. How should I go about this for choosing my supervised algorithm/model?


texts:
Sequential covering is a type of decision rule procedure that repeatedly learns a single rule to create a decision list (or set) that covers the entire dataset rule by rule.
Given a training dataset, we can find a set of rules that cover the whole data.
When it's time to predict, one goes through the rules till it finds one that fits the individual prediction.
What happens if there is no perfect rule for the individual prediction?
Does it predict the average of the class?


texts:
I am writing a publication about websites privacy. Quite a standard for such works [e.g., 1, 2] is to summarize categories of websites in the study, to either prove that the study is not biased, or the opposite that some privacy violations are skewed towards some categories.
I found that categories are usually IAB categories and there is a pleothora different APIs (BrandFetch, ClearBit, WebShrinker) offering resolution of website to this IAB category. However, these APIs seem to be meant for SEO analysts, with pricing fairly exceeding 1000$ for getting categories of Alexa/Tranco top 1M websites. They offer far more details (logo, Facebook link, and similar services for marketing analysis), while I want just an offline list of these IAB categories, at most business size and country would be useful.
The past works were not helpful. [1] refers to McAfee SmartFilter Internet Database (discontinued), [2] to SimilarWeb, another online service that does not seem to scale. Also Alexa lists by categories were discontinued in September 2020.
I feel like I am just not searching the right keywords. Maybe there is some plain text/csv file with all of this for top 1M websites that would solve all my issues. In parallel I am contacting SimilarWeb and WebShrinker if they would give me academic access for acknowledgement in the publication. Do you know source of such a list of top 1M websites with their IAB categories?
References
[1] Urban, Tobias, et al. "Beyond the front page: Measuring third party dynamics in the field." Proceedings of The Web Conference 2020. 2020.
[2] Trevisan, Martino, et al. "4 years of EU cookie law: Results and lessons learned." Proceedings on Privacy Enhancing Technologies 2019.2 (2019): 126-145.


texts: Compare similarities between two data frames using more than one column in each data frame

texts:
This work started by comparing two columns in each data set in pandas.
Previous research:here

A lot of results online show how to compare 2 data frames with 1 column
I'm trying to learn how to compare and extract similarities between two data frames (same & different sizes if possible) using more than 1 column in pandas.
sample input:
df1=pd.DataFrame([[1,None],[1,None,],[1,None],[1,'item_a'],[2,'item_a'],[2,'item_b'],[2,'item_f'],[3,'item_e'],[3,'item_e'],[3,'item_g'],[3,'item_h']],columns=['id','A'])
df2=pd.DataFrame([[1,'item_a'],[1,'item_b'],[1,'item_c'],[1,'item_d'],[2,'item_a'],[2,'item_b'],[2,'item_c'],[2,'item_d'],[3,'item_e'],[3,'item_f'],[3,'item_g'],[3,'item_h']],columns=['id','A'])

 df1
        id  A
    0   1   None
    1   1   None
    2   1   None
    3   1   item_a
    4   2   item_a
    5   2   item_b
    6   2   item_f
    7   3   item_e
    8   3   item_e
    9   3   item_g
    10  3   item_h

df2
    id  A
0   1   item_a
1   1   item_b
2   1   item_c
3   1   item_d
4   2   item_a
5   2   item_b
6   2   item_c
7   2   item_d
8   3   item_e
9   3   item_f
10  3   item_g
11  3   item_h

What I've tried  so far:
   1: df1[df1.A.isin(df2.A) & df1.id.isin(df2.id)]

   2: df1[   df1[['id', 'A']].isin(df2[['id', 'A']])  ]

The output I got for 1 is close to what I desire:
    id  A
3   1   item_a
4   2   item_a
5   2   item_b
6   2   item_f #this specific row is not desired in the output
7   3   item_e
8   3   item_e #this specific row was raised due to a duplicate in `df1`. It's permitted to show duplicates. Duplicates values are allowed in `df1` but not `df2`.
9   3   item_g
10  3   item_h

Desired output:
    id  A
3   1   item_a
4   2   item_a
5   2   item_b
7   3   item_e
8   3   item_e
9   3   item_g
10  3   item_h

What's not shown: Two data frames have 2500+ rows. df1 can have the same items associated with an id. No duplicate items for an id in df2.
My 2nd try 2: df1[   df1[['id', 'A']].isin(df2[['id', 'A']])  ] is definitely the wrong approach as its matching row and column in df1 to row and column df2 (This output is similar to equals(), I get values from df1 instead of True and NaN instead of False)
Any code, links, suggestions are appreciated.


texts:
I want my Mx2xN float tensor to become MxN complex tensor.
In this minimal example I supply the the 10x2 matrix which should become the vector [0+10j,1+11j,2+12k,...]
x = torch.stack([torch.arange(10.),torch.arange(10.,20)])
torch.view_as_complex(x.T)

gives the error RuntimeError: Tensor must have a last dimension with stride 1 because x.T.stride() is (1,10)
Trying
>>> x.T.as_strided(size=(2,10),stride=(10,1)).T.stride()
(1, 10)

Didn't get me any further, how do efficiently convert these?


texts: Datasource for regression model prediction : Machine Learning

texts:
I am trying to work using Amazon machine learning, but the data set that I have is small. The model I want to build is for regression based predictions and the domain I am aiming for the data set to belong is financial, say product price prediction, price and demand prediction based on macro/micro economic factor.

I am looking for a data set that contains factors that lead to variations in value of a product or commodity. For example, I would like to predict the value of 1 unit of polyester yarn after 1 year.  The factors which influence the yarn price are say - prices of crude oil, GDP of country,figures of IIP, inflation etc. So I would like a data set that contains the quotes of these factors on which the final price depends.

I find difficulty assembling this data myself because I don't know all the factors that contribute to a certain predictive price. Does anyone know of a dataset I can start with that sounds like it might contain these factors?


texts: Is it possible to load a model object to splunk to score?

texts: I have been working on some graph embedding deep learning algorithms for anomaly detection using some log files. My issue is that I need to use splunk found on a system different than my model make environment. I was wondering would it be possible to load a e.g. hdf5 or tf2 model object and make the scoring on splunk?


texts: Is hyperparameter tuning more affected by the input data, or by the task?

texts:
I'm working on optimizing the hyperparameters for several ML models (FFN, CNN, LSTM, BiLSTM, CNN-LSTM) at the moment, and running this alongside another experiment examining which word embeddings are best to use on the task of binary text classification.

My question is: should I decide on which embeddings to use before I tune the hyperparameters, or can I decide on the best hyperparameters and then experiment with the embeddings? The task remains the same in both cases.

In other words, is hyperparameter tuning more affected by the task (which is constant) or by the input data?


texts: If an NMT dataset is artificially enlarged by splitting sequences up, should it still train for the same number of epochs?

texts:
I have a neural machine translation dataset that looks something like this (letters represent words in an input sentence):

[Dataset A]
ABCDEFGHIJKLMNOPQRSTUVWXYZ
...
JFKJWKEFJMDNNFFKJWEKJFKWEJFK

We run Dataset A for 12 epochs, which is common for NMT.

Assume that I split the sentences from Dataset A in half to create a new dataset (which replaces the original one for training purposes):

[Dataset B]
ABCDEFGHIJKLM
OPQRSTUVWXYZ
...
JFKJWKEFJMDNNF
FKJWEKJFKWEJFK

Does it make sense to run Dataset B for 6 epochs (since no truly new information was added and the dataset is twice as large) or for 12 epochs?

For the sake of argument, what about these datasets? How many epochs "makes sense"?

[Dataset C]
Cut each sentence into phrases of two words each (AB, CD, EF...YZ)

or

[Dataset D]
Create multiple overlapping sentence phrases of length 10 from each sentence (ABCDEFGHIJ, BCDEFGHIJK, CDEFGHIJKL ...)


texts: What is the underlying pattern, or algorithm(s) used in these numbers?

texts: I have 11 lottery tickets (used) and I have discovered that in each ticket, the 3rd digit's value is +1 of the value of the 6th digit. I have 11 tickets, each ticket is composed of 16 digits. Would someone (anyone) help me find the logic used here, in creating the other digits and their interrelation? I am a complete noob in data analysis, and any help would be greatly appreciated. For those who would like to know, I incidentally discovered the certain pattern, and made me realize these numbers are not totally random and if I could find the underlying pattern I'd be able to predict. The excel file is here Excel file


texts: which machine learning technique can be used?

texts: I want to understand the intent of the customer using his search queries, let's say if a customer is interested in yoga pants, he can either search for yoga pants or exercise pants or workout tights etc. Is there a model that I can use to find out all the search keywords that can be related to yoga pants?


texts: Smith-Waterman-Gotoh Algorithm - how to determine an overall similarity percentage

texts:
Using the Smith-Waterman-Gotoh algorithm I want to get an overall similarity percentage between two sequences. What would be the best way to do this?

eg. comparing strings COELACANTH and PELICAN in this example gives a score of 4 with alignment:

ELACAN
ELICAN

How would I then go an determine the overall similarity percentage between COELACANTH and PELICAN based on this?


texts: How can I get total sum of each group by using pandas

texts:
I have a dataframe shaped like below

size ind weight weighted_hour
1    A    17       40
1    B    20       41
1    C    19       42
2    A    50       30
2    B    20       10
2    C    13       16
3    A    17       19
3    B    21       23
3    C    23       30

and I want to insert some sum of criteria like below

size ind     weight weighted_hour Total_WH
1    A       17       40          57
1    B       20       41          61
1    C       19       42          61
1    Total   56       123         179
2    A       50       30          80
2    B       20       10          30
2    C       13       16          29
2    Total   83       56          139
3    A       17       19          36
3    B       21       23          44
3    C       23       30          53
3    Total   61       72          133

I prefer to use pandas but if someone can suggest more efficient method
It's good either.
Give me some advice. Thank you!


texts: What is an intuitive explanation for the Importance Weighted Autoencoder?

texts: I have been reading a paper by Burda et al. on Importance Weighted Autoencoders(IWAE) but I can't quite grasp what they mean by sampling the terms h1...hk. Do they mean you have separate models from which you sample and then average over?


texts:
I am trying to do a simple ML re-sampling approach after the train-test split. However when I do this, it throws the below error. Can you please help me understand what this error is about?

KeyError: 'Only the Series name can be used for the key in Series dtype mappings.'

The code is given below:

# split into training and testing datasets
from sklearn.model_selection import train_test_split
from sklearn.utils import resample
from imblearn.over_sampling import SMOTE
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, random_state = 2, shuffle = True, stratify = y)
print("Number transactions X_train dataset: ", X_train.shape)
print("Number transactions y_train dataset: ", y_train.shape)
print("Number transactions X_test dataset: ", X_test.shape)
print("Number transactions y_test dataset: ", y_test.shape)
print("Before OverSampling, counts of label '1': {}".format(sum(y_train==1)))
print("Before OverSampling, counts of label '0': {} \n".format(sum(y_train==0)))

sm = SMOTE(random_state=2)
X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())   # error is thrown here

print('After OverSampling, the shape of train_X: {}'.format(X_train_res.shape))
print('After OverSampling, the shape of train_y: {} \n'.format(y_train_res.shape))

print("After OverSampling, counts of label '1': {}".format(sum(y_train_res==1)))
print("After OverSampling, counts of label '0': {}".format(sum(y_train_res==0)))

Here is the full error message:

KeyError                                  Traceback (most recent call last)
<ipython-input-216-af83b63865ac> in <module>
      3
      4 sm = SMOTE(random_state=2)
----> 5 X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())
      6
      7 print('After OverSampling, the shape of train_X: {}'.format(X_train_res.shape))

~\AppData\Local\Continuum\anaconda3\lib\site-packages\imblearn\base.py in fit_resample(self, X, y)
     86         if self._X_columns is not None:
     87             X_ = pd.DataFrame(output[0], columns=self._X_columns)
---> 88             X_ = X_.astype(self._X_dtypes)
     89         else:
     90             X_ = output[0]

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors, **kwargs)
   5863                     results.append(
   5864                         col.astype(
-> 5865                             dtype=dtype[col_name], copy=copy, errors=errors, **kwargs
   5866                         )
   5867                     )

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors, **kwargs)
   5846                 if len(dtype) > 1 or self.name not in dtype:
   5847                     raise KeyError(
-> 5848                         "Only the Series name can be used for "
   5849                         "the key in Series dtype mappings."
   5850                     )

KeyError: 'Only the Series name can be used for the key in Series dtype mappings.'


texts: Use LightGBM or FFM - imbalanced dataset

texts:
I have a highly imabalanced dataset but one that is not sparse. In train there are 1328 positives out of 104000. In validation and test set there are 400 positives out of 103000
Which algorithm should I experiment with: Boosting (LGBM/XGBoost) or FFM (field aware factorization machines)?


texts: Learning time of arrival (ETA) from historical location data of vehicle

texts:
I have location data of taxis moving around the city sourced from: Microsoft Research
Overall it has around 17million data points.

I have converted the data to JSON and filled up mongo. A sample looks like this:

{'lat': 39.84349, 'timestamp': '2008-02-08 17:38:10', 'lon': 116.33986, 'ID': 1131}
{'lat': 39.84441, 'timestamp': '2008-02-08 17:38:15', 'lon': 116.33995, 'ID': 1131}
{'lat': 39.8453, 'timestamp': '2008-02-08 17:38:20', 'lon': 116.34004, 'ID': 1131}
{'lat': 39.84615, 'timestamp': '2008-02-08 17:38:25', 'lon': 116.34012, 'ID': 1131}
{'lat': 39.84705, 'timestamp': '2008-02-08 17:38:30', 'lon': 116.34022, 'ID': 1131}
`{'lat': 39.84891, 'timestamp': '2008-02-08 17:38:40', 'lon': 116.34039, 'ID': 1131}
{'lat': 39.85083, 'timestamp': '2008-02-08 17:38:50', 'lon': 116.3406, 'ID': 1131}

It consists of a taxiID - ID field, timestamp of its latitude and longitude combination.

My question is: I want to use this data to calculate estimated time of arrival(ETA)

So far, I am doing it a crude way by querying mongoDB with aggregation. It is totally inefficient.

I am looking at some sort of learning algorithm where the historical data can be used to train it. In the end, given two points, the algorithm should traverse the possible route by referring historical data and give an estimate of time.
Calculating time estimate is not a problem at all if I get the array of JSON documents between the points. But, getting those right arrays is.

Any pointers in this direction will be very helpful.


texts: Time series classification of different amount of objects / Classification based on trajectory

texts:
My question is a rather open one, I have a problem that I am looking for solutions or problems that are alike. I know what format the data (which I do not have yet) will take. My goal is to find datasets and models that are similar to this task.
The data is of the following format:

Over the course of time there are multiple objects, of which I each have features f that include position, size, no image features (but possibly additional information). All objects have the same amount of features. For training I will also know the class (3 classes and one background class), and hopefully its instance (not yet sure whether I will be provided with this information). Each color corresponds to one instance in the table.
During inference I will only be given the features over time, but not matched to their instances. I want to know as soon as possible (after few time steps), which class the objects belong to, which I believe can be done based on the trajectory.

What kind of problems are close to this?
I thought of cell/molecule trajectory classification, movements of aeroplanes/boats.
What kind of techniques/models should I look for?
I thought of RNN/LSTM style ones, but yet am unsure about how well this works if the instances are not matched.

I am happy about any hints and advice. Thank you in advance.


texts: What is the best way to implement skip connections in a Dense Network in Keras?

texts:
I want to use skip connection in a Dense model and I chose the following implementation of the idea:

That is I use a Dense layer with linear activation to map the input into a dimension that will allow me to add it with the activation of the latest layer.

Is this the most effective way to implement the idea of skip-residual connections in a dense network?


texts: Sample Data Input Format for Tensorflow Saved Model

texts:
I'm new to tensorflow and I've served the saved model to Google AI Platform models. However, I am having issues with the format of sample input data. Can you please guide how should I format the data input based on the requested format below? Thanks in advance.

To request an online prediction, it is required to input data instances as a JSON object as follow.

{
  "instances": [
    <value>|<simple/nested list>|<object>,
    ...
  ]
}

below is part of the outputs from $saved_model_cli --dir path /home/.. --all

Defined Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          DType: list
          Value: [TensorSpec(shape=(None, 3), dtype=tf.float32, name='a_xf'), TensorSpec(shape=(None, 2), dtype=tf.float32, name='b_xf'), TensorSpec(shape=(None, 8), dtype=tf.float32, name='c_xf'), TensorSpec(shape=(None, 12), dtype=tf.float32, name='d_xf'), TensorSpec(shape=(None, 4), dtype=tf.float32, name='e_xf'), TensorSpec(shape=(None, 16), dtype=tf.float32, name='f_xf'), TensorSpec(shape=(None, 26), dtype=tf.float32, name='g_xf'), TensorSpec(shape=(None, 4), dtype=tf.float32, name='h_xf'), TensorSpec(shape=(None, 2), dtype=tf.float32, name='i_xf'), TensorSpec(shape=(None, 11), dtype=tf.float32, name='j_xf'), TensorSpec(shape=(None, 6), dtype=tf.float32, name='k_xf'), TensorSpec(shape=(None, 2), dtype=tf.float32, name='l_xf')]
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None


texts: DNN loss gets smaller but accuracy stays the same

texts:
I am learning a DeepNN to choose between three decisions in a simulation. Therefore, I can run the simulation as often as I want and can generate as many samples as I want. Based on this tutorial (https://pythonprogramming.net/openai-cartpole-neural-network-example-machine-learning-tutorial/), I generated a learning dataset by running 10.000 simulations and saving the simulations with the 10% highest rewards.

Making use of this training data, I try to learn a DNN. However, during the learning phase, the loss of the DNN decreases while accuracy stays constant. Further, accuracy is about 0.33, which is basically random guessing when having 3 options to choose from. In consequence, when applying my DNN to yet unseen examples, it performs about as good as the best strategy used solely (so it seems to learn at least what strategy is best to use when applying only one predefined strategy. And that is exactly what it does).

The thing is: In my learning dataset, each strategy is used in about 1/3 of the cases, so in the learning dataset, all decisions are made about identical often. How is this possible?

I already experimented with different layouts of the DNN. Currently, it is defined as:

model = tf.keras.models.Sequential([
            tf.keras.layers.Flatten(input_shape=(10,1)),
            tf.keras.layers.Dense(128, activation='relu'),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(256, activation='relu'),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(128, activation='relu'),
            tf.keras.layers.Dense(3, activation='softmax')
    ])
model.compile(optimizer='adam',
            loss='sparse_categorical_crossentropy',
            metrics=['accuracy'])

The output of the learning phase looks like:

Train on 10777 samples
Epoch 1/5
10777/10777 [==============================] - 2s 150us/sample - loss: 6.9044 - accuracy: 0.3320
Epoch 2/5
10777/10777 [==============================] - 1s 84us/sample - loss: 1.5840 - accuracy: 0.3340
Epoch 3/5
10777/10777 [==============================] - 1s 94us/sample - loss: 1.1936 - accuracy: 0.3383
Epoch 4/5
10777/10777 [==============================] - 1s 96us/sample - loss: 1.1279 - accuracy: 0.3367
Epoch 5/5
10777/10777 [==============================] - 1s 82us/sample - loss: 1.1144 - accuracy: 0.3327


texts: Dataframe has no column names. How to add a header?

texts:
I am using a dataset to practice for building a decision tree classifier.

Here is my code:

import pandas as pd
tdf = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data', sep = ',', header=0)
tdf.info()

The column has no name, and i have problem to add the column name, already tried reindex, pd.melt, rename, etc.

The column names Ι want to assign are:

Sample code number: id number
Clump Thickness: 1 - 10
Uniformity of Cell Size: 1 - 10
Uniformity of Cell Shape: 1 - 10
Marginal Adhesion: 1 - 10
Single Epithelial Cell Size: 1 - 10
Bare Nuclei: 1 - 10
Bland Chromatin: 1 - 10
Normal Nucleoli: 1 - 10
Mitoses: 1 - 10
Class: (2 for benign, 4 for malignant)

Thanks,


texts: How to animate growth of a social network?

texts:
I am seeking for a library/tool to visualize how social network changes when new nodes/edges are added to it.

One of the existing solutions is SoNIA: Social Network Image Animator. It let's you make movies like this one.

SoNIA's documentation says that it's broken at the moment, and besides this I would prefer JavaScript-based solution instead. So, my question is: are you familiar with any tools or are you able to point me to some libraries which would make this task as easy as possible?

Right after posting this question I'll dig into sigma.js, so please consider this library covered.

In general, my input data would be something like this:

time_elapsed; node1; node2
1; A; B
2; A; C
3; B; C

So, here we have three points in time (1, 2, 3), three nodes (A, B, C), and three edges, which represent a triadic closure between the three considered nodes.

Moreover, every node will have two attributes (age and gender), so I would like to be able to change the shape/colour of the nodes.

Also, after adding a new node, it would be perfect to have some ForceAtlas2 or similar algorithm to adjust the layout of the graph.


texts: I know next to nothing about Data Science. Where should I start? Where do you recommend I should take courses (hopefully free)? Or even a book.


texts: Matrix properties and machine learning/data mining

texts:
I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. When we are doing, for example, linear regression, we need to calculate some of those properties, and fit them into the equation.

So my question is, my question is about the intuition behind these matrix properties, and their implications in the ML/DM literature.

If anyone could answer, can you teach me what is the importance of eigenvalue, positive semi-definite matrix, and column independence for ML/DM. And possibly, other important matrix properties you think important in study the dataset, and why.

I'd be really appreciated if someone can answer this question.


texts: Do numerical inaccuracies play any role in training neural networks?

texts:
Are there publications which mention numerical problems in neural network optimization?

(Blog posts, articles, workshop notes, lecture notes, books - anything?)

Background of the question

I've recently had a strange phenomenon: When I trained a convolutional network on the GTSRB dataset with a given script on my machine, it got state of the art results (99.9% test accuracy). 10 times. No outlier. When I used the same scripts on another machine, I got much worse results (~ 80% test accuracy or so, 10 times, no outliers). I thought that I probably didn't use the same scripts and as it was not important for my publication I just removed all results of that dataset. I thought I probably made a mistake one one of the machines (e.g. using different pre-processed data) and I couldn't find out where the mistake happened.

Now a friend wrote me that he has a network, a training script and a dataset which converges on machine A but does not converge on machine B. Exactly the same setup (a fully connected network trained as an autoencoder).

I have only one guess what might happen: The machines have different hardware. It might be possible that Tensorflow uses different algorithms for matrix multiplication / gradient calculation. Their numeric properties might be different. Those differences might cause one machine to be able to optimize the network while the other can't.

Of course, this needs further investigation. But no matter what is happening in those two cases, I think this question is interesting. Intuitively, I would say that numeric issues should not be important as sharp minima are not desired anyway and differences in one multiplication are less important than the update of the next step.


texts: How to Read Photos for Deep Learning on Google Cloud Platform Compute Engine VM Instance, Through Jupyter Lab

texts:
I've set up a project on GCP with a Compute Engine VM and Storage Bucket.
Access Scopes set to Allow full access to all Cloud APIs
Have set a default Region and Zone
I believe I have completed SSH authorization but not 100% sure given the documentation I've read.

I have ~100,000 photos on both my local machine that I managed to upload to a bucket and am now trying to run my algo using the VM connected jupyter lab.  My first thought was to create a list object from the bucket object.  After 90 min, I tried to do the same but with my local folder.  After failing at that, I attempted to mount the bucket on the VM instance though, I failed at that too (i'd rather not go this rout anyway since I'd have to copy the files which seems nuts).

i've used this with different commands !gsutil ls gs://my_bucket

from google.cloud import storage

bucket_name = "my_bucket"

storage_client = storage.Client()

bucket = storage_client.get_bucket(bucket_name)

I'm new to all this, have read through more GCP documentation pages than I can count and you would be disgusted with me about how much time I've given this...please help!


texts: When projecting data with UMAP, should I use only the samples I need projected or the entire dataset?

texts: I have a neural network that maps my data samples to a 64-dimensional embedding. I wish to visualize a few of these embeddings (between 30 and 600) through a 2-dimensional projection, and I plan to use umap to do that. Would providing more embeddings sampled from the dataset along with the ones I want to project help the algorithm to identify the manifold and improve the quality of projection?


texts: ValueError: Input 0 of layer sequential_1 is incompatible with the layer:

texts:
Input 0 of layer sequential_1 is incompatible with the layer: expected axis -1 of input shape to have value 35 but received input with shape [None, 1]

state and action are just a number.. like 1,2 ,3.. but i define state as array 1*35 that all elements is zero except a current state is one like.. state=[0,0 ,0,1 ,0 , ...]
but i dont know what is the my mistake?
    model = Sequential()

    state_shape = self.M_VL_state_size    #35

    model.add(Dense(255, input_dim=state_shape, activation="relu"))

    model.add(Dense(255, activation="relu"))

    model.add(Dense(255, activation="relu"))

    model.add(Dense(self.M_VL_action_size)) #self.M_VL_action_size=35

model.compile(loss="mean_squared_error", optimizer=Adam(lr=self.learning_rate))#Loss function: Mean Squared Error
# Optimaizer: Adam (Feel free to check other options)
print("mode.summery=",model.summary())
return model
def act(self, state):
    self.epsilon += self.epsilon_decay
    self.epsilon = max(self.epsilon_min, self.epsilon)
    if np.random.random() < self.epsilon:
         print("random.randint(1, self.M_VL_action_size - 1)", random.randint(1,
            self.M_VL_action_size - 1))

         return random.randint(1, self.M_VL_action_size - 1)

    return np.argmax(self.model.predict(state)[0])

def remember(self, state, action, reward, new_state, done):
    self.memory.append([state, action, reward, new_state, done])
    print("self.memory=",self.memory)
    print("len(self.memory)=",len(self.memory))

def replay(self):
    batch_size = 32
    if len(self.memory) < batch_size:
        return
    samples = random.sample(self.memory, batch_size)   # random.sample()

   #state=[0----34]-- list of all state...

    for sample in samples:

        state, action, reward, new_state, done = sample
        print("state==",state)

        target = self.target_model.predict(state)

        print("target=",target)
        if done:
            target[0][action] = reward

        else:
            Q_future = max(self.target_model.predict(new_state)[0])
            print("Q_future=",Q_future)

            target[0][action] = reward + Q_future * self.gamma

        self.model.fit(state, target, epochs=1, verbose=0)

 def target_train(self):
    weights = self.model.get_weights()

    target_weights = self.target_model.get_weights()

    for i in range(len(target_weights)):
        target_weights[i] = weights[i] * self.tau + target_weights[i] * (1 - self.tau)
    print("target_weights=",target_weights)
    self.target_model.set_weights(target_weights)

 def save_model(self, fn):
     self.model.save(fn)


texts: Complete a Hungarian stem to a real word

texts:
I'm quite new to the NLTK package of Python and to NLP too (I usually work in R but for NLP purposes and scraping maybe Python is more able).

I scrap articles from Hungarian newsportals and want to make a wordcloud out of it to show what are the current trending news topics. First I filter out stopwords and then stem the remaining words. (nltk has Hungarian stemmer) So I'm able to make a frequency table which can be the base of the wordcloud. My problem comes afterwards because stems are usually meaningless chunks (and not lemmas) of real words. I want to somehow complete the stem to a real word.

My first idea was to assign the most common word or the shortest one (or some combination of this 2 rules) to the stem and represent that in the wordcloud.

Is there a better solution for stem completion or should I follow a different workflow?


texts: Two-class model with predicted scores needed - classification or regression approach

texts:
In my problem, step one is to build a model to classify cases as one of True or False (1 or 0 could also be used obviously). Once the optimum model is found, step two is to retrieve probabilities for these predictions and use these to work out the optimum threshold (based on an accuracy measure) to classify my data thereafter.

So I know for step two I need to use a classification accuracy measure e.g. f1 score to work out the optimum threshold however, I am debating on the best approach for step one. My initial thought was to do a two-class classification model. The model I'm using is a Radial SVM implemented in R. My issue with this however, is using caret, how I get predicted scores once the optimum model is found with classProbs=T (full code below). I don't know if this is ideal as I read somewhere (can't find link now sorry!), to do this, caret actually runs a second model which doesn't seem like a great idea to me. The only information on this is the package documentation seems to be

  classProbs a logical; should class probabilities be computed for classification models (along with predicted values) in each resample?

I have also searched SO for another way to get predicted values using caret but have had no success.

My other idea was to use a regression model for step one using, 0 and 1 as the only y values and then perhaps implementing a custom error function that allows for predicted values outside of 0 and 1. My thought here is that I don't necessarily want to penalise a model if the prediction is above 1 and the actual is 1.

I'm not sure which approach is better or if there is a third way I'm not thinking of, any help/suggestions would be great! I think the first approach is probably better if the class probabilities calculation is not done through using a separate model so if anyone has found further documentation on what goes on with this setting that would also be great!

svm_tests <- train(x = x_train,
                   y = y_train,
                   method = "svmRadial",
                   scale=F,
                   tuneGrid = expand.grid("C"=0.1,"sigma"= 0.05),
                   trControl = trainControl("repeatedcv",  #Do 5 fold cv, repeated 5 times with different seeds
                                            repeats = 5,
                                            number = 5,
                                            seeds = seeds,
                                            summaryFunction = m.c.c,
                                            classProbs=T, #Used to get probabilities when using predict
                   ))

scores_svm <-  predict(svm_tests,
                       x_valid,
                       type = "prob")


texts: Standardizing data from different distributions

texts:
I have energy data from different homes (~200), and I wish to merge all the energy data to create a combined dataset.

Since the individual homes have different power consumption, I get different distributions for every house. What will be the ideal strategy for normalizing the dataset?

The original energy data distribution of individual homes show lognormal characteristics.

Using log transform the energy distributions resembles mixtures of Gaussians.


texts: LSTM regression bias increases when targets go close to 0

texts:
I've build a LSTM model for time series forecasting. Results are not bad, with a mean normalized error of 7%. However, this normalized bias shows a clear pattern: The closer to 0 the value to predict, the higher the bias, as depicted in the figure below:

NOTE 1: True and Pred values are scaled in the figure for clarity

NOTE 2: The bias is not constant, the un-normalized bias (or bias) shows the same pattern

The model:

* 1 LSTM layer with 256 hidden units and no peep-holes
* Dropout in LSTM with keep_prob = 0.8
* One Dense Layer after the LSTM with 128 units with relu activation
* One Dense Layer after the first dense with 1 unit (predictor)
* Adam optimizer, with learning rate = 0.001
* Loss: Mean Squared Error

Any hint?


texts: Aggregate results in time series forecasting

texts:
I'm working on time series forecasting with some sales data, with no exogenous variables, only sales per day.

After some analysis, including seasonal decompose, plot autocorrelation and parcial autocorrelation, etc. I've tried some models, including ARIMA (with pmdarima) as baseline and some others like LSTM and adaBoostRegressor.

All of them work reasonably good with other datasets, but with my data all of them performs very poorly.
Anyway the daily data looks very random, but if I predict for example a month, the rate between all the (daly) predictions aggregated and all the (daly) real values aggregated looks very close to 0.

My question is:

Does make any sense to predict daly data and then aggregate by week, or month or quarter, etc.?

For example, if I want to predict 1 week, first predict daly data for 7 days and then aggregate all.

My report for baseline model AUTOARIMA looks like this (every new prediction is made with previous predictions, not all at once over test data):

As you can see, the aggregated results for one month are very close, so for me makes sense to make this prediction since I don't care about daly sales but we are interested in weekly or monthly predictions.

Thanks a lot


texts: How to add a new category to a deep learning model?

texts: Say I have done transfer learning on a pre-trained network to recognize 10 objects. How can I add a $11^{th}$ item that the network can classify without losing all the 10 categories I already trained and the information from the original pre-trained model? A friend told me that active research is going on in this field, but I cannot find any relevant papers or a names  to search for?


texts: R's mice imputation alternative in Python

texts: What is Python's alternative to missing data imputation with mice in R? Imputation using median/mean seems pretty lame, I'm looking for other methods of imputation, something like randomForest.


texts: How to find the right regression model for my project

texts:
I tried several things to find the best regression model with the best parameters but i can't go higher than 40% right predictions.

So i have 67741 rows in an excel file. the data looks like this after cleaning (4 columns only , is it enough ?) :

and the target rows like this :

I'll try to explain my process .

I went to this website https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html.
From this graph the models that should suit my data are Lasso and ElasticNet,
i got very bad score with this code :

classifiers = [
ElasticNetCV(cv=5, random_state=0,max_iter=40000), # i added the max_iter cause i got a warning saying that i should increase it
linear_model.Lasso(alpha=0.1,max_iter=40000)] # i added the max_iter cause i got a warning saying that i should increase it

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.33, random_state=42)

for clf, param in zip(classifiers, param_grid):
    name = clf.__class__.__name_
    clf.fit(X_train, y_train)
    print("=" * len(name))
    print("{}".format(name))

    print(clf.score(X_test, y_test))

The scores:

============
ElasticNetCV
0.002404878871672067
=====
Lasso
0.0066801704903023396

Then i tried several other models and i finally got something with:

BaggingRegressor(base_estimator=DecisionTreeRegressor(),
              max_features=0.5, max_samples=0.5)

Score :

================
BaggingRegressor
0.3460147571634854

So i used GridSearch and then cross validation score to get the best parameters but every time i launch it i got a different result something like that :

BaggingRegressor(base_estimator=DecisionTreeRegressor(), bootstrap=True,
bootstrap_features=False,
              max_features=0.2, max_samples=0.7, n_estimators=10,
              n_jobs=None, oob_score=False, random_state=None, verbose=0,
              warm_start=False)

i used this that way :

param_grid = [{'max_samples': [0.1, 0.2, 0.5, 0.7, 1],
           'max_features': [0.1, 0.2, 0.5, 0.7, 1],
           'n_estimators': [5,10,15,20,25]}

def grid_search(clf, param_grid):
    grid_search = GridSearchCV(clf, param_grid, cv=5)
    grid_search.fit(X, y)
    print(grid_search.best_estimator_)
    print("=" * len(name))
    print(grid_search.best_params_)
    print("=" * len(name))
    print(grid_search.best_score_)

and cross validation like that : scores = cross_val_score(clf, X, y, cv=10)

Sorry if it's a bit long, with 67000 rows i can't get something more than 30% correct predictions ? what's the problem ?

Thank you


texts: Optimization technique for iterative distribution

texts:
Apologies if this does not fit the proper format for this site, as it is a somewhat general question.
I have an application that sits on top of a SQL database, and need to handle a process which is very much like a linear algebra problem, whereby an amount a (any number between 0-1B) needs to get distributed among n entities (e), based on a few parameters (rank, weight, min/max requirements) set at the entity level.
Example:
a = 100

entity    weight    min    max
----------------------------------------------
X         0.25      10     40
Y         0.75      40     60
...

In this example, 25 (a * X[weight]) would go to entity X, and 75 would go to entity Y; however, 75 exceeds Y[max], so the remaining 15 need to go to another entity (in this case, X, since it stays at or under X[max]).
Intuitively, this is in iterative process. In a real example, there would be more entities, so more iterations would be necessary. SQL is not designed to iteration. I am looking for a way to better handle this in a set based method.
What I am looking for is something along the lines of:

a statistical method that I can use to minimize the number of iterations I need to make, or even better, a way that this can be distilled into a formula?

alternatively, maybe there is a way to store some of the data in a static way to minimize the steps needed to make the calculations on the fly?

Creating a lookup table, whereby for each entity there could be a stored outcome for min/max ranges, based on the other entities (they are in groups of 10 or less).


texts: What are the advantages / disadvantages of off-policy RL vs on-policy RL?

texts:
There are various algorithms for reinforcment learning (RL). One way to group them is by "off-policy" and "on-policy". I've heard that SARSA is on-policy, while Q-Learning is off-policy.

I think they work as follows:

My questions are:

How exactly is "on-policy RL" and "off-policy RL" defined?
What are the advantages / disadvantages of both?


texts: Target encoding with KFold cross-validation - how to transform test set?

texts:
Let's say I have a categorical feature (cat):
import random
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold

random.seed(1234)
y = random.choices([1, 0], weights=[0.2, 0.8], k=100)
cat = random.choices(["A", "B", "C"], k=100)
df = pd.DataFrame.from_dict({"y": y, "cat": cat})

and I want to use target encoding with regularisation using CV like below:
X_train, X_test, y_train, y_test = train_test_split(df[["cat"]], df["y"], train_size=0.8, random_state=42)
df_train = pd.concat([X_train, y_train], axis=1).sort_index()
df_train["kfold"] = -1
idx = df_train.index
df_train = df_train.sample(frac=1)

skf = StratifiedKFold(n_splits=5)
for fold_id, (train_id, val_id) in enumerate(skf.split(X=df_train.drop("y", axis=1), y=df_train["y"])):
    df_train.iloc[val_id, df_train.columns.get_loc("kfold")] = fold_id

df_train = df_train.loc[idx]

encoded_dfs = []

for fold in df_train["kfold"].unique():
    df_train_cv = df_train[df_train["kfold"] != fold].copy()
    df_val_cv = df_train[df_train["kfold"] == fold].copy()

    means = df_train_cv.groupby('cat')['y'].mean()
    df_val_cv['cat'] = df_val_cv['cat'].map(means)
    encoded_dfs.append(df_val_cv)

encoded_dfs = pd.concat(encoded_dfs, axis=0).sort_index()
encoded_dfs.drop('kfold', axis=1, inplace=True)

However, I have some doubts about the way how I should then encode test set. As there is no single mapping deduced from train set I think we should use the whole train set to fit the encodings and then use it on test set:
means = df_train.groupby('cat')['y'].mean()
X_test['cat'] = X_test['cat'].map(means)

It seems to be the natural way to do it as, in fact, this is exactly mimicked by CV step. But the results of the model I got were off and it made me think if I am missing something. Please note that, for sake of simplicity, I omitted additional smoothing I did as well. Therefore, my question is: is it the correct way to encode test set?


texts: Predictive Analytics on distributed systems vs standalone system

texts:
I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance knowledge on this subject can enlighten me. Here is where I am getting confused:

How can a statistical model (such as linear regression model) built on Hadoop where data is distributed across many nodes, and where work is done only on subset of original data-set residing in each node have same or better accuracy than linear regression model built on single server where all data resides locally. Would the error-rate (accuracy) of a model build on hadoop be same, better or worse than model build on local system. I was thinking that since all data reside in local system, algorithm can be tuned to optimize error in a linear regression model. However, in case of Hadoop where data is distributed, what if only local optimization lead to drastically bad algorithm overall? Are the algorithms written separately for distributed system to take into account data distribution?


texts: Multi label classification and sigmoid function

texts:
I'm new to neural networks so this may be silly question.
I have build standard CNN network for image classification. I want multi-label classification network so I
use binary_crossentropy as loss function:

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-4),
              metrics=['acc'])

and have sigmoid function as activation function in last layer (two neurons for two labels):

model.add(layers.Dense(2, activation='sigmoid'))

Output gives me something like this:

[[0.000497834], [0.99942183]]

Why does this two numbers add to 1, isn't sigmoid output suppose to be independent?
What should I do to get independent probability as output (for example, if image doesn't belong to any of two classes
output should be close to 0 for two neurons, something like this : [[0.001], [0.001]]

Thanks in advance for any help.


texts: Converting feature coefficients to probabilities scores for linear models

texts:
I have been looking into interpreting feature coefficient in a bounded capacity (0,1) as probability scores. So for example, if one has the following linear model for classification(Logistic regression),
then inverse logit should return one linear probability scores right as it is the inverse logs of odd ratio?

y<- c(1:3, 7, 6)
x1<- 1:5
x2<- c(20:25)
temp<-coef(lm(y ~ x1 + x2))
library('arm')
invlogit(temp[1]+temp[2])

However, in-case of linear regression for continuous value prediction,

coeff1/sum(coeff)

should return one probability scores right ?
Did I miss something ?


texts: Learn Unknown parameters from Input,output and Known parameters - Online training

texts:
I am new in Machine Learning field and I want to achieve one task using Machine Learning, but I am not sure how can I do that. Please help me, I am trying to do following task.

I have an external function (don't know its internal working) which accept input force ranging from 0 to 9.8 (it's gravity constant g) and it produces output percentage from 0%-100%. This percentage will apply to real object and depend on the physical environment, it will apply some force that could be different than the input force. My objective is to get applied force near to required force. The external function uses some parameters to come up with the percentage. Out of those parameters, some are constants and some are variables which changes according to the outside environment.

Using required force, actual gained force and constant parameters, I want to fine tune variable parameters.

Consider following diagram for better explanation

-------More simplified-----

In below table I have KF(Known constant features) UKF(unknown constant features). Using all data collected i.e. Input, output, and KF(known features) find value of X,Y,Z. These X,Y, Z will not change.

Input | KF1 | KF2 | KF3 |...| KF10 |UKF1|UKF2|UKF3  |...|UKF10| Output(achieve same value as Input)
0.1   | 5   | 10  | 0.5 |...| 80   |X   | Y  |  Z   |...| P   | 0.1
0.5   | 5   | 10  | 0.5 |...| 80   |X   | Y  |  Z   |...| P   | 0.5
2     | 5   | 10  | 0.5 |...| 80   |X   | Y  |  Z   |...| P   | 2
4     | 5   | 10  | 0.5 |...| 80   |X   | Y  |  Z   |...| P   | 4
.     | .   | .   | .   |...| .    |.   | .  |  Z   |...| .   | .
.     | .   | .   | .   |...| .    |.   | .  |  Z   |...| .   | .
.     | .   | .   | .   |...| .    |.   | .  |  Z   |...| .   | .
8     | 5   | 10  | 0.5 |...| 80   |X   | Y  |  Z   |...| P   | 8


texts:
I'm sure this is a standard problem in machine learning so links to pages or books that discuss the problem are also welcome.

The problem is: We have some input data points, $x_i$, and some output labels, $y_i$, that are drawn from a probability distribution whose parameters are somehow encoded in the input data. Now we want to learn a model, $f(x_i, \theta): x_i \rightarrow \lambda_i$ with some fit variables $\theta$, that outputs $\lambda_i$, which are the parameters of the distribution where $y_i$ are drawn from. The type of the distribution (e.g. normal, exponential, etc.) and number of its parameters is known.

My first idea would be to draw a new random variable from the distribution produced by $f$ for each sample, do this many times in batches and for each batch minimize $\sum_i (y^*_i - y_i)^2$. I can't really prove it, but my intuition tells me that this way the distribution parameters $\lambda_i$ should converge to those of the actual distribution.


texts:
I am new to Spark

I am using pyspark to predict a multi label results.

I have converted multi labels to binary

So my labels will look like this

Red  Green  Blue  Yellow  Black  White  Brown  Pink  Orange
1    1      1     0       1      0      1      0     0
0    0      0     1       0      0      0      0     1
1    1      0     0       1      1      1      1     0
1    1      0     1       1      0      0      1     0
0    0      1     0       0      0      0      0     0
0    1      1     0       1      0      1      1     1

my spark dataframe looks like this

vhouse_df.show(3)
+--------------------+---------------+
|            features|         labels|
+--------------------+---------------+
|[21.0,5.0,5.0,21....|(31,[30],[1.0])|
|[0.0,3.0,3.0,0.0,...|(31,[30],[1.0])|
|[32.0,8.0,8.0,32....|(31,[30],[1.0])|
+--------------------+---------------+

I found this question in StackOverFlow https://stackoverflow.com/questions/33551747/logistic-regression-pyspark-mllib-issue-with-multiple-labels

Tried to do the same but got error

Here is my code

from pyspark.mllib.classification import LogisticRegressionWithLBFGS  #<----- (1)
lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))     #<----- (2)
lr_model = lr.fit(train_df)                                           #<----- (3)
print("Coefficients: " + str(lr_model.coefficients))                  #<----- (4)
print("Intercept: " + str(lr_model.intercept))                        #<----- (5)

Line (2) causing this error

lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))
Traceback (most recent call last):

  File "<ipython-input-11-05522f46c27a>", line 1, in <module>
    lr = LogisticRegressionWithLBFGS.train(sc.parallelize(train_df))

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\context.py", line 527, in parallelize
    jrdd = self._serialize_to_jvm(c, serializer, reader_func, createRDDServer)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\context.py", line 559, in _serialize_to_jvm
    serializer.dump_stream(data, tempFile)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 352, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 143, in dump_stream
    self._write_with_length(obj, stream)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 153, in _write_with_length
    serialized = self.dumps(obj)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\serializers.py", line 583, in dumps
    return pickle.dumps(obj, protocol)

  File "C:\Users\GX\anaconda3\lib\site-packages\py4j\java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)

  File "C:\Users\GX\anaconda3\lib\site-packages\pyspark\sql\utils.py", line 63, in deco
    return f(*a, **kw)

  File "C:\Users\GX\anaconda3\lib\site-packages\py4j\protocol.py", line 332, in get_return_value
    format(target_id, ".", name, value))

Py4JError: An error occurred while calling o102.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Unknown Source)

I am not getting what this error is about? and how to fix it?


texts: how to visualize InceptionV3 hidden layers

texts:
I am following google codelab:

Tensorflow for poet  to train my custom model. This google codelab use the Inception-V3 model for training.

The inception-V3 model have 48 layer. My question is that how can i visualize image features at the hidden layers?


texts:
In follow up to my previous question, I was able to label survey responses based on 70 categories using RTextTools. Each survey gets one label while if manually coded some would have up to 8 labels.

The good news is my training data set does have multi-labels. So my question is, would be possible to train on 8 labels all at once? And then on the test data, get probabilities associated with each label and sort based on probability to get the top 8 labels?

Or can anyone suggest a better approach?


texts: Large Scale Personalization - Per User vs Global Models

texts:
I'm currently working on a project that would benefit from personalized predictions.  Given an input document, a set of output documents, and a history of user behavior, I'd like to predict which of the output documents are clicked.

In short, I'm wondering what the typical approach to this kind of personalization problem is.  Are models trained per user, or does a single global model take in summary statistics of past user behavior to help inform that decision?  Per user models won't be accurate until the user has been active for a while, while most global models have to take in a fixed length feature vector (meaning we more or less have to compress a stream of past events into a smaller number of summary statistics).