texts: How to estimate the not available observation in time series data?

texts:
Suppose, I have a 30 seconds time-step observations of sports data, in some of the intervals the game was partially/fully stopped. I'm trying to prep the data for a time series analysis. Is it justified to take it as zero when it was stopped fully? or I have to interpolate the value...

Here is the sample of data created without taking DeadBallMin(game paused) into account...

** columns A and B are actual data observed during the time-step.

** Exp_Win_A and Exp_Win_B are monotonic increasing. And assume all the features are uniformly distributed within the time-step.

    A      B    Exp_win_A   Exp_win_B   DeadBallMin
0   1      0    0.891713    1.074992    0.000000
1   0      1    0.893859    1.076465    0.000000
2   0      1    0.930300    1.077941    0.036633
3   0      1    0.932539    1.112289    0.000000
4   0      0    0.934783    1.122372    0.907834

From the above table, in the second row, the game was stopped for 33.67% of the time-step.

Question

Any suggestions on how to incorporate the 'DeadBallMin' time into  Exp_win_A & Exp_win_B while keeping the behaviour?


texts: Binary Neural Network Classification or Multiclass Neural Network Classification?

texts:
I am confused about the difference between a binary and multiclass neural network classification. If I am writing an algorithm that has 2 output classes (Obama or Romney), but not yes or no (so not like Obama or not Obama), then is it a binary neural network or a multi class (2 class) neural network classification?

What I do know:
A binary neural network classification outputs 1 unit. If a multi class classification neural network has k classes, then it has k outputs.

What I think:

I think it is a binary neural network classification because I am really only trying to output one thing, whether a county votes for Romney or Obama. I am confused because I thought binary was more like Romney or not Romney classification and I am not sure if that is the same as Romney or Obama. Just wanted to double check and clarify my understanding.


texts: The effect of removing pooling layers in the model's accuracy

texts:
I know that removing pooling layers will lead to an increase in dimensionality and subsequently, make the training to be more time-consuming. But I'm wondering if it worth it to remove pooling layers or not? does it lead to a higher accuracy?
Have you ever seen any relevant papers, articles, etc. about this issue? (I've searched and couldn't find much things except for this paper)
I even don't know how many pooling layers should I remove.


texts: GRU and LSTM does not "take risk" predicting

texts:
I tested LSTM and GRU models to predict the exchange rate between currencies. I do not take the raw price but a the delta with the previous day, so the data is stationnary around zero.
My problem is that my model always predict really close-to-zero values, like if it minimize the risk and does not want to guess wrong. It may be because it underfit but I wanted to be sure that it is not a common issue that I just totally don't know about.
I have really simple architecture, 1 layer of GRU or LSTM (tried both), data from the past 20 years, and using the 20 previous days to predict the next one. Tried to play with LR, dropout and number of epochs yet but this behaviour seems weird to me. Maybe I miss something?

Thanks!


texts: Architecture help for multivariate input and output LSTM models

texts:
I am working on a sequence prediction problem using Keras.  My end-goal is to have my input be several features, one being a categorical variable, that I have used 1 hot encoding for, and another which is a numeric variable, for which I would like to predict the next time step(s) for BOTH variables (any other variables just as inputs).

My first attempt was to eliminate the numeric and just use the sequence of "words" (not really words, but essentially the same concept here) to learn model that could predict the rest of the sequence, given partially complete "sentences".  I have built a working model that attempts this, with decent results.

Model thus far:

def LSTMModel(timestep_len,hidden_size,vocab_size):
    model = Sequential()
    model.add(Masking(mask_value=0.0, input_shape=(timestep_len,num_of_features)))
    model.add(LSTM(hidden_size, return_sequences=True))
    model.add(TimeDistributed(Dense(vocab_size)))
    model.add(Activation('softmax'))
    return model

input size to masking: (None, 27, 1)

output size from softmax: (None, 27, 6997)

I pad the sequences to all be of the same length, which is why there is a Masking layer.

I an stuck on how I can incorporate a numeric variable in with this.  How do I need to adjust my architecture to accommodate for this?  I realize this current one will not work for what I'm trying to do.  This was just a simple first pass approach.  Do I need a custom loss function for each variable and then add them together some how?  My input is size (batch_size, max_seq_len, 1), where the 3rd dimension is a unique integer encoding for the "word".  My input Y is of size (batch_size,max_seq_len,vocab_size) where vocab _size is the one-hot vector representation of the t+1 timestep. How do I extend this properly to add in another variable?  How can I use a Softmax on part of the output of a dense layer?  Is that even the path I should be thinking?

I am having not much luck finding good resources for this type of sequence prediction.  Everything seems to be more simple and restricted.

Thanks for any help!


texts: The GA package is a great package to use Genetic Algorithm for optimization. See for example this. I have a use case, where my possible values are integer (e.g. 1, 2, 4 ...). So far, I simply rounded the real values in my fitness function. Is there a better way to encode integers? Thanks!


texts: Question regarding strata in Geron's book

texts:
In the book "Hands-On Machine Learning with Scikit-Learn and Tensorflow" by Aurélien Géron. There is a regression project explained. My doubt is regarding his example for 'stratified sampling'.

He mentions:

  So far we have considered purely random sampling methods. This is
  generally fine if your dataset is large enough (especially relative to
  the number of attributes), but if it is not, you run the risk of
  introducing a significant sampling bias. When a survey company decides
  to call 1,000 people to ask them a few questions, they don’t just pick
  1,000 people randomly in a phone book. They try to ensure that these
  1,000 people are representative of the whole population. For example,
  the US population is 51.3% females and 48.7% males, so a
  well-conducted survey in the US would try to maintain this ratio in
  the sample: 513 female and 487 male. This is called stratified
  sampling: the population is divided into homogeneous subgroups called
  strata, and the right number of instances are sampled from each
  stratum to guarantee that the test set is representative of the
  overall population.

I do understand his analogy, but he is comparing a case, where we know all population (US population), whereas the project is about housing prices which we only have a tiny fraction... How can we stratify if we dont even know the correct strata?


texts: Leave - one - out - Cross Validation KNN R

texts:
I have a dataset and I divided it into test data and train data. Can anyone suggest how to perform LOOCV for KNN regression? Is there any library?

val(train.data,train.label,K,numfold)

Thanks in advance


texts: Machine learning techniques for estimating users' age based on Facebook sites they like

texts:
I have a database from my Facebook application and I am trying to use machine learning to estimate users' age based on what Facebook sites they like.

There are three crucial characteristics of my database:

the age distribution in my training set (12k of users in sum) is skewed towards younger users (i.e. I have 1157 users aged 27, and 23 users aged 65);
many sites have no more than 5 likers (I filtered out the FB sites with less than 5 likers).
there's many more features than samples.

So, my questions are: what strategy would you suggest to prepare the data for further analysis? Should I perform some sort of dimensionality reduction? Which ML method would be most appropriate to use in this case?

I mainly use Python, so Python-specific hints would be greatly appreciated.


texts: What is the difference between horizontal and vertical ensemble?

texts:
I am looking at different ways to do model ensembling and I came across the terms horizontal and vertical blending/ensembling but it is not well defined.

My questions will be:

What is the difference between horizontal and vertical blending?
In which cases which one is better?
Are there any python implementations for them?


texts: How does skewed data affect deep neural networks?

texts:
I'm playing around with deep neural networks for a regression problem. The dataset I have is skewed right and for a linear regression model, I would typically perform a log transform. Should I be applying the same practice to a DNN?
Specifically, I'm curious how skewed data affects regression with a DNN and, if it's negatively, are the same methods that would be applied to a linear regression model the right way to go about fixing it? I couldn't find any research articles about it but if you know of any feel free to link them in your answer!


texts: How to combine heterogeneous image features extracted with different algorithms for similar image retrieval?

texts:
Say I have access to several pre-trained CNNs (e.g. AlexNet, VGG, GoogleLeNet, ResNet, DenseNet, etc.) which I can use to extract features from an image by saving the activations of some hidden layer in each CNN. Likewise, I can also extract features using conventional hand-crafted techniques, such as: HOG, SIFT, LBP, LTP, Local Phase Quantization, Rotation Invariant Co-occurrence Local Binary Patterns, etc. Thus, I can obtain a very high-dimensional feature vector of an image that concatenates the individual features vectors outputted by these individual algorithms. Given these features, and given a data set of images over which I want to perform similar image retrieval (i.e. finding the top-k most similar images to a query image X), what would be the most appropriate ways to implement this task?

One possible idea I have in mind is to learn an image similarity embedding in euclidean space by training a neural network that would receive as input the aforementioned feature vectors, and perhaps down-sampled versions of the image as well, and output a lower dimensional embedding vector that ideally should place similar images close to each other and dissimilar images far apart. And I could train this network using for example Siamese Loss or Triplet Loss. The challenge of this approach though is generating the labels for the (supervised) training itself. For example, in the case of the Triplet Loss I would need to sample triplets (Q,X,Y) and somehow determine which one between X and Y is most similar to Q, in order to generate the label for the triplet (i.e., in order to "teach" the network I need to know the answers myself beforehand, but how? I guess this is domain dependent, but think of challenging cases where you have very heterogeneous images, such as photography galleries, artwork galleries, etc).

Anyways, this is just an idea and by no means I pretend to mean this is the right approach. I'm open to new suggestions and insights about how to solve this task.


texts: Can neural networks have multi-dimensional output nodes?

texts:
I'm trying to understand what's possible with TensorFlow's output layer. Specifically, are outputs always a flat array?
Since a neuron (or 'unit', in TF) has just one number, and there is only one set of outputs, it seems that output must have a single dimension. With one-hot probabilities, this is easy to understand. But what about an image?
If my output is going to be a picture, can I have TF output a multi-dimensional array of pixels, e.g. [[r0, g0, b0], [r1, g1, b1], ...]? If so, how would that network be constructed? How would I define the output layer's dimensionality/shape?
The only param I know of that defines output shape is this, from tf.layers.dense, which seems inherently one-dimensional:

units (number) Positive integer, dimensionality of the output space.

Any help you can provide is greatly appreciated!

Reference: https://js.tensorflow.org/api/latest/#layers.dense


texts: How to speedup K-Means used in 'for loop'

texts:
I'm trying to solve an interesting problem. One solution which seems to work well, involves using K-Means in a 'for loop'. The dataset per loop is independent and fairly small (Minibatch not required).

e.g.

for i in range(100):
    y = Kmeans(x) # x is new data per loop

Results are good but execution is quite slow. Debugging shows that most of the delay is in the K-Means algorithm (from scikit-learn).

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    40960   13.402    0.000   92.785    0.002 k_means_.py:426(_kmeans_single_elkan)
   243337    8.506    0.000   51.421    0.000 pairwise.py:165(euclidean_distances)
   931529    6.950    0.000    6.950    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   341357    5.178    0.000   32.424    0.000 validation.py:332(check_array)
2157218/1952418    5.010    0.000   28.877    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
   341357    4.617    0.000   15.927    0.000 validation.py:36(_assert_all_finite)
   117456    3.356    0.000    3.849    0.000 _k_means.pyx:258(__pyx_fuse_1_centers_dense)
   789264    2.617    0.000    9.859    0.000 fromnumeric.py:73(_wrapreduction)
    40960    2.584    0.000   37.103    0.001 k_means_.py:43(_k_init)
  1265786    2.053    0.000    2.053    0.000 {built-in method numpy.array}
   253435    2.015    0.000    2.015    0.000 {built-in method numpy.core._multiarray_umath.c_einsum}

I've already reduced the number of iterations and initialisations. Is there a way to speed up this execution?


texts:
I want a simple image classifier to detect (No need to recognize) whether an image contains text or not.
Some of the approaches I thought of:

Single class image classifier with a Threshold
Fast/Faster RNN technique

Which is the best approach?

And if I have to detect text or logo, is it efficient to use single class and feed all data together or use 2 classes?

Thank you


texts: Are there any examples other than anomaly detection where unsupervised deep learning could be useful?

texts:
I am new to deep learning and its concepts. After reading a while I understood that unsupervised deep learning techniques usually try to reconstruct the input data(probably with less number of dimensions using encoder-decoder) and train the network by optimizing reconstruction error. But I am unable to image how these could be used for solving real life tasks(other than anomaly detection, for example clustering).

Note: You can correct me, if my understanding about the unsupervised deep learning techniques is wrong.


texts: How does QUEST compare to other decision tree algorithms?

texts: SPSS Modeler has an implementation of QUEST, along with C&RT, C5.0 and CHAID. QUEST is relatively rarely covered in textbooks - what are its pros and cons compared to other decision tree algorithms? How does it make splits? Why is it (apparently) not in as widespread use as C&RT or C5.0?


texts: How to extract the values used in columns by a Classification model in R?

texts:
I want to find the variables (and its values) used to build a  classification model?

I came to know that the following code does the same

get_all_vars(model_built, dataset_used_for_building_a_model)

However I would not be able to use the above code as I am going to use the "model_built" alone in the R code and i will not be having the "dataset_used_for_building_a_mode.

Simply put i want to use this "Model_built" alone and fetch the variables used along with the values.


texts: Can a Warm-up loss period easily be implemented in Keras?

texts:
I am creating a RNN in Keras. It was suggested that I utilize a warm-up period before loss is calculated to increase accuracy down the line.

I saw some people achieved this by creating a loss function via tensorflow.

Does Keras have any simple way to directly add a warm-up period?


texts: What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance?  Are there specific problem types that fall into this category?


texts: How does keras train without disrupting the data set order

texts:
I want to train a neural network using keras.

model.fit(trainX, trainY, batch_size=400, epochs=1000,shuffle=False)

I want the model training using the sequence of original data set order, not shuffle.


texts: Choosing One vs All and One vs One for Multiclass SVM

texts: What consideration do I need to take when choosing between One vs All or One vs One algorithm for Multiclass SVM?


texts: ggplot (in R)/matplotlib (in Python) with linear model?

texts:
How can I plot the histogram below, using ggplot (R) and/or matplotlib (Python)?

In base plot I use:

histogram(~ Wrkday | Year+Avg, data=Data, layout=c(3,2))

The data I work on:

Code in R:

Input <- ("
Year       Student  Wrkday
      '1st year'  a        1200
      '1st year'  b        1400
      '1st year'  c        1350
      '1st year'  d         950
      '1st year'  e        1400
      '1st year'  f        1150
      '1st year'  g        1300
      '1st year'  h        1325
      '1st year'  i        1425
      '1st year'  j        1500
      '1st year'  k        1250
      '1st year'  l        1150
      '1st year'  m         950
      '1st year'  n        1150
      '1st year'  o        1600
      '1st year'  p        1300
      '1st year'  q        1050
      '1st year'  r        1300
      '1st year'  s        1700
      '1st year'  t        1300
      '2nd year'  u        1100
      '2nd year'  v        1200
      '2nd year'  w        1250
      '2nd year'  x        1050
      '2nd year'  y        1200
      '2nd year'  z        1250
      '2nd year'  aa       1350
      '2nd year'  ab       1350
      '2nd year'  ac       1325
      '2nd year'  ad       1525
      '2nd year'  ae       1225
      '2nd year'  af       1125
      '2nd year'  ag       1000
      '2nd year'  ah       1125
      '2nd year'  ai       1400
      '2nd year'  aj       1200
      '2nd year'  ak       1150
      '2nd year'  al       1400
      '2nd year'  am       1500
      '2nd year'  an       1200
      '3rd year'  u        1600
      '3rd year'  v        1700
      '3rd year'  w        1450
      '3rd year'  x        1650
      '3rd year'  y        1800
      '3rd year'  z        1550
      '3rd year'  aa       1950
      '3rd year'  ab       1750
      '3rd year'  ac       1925
      '3rd year'  ad       1825
      '3rd year'  ae       1625
      '3rd year'  af       1525
      '3rd year'  ag       1800
      '3rd year'  ah       1725
      '3rd year'  ai       1200
      '3rd year'  aj       1600
      '3rd year'  ak       1950
      '3rd year'  al       1100
      '3rd year'  am       1400
      '3rd year'  an       1600
      ")

Data <- read.table(textConnection(Input),header=TRUE)
los<-rbinom(nrow(Data), size = 1, prob=0.7)
Data$Avg<-ifelse(los==1,"Above 4.0","Below 4.0")

I would like to plot linear model in each box instead of histogram.


texts: Is there any resource with a list of feature engineering techniques? A mapping of type of data, model and feature engineering technique would be a gold mine.


texts: How to select optimal threshold which separate different classes?

texts:
I have trained a network to find the similarity between two images. The test dataset contain equal number of similar and dissimilar samples. Each class has approx. 13822 samples.

I tried different threshold, for example, in the last layer of the network I used sigmoid activation function therefore the output is between [0,1].

I tested threshold such as from 0.5 to 0.9 (for example, if the prediction score is greater then 0.5/0.6/,....,/0.9, consider it as positive sample else negative sample).

Based on this strategy I got the following results:

Threshold   TP  TN  FP  FN  Accuracy    Precision   Recall  F1
0.5 --> 12570   13804   18  1252    0.95    1.00    0.91    0.95
0.6 --> 12115   13813   9   1707    0.94    1.00    0.88    0.93
0.7 --> 11451   13819   3   2371    0.91    1.00    0.83    0.91
0.8 --> 10124   13822   0   3698    0.87    1.00    0.73    0.85
0.9 --> 5132    13822   0   8690    0.69    1.00    0.37    0.54

What I am interested in is to find the optimal threshold based on some method which separates positive and negative classes.


texts: Monte Carlo Dropout as Uncertainty predection

texts:
I am pretty new to Python and this board so I am not sure, if I am at the right place for my question since it doesn't include any code. If not so, please give my a hint for a better way/place to ask.

I am struggling with using Monte Carlo Dropouts for determine Uncertainty for my image classificator using ResNet18.

I have read several papers to this topic and I am still kinda confused about this topic.

I know already to use dropout like

def dropout2d(input, p=0.5, training=True, inplace=False):

multiply times for getting a variance which can be interpreted as the uncertainty. I am pretty sure, that I already understood how MC-Dropout works in general.

So coming to my questions:

Do I use MC-dropout while training or testing and why? I feel like I have read different ways.

I have searched for a coding example from the dropouts but didn't find anything. Do you know Paper/ example code/ searchlink for a proper code?

Is

nn.dropout2d(imput, p=x, ...) the right way to use dropout for a image classificator?

Thanks for your help.


texts: How to get different results running sklearn's MeanShift in a single program? (Python3)

texts:
I ran into a quirk with sklearn's MeanShift that I don't know how to get around.

MeanShift doesn't predictably give the same results on every run, so I wanted to run it multiple times within one program, but the results are the same on every iteration. They are different across program runs, but the same over every loop.

Minimal code sans convert_non_numerical_data(...).

import numpy as np
import pandas as pd
from sklearn.cluster import MeanShift
from sklearn import preprocessing

# EDIT: added on request
def convert_non_numerical_data_to_numerical(data_frame):
    columns = data_frame.columns.values

    for col in columns:
        text_digit_vals = {}

        if data_frame[col].dtype in [np.int64, np.float64]:
            continue

        x = 0
        for unique in set(data_frame[col].values):
            if unique not in text_digit_vals:
                text_digit_vals[unique] = x
                x += 1

        def convert_to_int(val):
            return text_digit_vals[val]
        data_frame[col] = list(map(convert_to_int, data_frame[col]))

df = pd.read_csv("test_data/titanic.csv")
df.fillna(0, inplace=True)  # bad idea to shove in 0?
convert_non_numerical_data_to_numerical(df)

y = np.array(df["survived"])
X = np.array(df.drop(["survived"], 1))
X = preprocessing.scale(X)

run_data = []
for run_count in range(3):
    clf = MeanShift()
    clf.fit(X)
    run_data.append(clf.labels_.copy())
    clf = None

run_0 = run_data[0]
for other_run in run_data[1:]:
    if np.array_equiv(run_0, other_run):
        print("same")
    else:
        print("different")

Output:

same
same

Any idea what is causing this, or at least how to get around it?

Note:

Data set from pythonprogramming.net


texts: What is the right way to store datasets for a CNN project

texts: Our image classification project has thousands of raw photos, masks and reshaped images. We store source code in git. But datasets don't belong to source code version control. How should we store thee sets of images?


texts: What does a predicted probability really mean, without considering the accuracy of the underlying model?

texts:
Say I've built a (completely unrealistic) classification model in Keras that gives me 1.00 accuracy.

And next, I would like to use my model on some new, unseen data, and use model.predict_proba to get a probability that the observation belongs to class "A". Say this returns to me a 0.75.

Am I interpreting this correctly in English:  "100 percent of the time, the model is confident that this new observation is 75 percent likely to be class A" ?

If this is correct, then let's consider if my model was not totally perfect, like in real life, and instead it gave me a 0.40 accuracy. Say my predict_proba is still 0.75.  Then, is this correct:

"40 percent of the time, the model is confident that this new observation is 75 percent likely to be class A." ?

If so...this makes it seem like predict_proba() is not tell a complete story.

I could mislead someone (say a journalist...or a judge, whoever) by saying, "There's a 75 percent chance this unseen observation belongs to class A"...and that might sound great, if I fail to reveal that this statment was based on a model that had a low accuracy like  0.40.

Am I stating this correctly, and does my apprehension have validity?


texts: Recursive Feature Elimination (RFE) with Logistic Regression and little correlation between the features and the target (SKLearn)

texts:
I have little experience in the field of Machine Learning, so I apologise in advance for my ignorance and lack of intuitions.

I'm trying to build a classifying algorithm to identify if a subject is healthy or ill. I have the following data:

17 subjects.
Each subject has 1140 features, ranging from -1 to 1.
Each subject has a label from 0 to 6. 0 meaning very ill and 6 completely healthy.
I only have data of subjects with labels from 0 to 2.
The features (X) have low correlation with the labels (Y).

The way I obtained the features is irrelevant to this question, so I'm not going to go through it.

This is how the data looks:

Each colour represents a subject. As I have mentioned, I only have data of subjects with labels from 0 to 2, that is, Y=0, Y=1 or Y=2. And the features (X) are then 17 Subjects x 1140 features.

What I want to do is the following:

Fit the data into a Logistic Regression.
Use the Recursive Feature Elimination algorithm in order to fit the data into the classification function and know how many features I need to select so that its accuracy is high.
Use Stratified Cross Validation to enhance the accuracy.

This is how I've implemented the algorithm in Python.

skf = StratifiedShuffleSplit(n_splits=50, test_size=0.3) # Cross-validation 50 times
estimator = LogisticRegression(C=1) # The estimator used is Logistic Regression
selector = RFECV(estimator, step=1, cv=skf, scoring="accuracy") # Run RFE
selector = selector.fit(X, Y) # Fit the data
print(selector.grid_scores_) # Print accuracy

And I obtained an accuracy around 60% for all the features, as shown below:

So, my question is. Do you think I implemented the code in the right way? Do you have any suggestion to enhance the accuracy? Maybe adjust a bit the parameters?

Thanks a lot. I hope I made myself clear.


texts: Anomaly Detection: Model Creation & Implementation

texts:
I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information.

Suppose I want to identify sales anomalies in an e-commerce business with a lot of product diversity. Further, this business has 300 dimensions across their business.

Example Data Columns:

Timestamp (categorical)
Store # (categorical)
State (categorical)
Region (categorical)
Market (categorical)
(293+ other dimensions or columns)
Product Name (categorical)
Total Sales (numerical)

Questions trying to solve for:

What is the best model to identify anomalies in a dataset like this? Isolation Forests look promising.
Should I build different models for different types of anomalies? (i.e. positive/negative slopes, new sales that escalate from 0 to something, spikes, etc.)
If a model is built, what is the best way to not only identify an anomaly but identify the combinations/columns leading to the anomaly (i.e. root cause)? Brute force?
If combinations are identified, what is the best way to ensure the combinations have enough data to be statistically relevant? Or what if I identified the anomaly in another combination already?

Given the high number of dimensions, I'm struggling to identify the best type of model to train for anomalies, how best to use the model to identify root causes of anomalies, and ensure the root causes are statistically relevant enough to review. Looking across 300 dimensions is not feasible to review manually and the data isn't always normal like products coming off a conveyor belt. I am trying to eliminate many of the dimensions but it seems difficult to narrow this down to 1 or 2 dimensions.

Appreciate any recommendations or suggestions on how best to approach the problem. Any relevant books is also recommended. Thanks!


texts: Quadratic approximation of L1 regularized cost function

texts:
I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't give me (7.23).

Thanks for your help!


texts: How the Pandas Dataframe and Numpy masking technique works

texts:
I'm trying to understand how the masking technique works.

I've a dataframe with some values (say one_row_incidents) some of the are not unique. By grouping them .by the duplicate index, I can get the list of items which are used only once.

unique = df.groupby(by="Accident_Index").Accident_Index.nunique()

At this stage, I can create a second DataFrame with unique items with a masking technique like the following:

import numpy as np

pos = np.flatnonzero(one_row_incidents)
unique = df.iloc[pos]

The above works perfectly fine, but my question is, how does this work underneath?

In exploring the pos object, this is a one dimension sequencial array - with no gaps - starting from 0 to the length of unique. Hence, I can't see where it stores the reference to the corresponding index to the original array.

Any idea?


texts: Pandas - Avoid boolean result when using groupby()

texts:
I have this script:
    sectors = df.groupby(['company_sector']).mean()['investment_in_millions']

Output:

I wanted to keep the same groupy() but having a result in "investment_in_millions" column filtered as mean > 10 or another value.
If apply this:
     sectors = df.groupby(['company_sector']).mean()['investment_in_millions']>10

I keep the groupby() but it returns a boolean into the investment column.
If I use:
     filtered = df[df['investment_in_millions']>10]

I get the filtered values mean>10 but the groupby() is not there anymore and I get all the other columns in the excel.
How can I get the groupby() together with the mean>10 without getting a boolean result?
Thanks!


texts: How to deal with TypeError: ufunc 'isnan' not supported for the input types

texts:
I have dealt with all the Nan values in the features dataframe, then why I am still getting this error?

sns.heatmap(features, annot=True, annot_kws={"size": 7})
sns.plt.show()

TypeError                                 Traceback (most recent call last)
    <ipython-input-11-534a699b432d> in <module>()
    ----> 1 sns.heatmap(features, annot=True, annot_kws={"size": 7})
          2 sns.plt.show()

    c:\users\jetjo\appdata\local\programs\python\python36\lib\site-packages\seaborn\matrix.py in heatmap(data, vmin, vmax, cmap, center, robust, annot, fmt, annot_kws, linewidths, linecolor, cbar, cbar_kws, cbar_ax, square, xticklabels, yticklabels, mask, ax, **kwargs)
        515     plotter = _HeatMapper(data, vmin, vmax, cmap, center, robust, annot, fmt,
        516                           annot_kws, cbar, cbar_kws, xticklabels,
    --> 517                           yticklabels, mask)
        518
        519     # Add the pcolormesh kwargs here

    c:\users\jetjo\appdata\local\programs\python\python36\lib\site-packages\seaborn\matrix.py in __init__(self, data, vmin, vmax, cmap, center, robust, annot, fmt, annot_kws, cbar, cbar_kws, xticklabels, yticklabels, mask)
        166         # Determine good default values for the colormapping
        167         self._determine_cmap_params(plot_data, vmin, vmax,
    --> 168                                     cmap, center, robust)
        169
        170         # Sort out the annotations

    c:\users\jetjo\appdata\local\programs\python\python36\lib\site-packages\seaborn\matrix.py in _determine_cmap_params(self, plot_data, vmin, vmax, cmap, center, robust)
        203                                cmap, center, robust):
        204         """Use some heuristics to set good defaults for colorbar and range."""
    --> 205         calc_data = plot_data.data[~np.isnan(plot_data.data)]
        206         if vmin is None:
        207             vmin = np.percentile(calc_data, 2) if robust else calc_data.min()

    TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''


texts: How to predict whether or not a customer will renew

texts:
I have a dataset of customer contracts that specify a start date and if applicable an end date. Each month a customer is up for renewal. Below is an example of how the data is organized in excel:

ID   Customer Start Date   Customer Drop Date
1    Jan. 2018             Dec. 2018
2    Feb. 2018             July 2018
3    Mar. 2018

Using the above example, I'm trying to predict whether customer 3 will drop in Jan. 2019, Feb. 2019, Mar. 2019, etc.. Essentially I'm trying to calculate the probability that a customer that's still active will renew their contract for a given month after Dec. 2018. What is the remaining life-time value?

Should I graph the the length of all historic contracts and see what distribution they match? If so how would I apply the distribution to the open contracts?


texts: What is the problem classification (e.g. sequence-to-sequence) for prediction of an autocorrelated sequence from multiple autocorrelated sequences?

texts: I will try to provide a simple example to illustrate my question. I have training data for many oil wells. The training data consists of the well identifier, a timestamp, other relevant properties like temperature and pressure and the response variable is oil production rate at the timestamp. I'm trying to predict the oil production rate for a separate set of wells for a period of time given the temperature, pressure, etc. For an individual well, the oil production rate is correlated with the production rate at the timesteps before it. However, between wells there is no relationship between one another. I'm trying to research strategies to improve my predictions for this kind of problem, but I don't really know what to search. It doesn't seem to quite fit as a sequence problem. How would you classify this? I would appreciate any tips for tackling this kind of problem.


texts: How valuable is a categorical feature that has a predominant category over all other ones?

texts:
Is a categorical feature that has almost equally distributed in it's category more important or the one which one of it's category is predominant over all other ones?
In data prepossessing step for "House Price" competition, I want to decide if Street feature is important or I can drop it from data set in order to avoid over-fitting. So I have plotted a swarm-plot as follows:

How should I interpret this plot? Does it show that the Street plot can be dropped, or it says it is valuable for creating a model?


texts: Learning capacity: Deep Learning vs Traditional (Shallow) Learning

texts:
I am currently doing a course in coursera in which Andrew Ng draws the following image:

Does anybody know any references/reasoning that justify the drawn graph? Were any experiments conducted to obtain it? If so, could you point me to them?

I have already asked in the coursera forum and sent an e-mail to the support team with no avail.

Thank you


texts:
I have a dataset including a set of customers in different cities of California, time of calling for each customer, and the status of call (True if customer answers the call and False if customer does not answer).

I have to find an appropriate time of calling for future customers such that the probability of answering the call is high. So, what is the best strategy for this problem? Should I consider it as a classification problem which the hours (0,1,2,... 23) are the classes? Or should I consider it as a regression task which the time is a continuous variable? How can I make sure that the probability of answering the call will be high?

Any help would be appreciated. It also would be great if you refer me to similar problems.

Below is a snapshot of the data.


texts: R: Error when using Aggregate function to compile monthly means into yearly means

texts:
Disclaimer: I'm extremely new to R and have been getting by with using google as my professor.

I have a somewhat large collection of monthly values over a period of several years from several different locations. I am attempting to use the aggregate function to calculate the yearly means for each location so that yearly rates of change can be calculated. However, when I run the code

read_csv_filename <- function(filename){
  ret <- read.csv(filename)
  ret$Source <- filename #EDIT
  ret
}

import.list <- ldply(filenames, read_csv_filename)
by1 <- import.list$Source
by2 <- import.list$Result
by3 <- import.list$Year
Yearly_Mean <- aggregate(import.list, by==list(by1, by2, by3), FUN= "mean")

I get an error like this

> Yearly_Mean <- aggregate(import.list, by==list(by1, by2, by3), FUN= "mean")
Error in by == list(by1, by2, by3) :
  comparison (1) is possible only for atomic and list types

I've spent quite a bit of time looking here and elsewhere for similar issues, but haven't found a case that helped me out at all. Any advice on how to fix this (or a completely new, easier method) would be appreciated.

Thanks!


texts: Poor performance of SVM after training for rare events

texts:
I found out that Weighted SVM is a classification approach to handle class imbalance problem. My data set is highly imbalanced with rare event (minority class, labeled as 1) and the majority class (label 0). So I implemented the supervised classification weighted svm technique with stratified cross-validation as these are able to handle class imbalance. I added an additional tuning for the C parameter (boxconstraint). The training is done using 5 folds cross-validation approach. The method works well on the training set. I get good performance after the training. This I can say because by looking at the confusion matrix after the training.

cmMatrix =

        1443          27
           0          30

It is generally recommended to re-train using the optimized hyperparameters. So, I ran the trained model on the entire dataset again (re-trained it) an predicted on the same dataset.

PROBLEM: If I give a highly imbalanced unseen new data set (this set is never used by the model and is the Test set) to the trained SVM model, the prediction on this data is totally biased towards the majority class as shown below

cmMatrix_TestData =

        98     2
         5     0

Where did I go wrong? Please help, I practically have no method working for class imbalance problem whereas several articles and suggestion suggest these two approaches which I am not able to make it work for me.


texts: Time complexity of function minimizers for neural networks

texts:
I am trying to train a neural network for recognizing handwritten letters from A to J . I have a training set of size 200000 . Each training set is a list of  784 pixel values. My neural net has input layer of size 784 , hidden layer of size 50 and output layer of size 10.

I am using fmin_cg minimization function of scipy library of python. The problem I am facing is that each iteration is taking a lot of time.

The first iteration took almost 7-10 minutes.
The second iteration took 20 minutes.
Third is still running.

This might be due to my outdated computer with only 2 gb of memory and a slow processor but I have previously trained a neural net with the training set of size 5000 , input layer size if 400 , hidden layer size 25 and the output layer of size 10 . This neural net recognized handwritten digits and it was an exercise problem of coursera course on machine learning by Andrew Ng .

So  yes I know that the current neural network should take more time to train than the previous one as the training set , input layer, and hidden layer are all much larger than previous neural net but still, I think it's taking a lot of time . Why is it so slow ?

Is it normal for the neural network of this size ? Or should I use other faster optimization algorithm ? Is there a way to measure time complexity of neural networks ?


texts: How do CNNs find different feature maps?

texts:
Assume I have a CNN that in the first (conv) layer takes a 1-channel signal (the input) and gives a 2-channel output. Let's further assume that the rest of the net has symmetric architecture from the point-of-view of any of those channels. Moreover, let's assume all the weights are initialized by zero (or the same number).

How do we know that the feature maps (kernels of the channels) after training are not similar given the symmetry in such network? Specifically, what is unreasonable to have kernels with identical feature maps?


texts:
Should I add csv as text in SO question? There's lot more data.
%matplotlib inline
plt.xlabel('Year')
plt.ylabel('Income($US)')
plt.scatter(df.year,df.income,color='red',marker='+')

reg = linear_model.LinearRegression()
reg.fit(df[['year']],df.income)

Output : LinearRegression()

reg.predict('10000')

--------------------------------------------------------------------------- ValueError                                Traceback (most recent call
last)  in
----> 1 reg.predict('10000')
~/.local/lib/python3.9/site-packages/sklearn/linear_model/_base.py in
predict(self, X)
236             Returns predicted values.
237         """
--> 238         return self._decision_function(X)
239
240     _preprocess_data = staticmethod(_preprocess_data)
~/.local/lib/python3.9/site-packages/sklearn/linear_model/base.py in
decision_function(self, >X)
218         check_is_fitted(self)
219
--> 220         X = check_array(X, accept_sparse=['csr', 'csc', 'coo'])
221         return safe_sparse_dot(X, self.coef.T,
222                                dense_output=True) + self.intercept
~/.local/lib/python3.9/site-packages/sklearn/utils/validation.py in
inner_f(*args, **kwargs)
61             extra_args = len(args) - len(all_args)
62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
64
65             # extra_args > 0
~/.local/lib/python3.9/site-packages/sklearn/utils/validation.py in
check_array(array, >accept_sparse, accept_large_sparse, dtype, order,
copy, force_all_finite, ensure_2d, allow_nd,

ensure_min_samples, ensure_min_features, estimator)
628             # If input is scalar raise error
629             if array.ndim == 0:
--> 630                 raise ValueError(
631                     "Expected 2D array, got scalar array instead:\narray={}.\n"
632                     "Reshape your data either using array.reshape(-1, 1) if "

ValueError: Expected 2D array, got scalar array instead: array=10000.
Reshape your data either using array.reshape(-1, 1) if your data has a
single feature or >array.reshape(1, -1) if it contains a single
sample.

I am not sure why I am getting above error. I have income lists of some years. So, I graph it. When I was trying to predict a data from linearRegression I got the error. I am new to ML(Machine Learning)
How to solve it? What am I missing?


texts: Autoencoder feature extraction without validation set?

texts:
I plan to use autoencoder for feature extraction, then use the latent vector for clustering.

My autoencoder performs very very well on my training set (loss small and reconstructed image look very similar to input) but shows lower performance on validation set.

Is it a good idea to NOT use validation set but only a training set? Since I can reach good performance extracted vector should be "good" features

If I use new data I'll need to retrain the model, but I should always be able to reach good performances since new data will be very similar to current data


texts: When and where dummy.data.frame would be used?

texts:
I am learning this post, in which, the author gives the piece of code:

outcome<-dataf1$outcome
dataf1$outcome<-NULL
datafd<-dummy.data.frame(dataf1)
hdata<-cbind(data,datafd,outcome)
head(str(hdata))
data<-hdata

Other parts are easy to understand though, what does this line: dummy.data.frame(dataf1) do?

I constructed a tiny dataset to figure this out:

title <- c('engineer','manager','sales','engineer','sales')
salary <- c(51000, 33700, 26800, 53700, 36800)
df = data.frame(title, salary)
df
dummy.data.frame(df)

Which produces:

  titleengineer titlemanager titlesales salary
1             1            0          0  51000
2             0            1          0  33700
3             0            0          1  26800
4             1            0          0  53700
5             0            0          1  36800

I cannot imagine what this is used for.


texts: What algorithm can be used to divide a time series

texts:
Hi everyone

I have a data as on the image. I would like to derive a line that separates the data (like the red line), but not sure what algorithm can be used.

I thought about k-means in 1-d data, but then the high points (i.e. y-value ~0.01 are not really in the same cluster with the data point of 0.002).

Thanks a lot.

P/S: In fact, the values here are reconstructed MSE from h2o Autoencoder, and now I want to find an automatic way to determine the threshold, to say a point is an anomaly or not.


texts: How to process the hyphenated english words for any nlp problem?

texts:
Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful

if I remove the hyphen as special character and treat it as a single word 'wellknown' or
separate the word into 2 'well' and 'known' or
use all 3 words 'well' , 'known', 'wellknown' in vector creation(BOW/TF-IDF) process for model input.

Any quick help on this would be more appreciated. Thank you.


texts: How to match up categorical labels in training and evaluation

texts:
I am creating a CNN to categorise a sentence into one of N possible labels.

I have used the tutorial from WildML to start the code, and I have modified it to allow multiple outputs instead of just true/false.

I am using VocabularyProcessor to convert the words into numbers and then using OneHotEncoder to transform the labels for evaluation. I am currently using the same VocabularyProcessor for converting both X and y

Where x_text is the loaded sentences and y_text is the correct labels

max_document_length = max([len(x.split(" ")) for x in x_text])
vocab_processor = learn.preprocessing.VocabularyProcessor(max_document_length)
X = np.array(list(vocab_processor.fit_transform(x_text)))

y_val = np.array(list(vocab_processor.transform(y_text)))

from sklearn.preprocessing import OneHotEncoder
onehotencoder = OneHotEncoder(categorical_features = [0])
y = onehotencoder.fit_transform(y_val).toarray()
y = y[:, 1:]

This then goes through the training.

The problem that I am finding is that during evaluation, if the labels are not exactly the same then the indexes do not match the prediction that is coming out.

Basically what my question is, is how do I match up the output of the evaluation with the text label so that I can use it after training without hard coding the labels up front


texts: averaging multiple scores on small chunks of data or raw score on single collated data

texts:
I am using IBM Watson tool to determine tones (https://tone-analyzer-demo.mybluemix.net/) and personality scores (https://personality-insights-livedemo.mybluemix.net/) on different files containing natural language. What would be the recommended strategy and rationale for choosing between:

Getting scores from different files and averaging/manipulating them.
Getting one raw score on aggregated data and using it.


texts: How to fix "'The `start` argument could not be matched to a location related to the index of the data." Error?

texts:
When I try to predict the results using ARIMA for a specific train/test split, its throwing an error like this:

"'The start argument could not be matched to a location related to the index of the data"

This the code i have been trying:
pred0 = results.get_prediction( start = '2019-06-20', dynamic=False)

Same happens for get_forecast() method.


texts: Imbalanced dataset with 3 classes xgboost scale_pos_weight parameter

texts:
The xgboost classifier states the use of parameter scale_pos_weight for 2-class problems.

I have a highly imbalanced dataset with 3 classes. Classes '1' and '-1' are very rare (~1% of dataset) and class '0' is very common.

How do I set this scale_pos_weight parameter in the xgboost classifier correctly for my classification problem?


texts:
I understand how doc2vec works, but I am unclear the best practice on feeding in data.

Suppose we have a document with multiple sentences

['I really love football. Peyton Manning was a great player']

If we feed this into the algorithm as is, the window for 'Peyton' could include ['love','football','Manning','was']

This doesn't make intuitive sense to me, however, since the words come from different sentences.

Any suggestions?


texts: What are some good sources to learn fraud/anomaly detection in normal/time-series data?

texts:
I would like to know more on fraud/anomaly detection. I am looking for good source or survey article/book etc out there which will give me some preliminary idea of the area.

Any suggestion is greatly appreciated.

Thanks


texts:
I am doing credit risk modelling on costumer transaction data a part of which looks like this :

str(x)
'data.frame':   412516 obs. of  26 variables:
 $ Tenure           : num  1.26 1.25 1.26 1.31 1.32 ...
 $ Product          : Factor w/ 24 levels "BACKHOE LOADER",..: 4 4 4 9 9 9 9 9 9 9 ...
 $ Net.Exposure     : num  333339 528049 327335 350000 460000 ...
 $ OD.On.31.01.2017 : num  0 90386 0 0 1099692 ...
 $ LM.Bucket        : Ord.factor w/ 11 levels "0"<"1 TO  30"<..: 1 1 1 1 11 11 11 11 11 11 ...
 $ Bucket           : Ord.factor w/ 11 levels "0"<"1 TO  30"<..: 1 3 1 1 11 11 11 11 11 11 ...
 $ Billing          : num  65380 0 8800 6339 8331 ...
 $ Fin.IRR          : num  13.5 14.6 14.6 18.1 23.3 ...
 $ NPA.Flag         : Factor w/ 2 levels "No","Yes": 1 1 1 1 2 2 2 2 2 2 ...
 $ Inst.Due         : num  0 0.85 0 0 3 3 3 3 3 3 ...
 $ FR.On.31.01.2017 : num  65380 0 38940 35043 499860 ...
 $ POS.On.31.01.2017: num  56453 0 32920 33368 293943 ...
 $ Del.String       : int  2 1 1 1 53720 53720 53720 53720 53720 53720 ...
 $ Territory        : Factor w/ 43 levels "AGRA","AHMEDABAD",..: 41 41 41 41 41 41 41 41 41 41 ...

The variables like OD(Overdue) and LM.Bucket( How many months he has been due on his loan payment till last month) change every month .I have 2 tasks :Predict Bucket and NPA Flag(Non performing asset)

I built a model for this based only on the Jan data(x). But my question is since these variables change every month, should i treat this as a sequential data and build a deep learning model(HMM/NN) on it? If i should what should I do with the static variables like Product type etc.?

I asked my boss regarding the same and he said it shouldn't be done because external economy factor change with time. Is that a reason for concern?


texts:
I have rather theory-based question as I'm not that experienced in encoders, embeddings etc. Scientifically I'm mostly oriented around novel evolutionary model-based methods.

Let's assume we have data set with highly correlated attributes. Usually encoders are trained to learn representation in lesser number of dimensions. What I'm wondering about is quite the opposite. Would it be possible to learn encoding to higher number of dimensions but less correlated (wishfully non-correlated)? The idea is to turn less-dimensional, very tough problem to high-dimensional but easier one. Kinda unwrap those intricate correlations using NN and decode solutions later.

Edit 1
Of course we assume we know correlation mapping really good. How exactly could I use correlation mapping to unwrap it? Is it fundamentally possible to unmap attribute dependencies?


texts: Is it possible for a DCGAN to do regression? What are some examples of this?

texts:
I'm currently a student doing some machine learning projects, and I want to use generative adversarial networks to train some data to discern for example, how old someone is. The intended output is a regression continuous output in term of months.

I see the DCGANs and GANs are targeting mainly classification problems. Can they be used to solve regression problems?

Thank you.


texts: In supervised learning, why is it bad to have correlated features?

texts: I read somewhere that if we have features that are too correlated, we have to remove one, as this may worsen the model. It is clear that correlated features means that they bring the same information, so it is logical to remove one of them. But I can not understand why this can worsen the model.


texts:
From examples outside of the documentation, I thought I understood the examples of the .where() method.  Basically, it seems to be a another way to filter a dataframe.
However, when I checked the documentation itself for an example of how to use .where(), it was counterintuitive.
The documentation provides this example:
 df = pd.DataFrame({'A': [1, 2, 3],
                    'B': [4, 5, 6],
                    'C': [7, 8, 9]})

df.where(lambda x: x > 4, lambda x: x + 10)

[output]:
    A   B  C
0  11  14  7
1  12   5  8
2  13   6  9

It seems to me that this code should filter for all values greater than four.  According to my logic,
df.where(lambda x: x > 4, lambda x: x + 10)

should add 10 to all values greater than 4, changing the output to
    A   B  C
0   1   4 17
1   2  15 18
2   3  16 19

Could someone please explain to me the error in my logic?


texts: Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors

texts:
I am working on using the DAIC-WOZ Dataset of audio and video extracts for depression detection. While training the audio files, which has three input of eGeMAPS, MFCC, and DENSENET arrays. They are input into a BLSTM model.  I am using an AzureML server GPU-N6 for training.
machine specs : STANDARD_NV6 (6 Cores, 56 GB RAM, 380 GB Disk) GPU - 1 x NVIDIA Tesla M60
It exits with code 137, which is an out of memory error. I see the data is downloaded into the local node and NumPy array is created. the OOM happens when the batch processing starts.
I have tried to run the model with only one input (eGeMaps) but run into memory issues.
My question is, is there a way to batch load the NumPy file, similar to an image dataset? I have tried to split the input into individual data files and use tensor dataset or slices, but it doesn't work when loading a NumPy file. Ideally, I would like an equivalent of ImageDataGenerator for NumPy files.
EGEMAPS={'FEATURES':100,'TIMESTEPS':14050}
MFCC={'FEATURES':100,'TIMESTEPS':14050}
DENSENET201={'FEATURES':1920,'TIMESTEPS':1415}
The training dataset consists of 136 participants each with an array of the sizes mentioned above. I created a single large NumPy array, each row representing a participant and array of the features.
This is the model I am training.
#Create Model :
inputs_eGeMAPS = Input(EGEMAPS_INPUT_SHAPE)
inputs_MFCC = Input(MFCC_INPUT_SHAPE)
inputs_Densenet = Input(DENSENET201_INPUT_SHAPE)

blstm1_eGeMAPS = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")

blstm2_eGeMAPS= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_eGeMAPS)
print(f"blstm1_eGeMAPS:{blstm1_eGeMAPS}")

blstm1_MFCC = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")

blstm2_MFCC= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_MFCC)
print(f"blstm1_MFCC:{blstm1_MFCC}")

blstm1_Densenet = Bidirectional(LSTM(200,return_sequences=True,recurrent_dropout =0.2,activation='relu'))(inputs_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")

blstm2_Densenet= Bidirectional(LSTM(200,recurrent_dropout =0.2,activation='relu'))(blstm1_Densenet)
print(f"blstm1_Densenet:{blstm1_Densenet}")

audio_lstm_output = concatenate([blstm2_eGeMAPS,blstm2_MFCC,blstm2_Densenet])

dense1= Dense(500,activation='relu')(audio_lstm_output)

print(f"dense:{dense1}")

dense2= Dense(100,activation='relu')(dense1)
print(f"dense:{dense2}")

dense3= Dense(60,activation='relu')(dense2)
print(f"dense:{dense3}")

output = Dense(1)(dense3)
print(f"output:{output}")

model = Model(inputs=[inputs_eGeMAPS,inputs_MFCC,inputs_Densenet], outputs=output)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])

print(model.summary())

epochs = 1
hist = model.fit([BoAW_eGeMAPS_Dev_tf,BoAW_MFCC_Dev_tf,Densenet_Dev_tf],Y_train_tf,epochs=epochs, batch_size=batch_size,validation_split=0.2,shuffle=False)


texts: Can we quantify how position within search results is related to click-through probability?

texts:
Suppose, for example, that the first search result on a page of Google search results is swapped with the second result. How much would this change the click-through probabilities of the two results? How much would its click-through probability drop if the fifth search result was swapped with the sixth?

Can we say something, with some level of assurance, about how expected click-through probabilities change if we do these types of pairwise swaps within pages of search results?

What we seek is a measure of the contribution to click-through rates made specifically by position bias.

Likely, how position ranking would affect the sales in Amazon or other online shopping website? If we cast the sales into two parts, the product quality and its ranking effect.

sales = alpha*quality + beta*position + epsilon

How can we quantify the beta?


texts: My Keras bidirectional LSTM model is giving terrible predictions

texts:
I am trying to predict velocity (dynamics) values for notes that make up a piece of music using a bi-directional LSTM, following this blog post pretty closely: http://imanmalik.com/cs/2017/06/05/neural-style.html

I think I have set up everything correctly, but when predicting my model seems to produce only gibberish. Here's a plot of a prediction done on a validation sample (y-axis is always time and x-axis is, for I., a note activation array; for II. the predicted velocities and for III. the true, expected velocities). As you can see, the predicted velocity values (in the middle graph) are identical for all time steps (save for a few in the very beginning and at the very end, but that's hard to see in the plot):

For this particular prediction I had trained my model with 44 batches of 4 samples each and for 1 epoch, but I have also tried training it for 20 epochs and with different batch sizes and it doesn't seem to give better results.

Using TensorFlow backend.
Loading numpy data ...
173 train sequences
10 test sequences
x_train shape: (173,)
y_train shape: (173,)
Setting up model ...
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
bidirectional_1 (Bidirection (4, None, 128)            123392
_________________________________________________________________
bidirectional_2 (Bidirection (4, None, 128)            98816
_________________________________________________________________
bidirectional_3 (Bidirection (4, None, 128)            98816
_________________________________________________________________
time_distributed_1 (TimeDist (4, None, 88)             11352
=================================================================
Total params: 332,376
Trainable params: 332,376
Non-trainable params: 0
_________________________________________________________________
None
Training model ...
Epoch 1/1
2018-03-10 14:47:27.650226: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX
44/44 [==============================] - 2145s 49s/step - loss: 0.0783 - mean_squared_error: 0.0783
Saving model ...
Loss and metrics: [0.002714238129556179, 0.0027141105383634567]

So, in other words, I end up with a validation MSE of 0.0027 after 1 epoch. Here is the code for my Keras model:

number_of_notes = 88
input_size = number_of_notes * 2 # 88 notes * 2 states (pressed, sustained).
output_size = number_of_notes # 88 notes.

units = 64
dropout = 0.2 # Drop 20% of units for linear transformation of inputs.

model = Sequential()
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout),
                        input_shape=(None,input_size),
                        batch_input_shape=(args.batch_size,None,input_size)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(TimeDistributed(Dense(output_size, activation='sigmoid')))
model.compile(loss='mse', optimizer=Adam(lr=0.001, clipnorm=10), metrics=['mse'])

print('Training model ...')

number_of_train_batches = np.ceil(len(x_train)/float(args.batch_size))
model.fit_generator(batch_generator(x_train, y_train),
                    steps_per_epoch=number_of_train_batches,
                    epochs=args.epochs)

Here is how I predict:

prediction_data = np.load(path)

# Copy prediction input N times to create a batch of the right size.
tiled = np.tile(prediction_data, [args.batch_size,1,1])

raw_prediction = model.predict(tiled, batch_size=args.batch_size)[0]
prediction = (raw_prediction * 127).astype(int) # Float -> MIDI velocity.

So what I am asking is, where am I likely to be going wrong with my model? Which parameters should I try changing in order to get better results? Or how can I change my model or training set-up to get better results? Etc. Any and all help is appreciated!


texts: Logistic regression if 3 categories in outcome variable

texts:
Logistic regression is generally performed if there are 2 categories in outcome variables. I just tried it for iris dataset with species as y variable which has 3 categories. I used following code:

import pandas as pd
import matplotlib.pylab as plt

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()

from sklearn import datasets
iris = datasets.load_iris()

clf.fit(iris.data, iris.target)
logcoefdf = pd.DataFrame(data=clf.coef_,
     columns=["SL", "SW", "PL", "PW"],
     index=['setosa','versicolor','virginica'])
print(logcoefdf)
logcoefdf.plot.bar()
plt.show()

The printout and plot of coefficients is as follows:

                  SL        SW        PL        PW
setosa      0.414988  1.461297 -2.262141 -1.029095
versicolor  0.416640 -1.600833  0.577658 -1.385538
virginica  -1.707525 -1.534268  2.470972  2.555382

(I have labelled rows by names of species but I am not sure if this is correct).

From above output I get following plot:

What is the interpretation of these results? Does it mean that petal length (PL) is lowest in setosa and highest in virginica group? And both sepal width and petal width are less in versicolor species? Thanks for your insight.

Edit:
If I use only 2 categories of iris dataset, I get only one set of coefficients:

clf.fit(iris.data[0:100,:], iris.target[0:100])
print(clf.coef_)

Output:

[[-0.40731745 -1.46092371  2.24004724  1.00841492]]

Is it that Logistic regression is being performed for all possible combinations of categories, i.e. setosa vs versicolor, versicolor vs virginica and virginica vs setosa?


texts: Using LSTM to clear up corrupted text files

texts:
LSTM can be used to generate text, can they be used to fix corrupted text files?

Say that my original was:

Alice was beginning to get very tired of sitting by her sister on the
bank, and of having nothing to do

Now it has become:

Alice0004 XGFT was beginning to get 900io ALPHA very tired 00New York 333 of siing XXXYYY by her siter on THIS IS 898 the
bank, and of &*& HHH GTY DOG AND CAT having nothing to do.

Basically, a lot of crap was added (some meaningless words but also meaningful words, and some letters may have been erased).

Is there any way LSTM would be able to fix this? (Maybe by cleaning some beforehand and then let LSTM do the rest)


texts: Application of Deep Reinforcement Learning

texts:
I'm new to deep learning, and especially to reinforcement learning. I would like to know if it's possible to predict which combination of hashtags (from a subset of chosen hashtags) would produce the most likes for a certain image.

Is it possible to have a convolutional neural network with each hashtag as a label, and take something like reward = likes / followers as a reward in a reinforcement learning like scenario?

In what other way could I face this problem? My goal isn't to predict the amount of likes, but to maximize the probability to get the most likes.

I chose this title because I think the answer could actually be question agnostic: I could use the same knowledge to define which combination of stocks would maximize my investment.

Thanks in advance.


texts: What are the merges and vocab files used for in BERT-based models?

texts: The title says it all. I see plenty online about how to initialize RoBERTa with a merges and vocab file, but what is the point of these files? What exactly are they used for?


texts: Clustering users based on buying behaviour

texts:
I have a set of data which indicates purchase transaction of users (~1 million records). User can have more than 1 purchase across time. Data is spread over 6-7 months.
Attributes that I have are

userid
merchant
amount
day of purchase
date of purchase
time of purchase

Goal -

Cluster users based on their purchase behaviour, so that I can target set of users (Promotions) based on their cluster behaviour.
Eg - I want to send promotions to user who execute XX behaviour

What I have tried -

I tried KMeans Clustering in Python Sklearn for 10,000 sample records. By using "elbow method" I have identified that 6-7 cluster are optimal.
Still, Clusters are not really helpful in identifying similar group of users. Clusters does not make sense.

Questions -

1) Am I doing this correctly?

2) What would be the ideal algorithm for this type of use case. Should I be looking in Clustering or something like Recommendation algorithm (Group similar users)

3) Questions/Goal/Target group of users might change over time. How can we handle that?


texts: Sparse connections in feedforward network tensorflow or pytorch?

texts: I want to create sparse feed-forward networks in Pytorch and Tensorflow, i.e., say each node is only connected to k number of neurons of the next layer where k is strictly less than the total number of nodes in the next layer. But all the tutorials/examples I have seen so far are for fully connected feed-forward networks. Is there any way to construct such sparse networks in these tools?


texts:
I want to build a recommender system that suggests the next step in your career.

About the dataset. About 50'000 Users with following informations:

Skills (tags, string value)
every job they did (embedded job titles)
every school they went

Now the recommender system should tell you what the next step in your career can be, and if you need a another skill or degree for that step.

How should you build such a system? (Supervised, Unsupervised (Recommender System,...), ...)


texts:
I am trying to use the groupby functionality in order to do the following given this example dataframe:
dates = ['2020-03-01','2020-03-01','2020-03-01','2020-03-01','2020-03-01',
        '2020-03-10','2020-03-10','2020-03-10','2020-03-10','2020-03-10']
values = [1,2,3,4,5,10,20,30,40,50]
d = {'date': dates, 'values': values}
df = pd.DataFrame(data=d)

I want to take the largest n values grouped by date and take the sum of these values. This is how I understand I should do this: I should use groupby date, then define my own function that takes the grouped dataframes and spits out the value I need:
def myfunc(df):

    a = df.nlargest(3, 'values')['values'].sum()

    return a

data_agg = df.groupby('date').agg({'relevant_sentiment':myfunc})

However, I am getting various errors, like the fact that the value keep is not set, or that it's not clearly set when I do specify it in myfunc.
I would hope to get a dataframe with the two dates 03-01 and 03-10 with respectively the values 12 and 120.
Any help/insights/remarks will be appreciated.


texts:
This is a small conceptual question that's been nagging me for a while: How can we back-propagate through a max-pooling layer in a neural network?

I came across max-pooling layers while going through this tutorial for Torch 7's nn library. The library abstracts the gradient calculation and forward passes for each layer of a deep network. I don't understand how the gradient calculation is done for a max-pooling layer.

I know that if you have an input ${z_i}^l$ going into neuron $i$ of layer $l$, then ${\delta_i}^l$ (defined as ${\delta_i}^l = \frac{\partial E}{\partial {z_i}^l}$) is given by:
$$
{\delta_i}^l = \theta^{'}({z_i}^l) \sum_{j} {\delta_j}^{l+1} w_{i,j}^{l,l+1}
$$

So, a max-pooling layer would receive the ${\delta_j}^{l+1}$'s of the next layer as usual; but since the activation function for the max-pooling neurons takes in a vector of values (over which it maxes) as input, ${\delta_i}^{l}$ isn't a single number anymore, but a vector ($\theta^{'}({z_j}^l)$ would have to be replaced by $\nabla \theta(\left\{{z_j}^l\right\})$). Furthermore, $\theta$, being the max function, isn't differentiable with respect to it's inputs.

So....how should it work out exactly?


texts: Computing class_weights using native tf.Dataset

texts:
There are already some answers on DSE about computing class-weights here but they require you to already know the number of samples in each class and assume it is numerically labeled.
I used tf.keras.image_dataset_from_directory function to construct the Data Generator with classes as string with a little optimization to yield a tf.Dataset.
Since the function already knows the count of each class and its corresponding name, can we directly calculate/implement those class_weights natively instead of constructing a dict by hand (and using Scikit-Learn?)


texts:
I have 400 time series of length 50.
200 of them have values between 1-10 and are considered of type A.
The rest 200 have values 1-10 with the exception that 3 from the total of 50 data points have value 20, and are considered of type B.

I am trying to use a RNN network to make it learn these differences using tensorflow.

I use GRU cell with input size is 50 (the whole time series), internal state is 100, 3 layers, feature length is 1 and dropout 0.8.
Batch size is 1 (1 whole time series of 50 values)
80% of the series are used for training and 20% for test, evenly distributed between type A and type B.

I use softmax as activation function and for gradient descent optimization i use RMSPropOptimizer. Also i am not changing the data at all before feeding it into the neural network, i tried normalizing them using (x-min()/max()-min()) but it didn't work.

When i am trying to train it, i always get 50% error. Maximum epochs i tried are 10 and it still was 50%.

The same network can classify correctly if i use it on data that have way more differences than just on these 3 points, even on small dataset of about 50 runs of each type.

Is it impossible for a RNN to learn these small differences on just those 3 data points out of the 50?
What would you suggest i can change in my configuration?
I also tried with 2000 time series and still the error is the same.

Also note that these 3 data points exist at the beginning of the time series. (in case this has something to do with my issue).


texts: Training of Region Proposal Network (RPN)

texts:
There are some interesting literature about RPNs (Region Proposal Network). The most concise and helpful documentation that I found so far is the following: https://www.quora.com/How-does-the-region-proposal-network-RPN-in-Faster-R-CNN-work?share=1.

But there is something that I still don't understand through my various lectures. RPNs are designed to propose several candidate regions. From which, a selection will be done to know which candidates fits our needs.

But, RPNs and neural network in general are deterministic. Thus, once trained, they will always produce the same output for a given input; there is no way to query new candidates given the same input image. As far as I understood, RPNs are trained to produce a fix number of proposal, for each new image. But how the training work then? If the RPN has to produce 300 candidates, what should be the labeled data that we use for the training, knowing that a training image probably won't have more than 5 golden truth bounding boxes?

And then, knowing that the bounding box sizes are not consistent among candidates, how does the CNN behind operates with the different size of the input?


texts: normalization/denormalization for linear regression problem

texts:
My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after prediction, I mulitiplied this scale value back.

But since I used a DNN, wouldn't the nonlinear change the scale so make the multiply not valid? e.g.

given input data: X, label: y;
y' = y/scale
X' = X/scale
predicted = f(X')
predicted_update = predicted * scale

Anyone could provide some advice on whether I could do this or it's actually not correct? How do we handle this kind of problem?


texts: Keras: extreme spike in loss during training

texts:
I am training an LSTM for time series forecasting and it has produced an extremly high loss value during one epoch:

Epoch 00043: saving model to /...
904/904 - 2s - loss: 0.7537 - mean_absolute_error: 0.5772 - val_loss: 1.4430
- val_mean_absolute_error: 0.7124

Epoch 00044: saving model to /...
904/904 - 2s - loss: 240372339275.7649 - mean_absolute_error: 56354.0078
- val_loss: 4.6229 - val_mean_absolute_error: 1.5681

Epoch 00045: saving model to /...
904/904 - 2s - loss: 1.3348 - mean_absolute_error: 0.7894 - val_loss: 2.2875
- val_mean_absolute_error: 1.1510

My model:

model = keras.Sequential()
model.add(keras.layers.LSTM(360, activation='relu', input_shape=(N_STEPS, n_features)))
model.add(keras.layers.Dropout(0.1))
model.add(keras.layers.Dense(1, activation='relu'))
model.compile(optimizer='adam', loss='mse', metrics=['mae'])

What is the cause of this?

Theoretically, it shouldn't be able to have such a high loss unless it outputs very high values for that epoch. Which is strange since the model's output makes sense during other epochs.


texts: How to deal with different length entities in a Keras DataGenerator?

texts:
I'm solivng a prediction problem where I need to predict the demand of multiple articles based on their performance during the last 7 days. To get the most out of the data I am trying to implement a rolling window approach where I use 7 days as the training input (x) and the 8th day's performance as the training label (y).
As I am using Keras as a deep learning framework, how do I combine this rolling window approach with a custom data generator that feeds my network during training? The different articles have different data lengths (i.e. article A has 200 days worth of data, article B only has 10 days worth of data).
I implemented a version where I only use one sample per article but that drastically decreases the amount of data I have for training.


texts: How do you classify groups when the inputs are paired?

texts: My data is like this: there is a choice between two alternatives and the customer chooses one. Each time the alternatives are different. I would like to characterize a desirable alternative. I don't want to use a normal classification algorithm because I don't want to lose the information that one alternative was chosen over the other in the pair. Does anyone have suggestions for ways to do this, or types of algorithms for solving this problem? Thanks!


texts: AdaBoost decision_function() outputs in binary classification with sklearn

texts:
As I understand it based on some study of the source code, I would expect, when using AdaBoost, that values obtained by calling decision_function() would be bounded between -1 and 1. This is because it's the weighed average of the probabilities. However, as you can see in the histogram below, the values seem to range from a little under -2 to a little over +2.
Why is this? Am I under some misunderstanding about how these values are calculated?


texts:
I am working on machine learning classification problem with two classes (0/1). I would like to build a prediction model. The problem is that I have a small dataSet of shape=(89, 21) which may caused over-fitting. problem (20 independent variables). I notice that the results are highly affected by the train data and test data sizes( i.e. how the split was done ). the best results with LR was 0.90 and the worst I got 0.74.
Algo
I Split data using this instruction :
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.37, random_state=2)

then I did an upsimpling of training set since it was unbalanced {0}=62 {1}=27
the  table below  contain the best results on the test-set I got (test error).
$$\begin{array}{c|c}
 & \text{Accuracy} & \text{ROC } \\ \hline
\text{LogisticRegression } & 0.909091 & 0.928571 &  \\ \hline
\text{DecisionTree } & 0.939394 & 0.934524 &  \\ \hline
\text{adaBoost } & 0.848485 & 0.845238 &  \\ \hline
\text{RandomForest } & 0.878788 & 0.928571 &  \\ \hline
\text{GradientBoosting } & 0.848485 & 0.827381 &  \\ \hline
\end{array}$$
the next table contain the training error
$$\begin{array}{c|c}
 & \text{Accuracy} & \text{ROC } \\ \hline
\text{LogisticRegression } & 0.963415 & 0.963415 &  \\ \hline
\text{DecisionTree } & 1.00 & 1.00 &  \\ \hline
\text{adaBoost } & 1.00 & 1.00 &  \\ \hline
\text{RandomForest } & 1.00 & 1.00 &  \\ \hline
\text{GradientBoosting } & 1.00 & 1.00 &  \\ \hline
\end{array}$$
I need some guideline (ideas, tutorials,...) about how to manage the over-fitting problem .
Thanks


texts: Targeted information extraction / focused extractive summarization

texts:
I have a large collection of project manuals, each with a large number of pages. Each manual contains some form of summary paragraphs, although these are not necessarily similar in structure or format from one to the next. The rest of the manual generally contains a large amount of various information in relation to the project, and is not always relevant to the desired content to be extracted and summarized.

In theory-

paragraph 1 - Project Summary (Extract this)
paragraph 2 - Background info (ignore)
paragraph 3 - Background info (ignore)
paragraph 4 - Background info (ignore)
paragraph 5 - Project Requirements (extract this)

Is it possible to somehow selectively target paragraphs/ sentences within a document for extractive summarization, and if so, is possible to train a model with datasets containing full texts and their desired summarizations?

So far, I have tried general extractive summarization methods utilizing TF-IDF. However, due to the substantial amount of extraneous information within the document, critical text within the summary paragraphs is usually ignored. I've considered manually increasing word counts across various keywords, but I think this will still ignore relevant sentences and phrase within close proximity (same worry with binary classification of desired paragraphs/ sentences).

Can this be done, and is there a better approach than what I've been trying so far?


texts: Visualization of an Order Verified by Wilcoxon

texts:
The background: After I've conducted a survey about preferences of users in ridesharing, I now want to construct an order of importance of these preferences (this is to answer one of my research questions). For instance price might be more important than comfort of the vehicle might be more important than the played music. I collected the data based on a five-point likert-scale.

My procedure so far:

Of course clearing the data
Generate an order via a simple approach: very important gets $4$, important gets $3$ and so on. Then I build a sum over all answers and I sort the preferences based on their sums.

The second step provides me with an order, which is not statistically verified. Therefore,

I perform a one-sided Wilcoxon signed-rank test for each pair of preferences.

The resulting p-values are in the following matrix:

The visualization can be read as follows: preference a is statistically proven to be not less important than preference d to j for all common significance levels. (The order of preferences comes from the second step.)

The problem is, I've collected $42$ preferences and the corresponding visualization has the following problems:

$42$ is quite a lot and the visualization gets confusing.
Moreover, to understand what is shown in the graphic a lot of brain work is needed, especially for an unexperienced user.

Do you have an idea how to visualize the result better?


texts: How do I "tie" the weights of parallel layers of an ANN?

texts:
My problem is fairly simple, but I am not sure if my implementation in Keras does what I think it does. I have a set of features in a certain space of dimension N. I want to use an ANN to learn the "mapping" of these features to a different space (with the same dimension N). The mapping is done with a custom loss function that uses triplets of training data to infer this mapping. My first approach is shown below

I think that all 3 "mapping" layers should actually learn the same weights, so that the mapping of all 3 inputs follows the same pattern, and the sequence of the inputs doesn't matter. The green arrows signify that I want the weights of these layers to be the same.

My first implementation with Keras looks like this:

X1      = tf.keras.layers.Input(shape=(NO_FEATURES,))
X2      = tf.keras.layers.Input(shape=(NO_FEATURES,))
X3      = tf.keras.layers.Input(shape=(NO_FEATURES,))

shared_mapping = tf.keras.layers.Dense(NO_FEATURES, activation='relu')

Y1 = shared_mapping (X1)
Y2 = shared_mapping (X2)
Y3 = shared_mapping (X3)

model   = tf.keras.Model(inputs=[X1, X2, X3], outputs=[Y1, Y2, Y3])

Does that make sense? Does the above implementation implement my idea? Or is there another way to describe this? (Or is my idea totally wrong?)


texts:
In batch gradient descent, it is said that one iteration of gradient descent update takes the processing of whole entire dataset, which I believe makes an epoch.On the other hand, in mini batch algorithm an update is made after every mini batch and once every mini batch is done, one epoch is completed. So in both cases, an epoch is completed after all  the data is processed.I do not quite get what makes mini batch algorithm more efficient.

Thanks,


texts: What is the difference between residual sum of squares and ordinary least squares?

texts:
They look like the same thing to me but I'm not sure.

Update: in retrospect, this was not a very good question. OLS refers to fitting a line to data and RSS is the cost function that OLS uses. It finds the parameters that gives the least residual sum of square errors. It is called ordinary in OLS refers to the fact that we are doing a linear fit.


texts: How to interpret FPR and TPR in ROC curve?

texts:
I am playing with ROC and trying to draw some curves. I am using example from this scikit page. I do not understand one thing: when I print out the content of tpr and fpr, I see two arrays of numbers (21 elems in each) - why is that?

TPR (True Positive Ratio) is a proportion of those tuples classified as positives to all real positive tuples. So, it should be one number

I see it as follow: I take classifier (like Decision Tree), train it on some data and finally test it. Then I can calculate TPR and FPR and I should have only two values.

Instead, I receive arrays. Why is that? I tried to change classifier from that example to Decision Tree and then I receive only one point (which is usually [0, 1]). How can I draw ROC curve on decision tree? Or - to be more precise with my question - why do I get an array of values as my TRP/FPR instead of single values?


texts: What is the term for when a model acts on the thing being modeled and thus changes the concept?

texts:
I'm trying to see if there is a conventional term for this concept to help me in my literature research and writing.  When a machine learning model causes an action to be taken in the real world that affects future instances, what is that called?

I'm thinking about something like a recommender system that recommends one given product and doesn't recommend another given product.  Then, you've increased the likelihood that someone is going to buy the first product and decreased the likelihood that someone is going to buy the second product.  So then those sales numbers will eventually become training instances, creating a sort of feedback loop.

Is there a term for this?


texts:
I am training a model in Spark using Dl4J library in Yarn-Cluster mode. When I train the model on 2 lakh data (approx 200MB) then the job succeeds but when I go to train the model with 3 lakh data (approx 300MB) then the job finishes with a fail status and the model doesn't get saved in HDFS. So somehow I am reaching to a bottleneck over there.

I am using 9 containers with 1 core and 1.5GB memory to run the training job.

There is no error in yarn or spark logs.

How do I identify where the problem has occurred? How do I identify if it is the insufficient resources that is causing the problem or is the code that is the problem?


texts:
I am a beginner in CNN theory and would like to understand the usage of residual modules better.
As far as I understand residual modules can be skipped, only the activation function must be computed with the given input.

My question is: How does the network know if it should skip the residual module or not?


texts: What is the best question generation state of art with nlp?

texts:
I was trying out various projects available for question generation on GitHub namely NQG,question-generation and a lot of others but I don't see good results form them either they have very bad question formation or the questions generated are off-topic most of the times, Where I found one project that actually generates good questions

bloomsburyai/question-generation

It basically accepts a context(paragraph) and an answer to generate the question and I am trying to validate the questions generated by passing the generated question along with the paragraph to allenNLP

Answer generation for a question

And then I am trying to make sure the generated answers are correct for the questions generated with calculating the sentence embedding for both the answers(AllenNLP and PotentialAnswer) using Universal Sentence Encoder and a cosine distance to get how similar the answers match and the filtering question that has least cosine distance.

Wanted to know if this is the best approach or Is there a state of the art implementation for question generation? Please suggest


texts: Varying strength of prior for MCMC hierarchical linear model

texts:
I am training an MCMC model in using Pymc3.

My aim is to build a series of linear regression models which will predict the time to unload a truck, based on the number of crates to unload.

I have data for 2000 locations and they are divided into 4 categories of location. Locations of the same category tend to have similar unloading times.

For each location I have a series of recorded data points:

number of crates on the truck
how long it took to unload

I want to train a linear model for each location of this form:

y = α + βx

where y is the time, x is the number of crates.

For some locations I may have only 20 data points, others I may have 10,000.

Obviously when I have 10,000 data points there are enough data points to train a simple linear regression at that location with no other input.

However when I have smaller amounts of data, such as 20 points, I am hoping to train an over-arching model on all data points of locations in that category, and use its α and β as priors for the α and β of that location.

I have been following the very helpful example at https://github.com/parsing-science/pymc3_quickstart_guide/blob/master/Quickstart_Guide_no_errors.ipynb.

My questions are:

how can I vary the strength of the prior so that it's a really strong prior if I have few data points, and it weakens to zero for the locations where I have enough data points?
what values should I set for sigma_alpha and sigma_beta, the stdevs of the prior distributions of α and β?  The Pymc3 examples use a uniform distribution, would the best idea to be to work out the approximate distribution by training a linear model at each location and getting the stdev of the parameters?

Here's my MCMC code:

import theano.tensor as T
import numpy as np
import pymc3 as pm
import pandas as pd

# load pandas dataframe df_train which has columns location_id, crates, unloading_time
# load location_category_lookup_dict which gives the category of each location
# load category_model_lookup which gives the pre-trained linear model for each of the 4 categories

num_features = 1

grouped = df_train.groupby(["location_id"])

location_no_to_mcmc_models = {}

for location_id, drops_at_this_location in grouped:
    location_category_id = location_category_lookup_dict[location_id]

    # Look up the intercept and coefficient in the previously trained linear model for this category
    model_for_this_locations_category = category_model_lookup[location_category_id]

    alpha_this_category = model_for_this_locations_category.intercept_
    beta_this_category = model_for_this_locations_category.coef_

    # Get the features (number of crates) and regressand (unloading time)
    features = np.asarray([[c] for c in drops_at_this_location["crates"]])
    y = np.asarray(drops_at_this_location["unloading_time"])

    # Set up the PyMC model
    hlm = pm.Model()
    with hlm:

        # Both alpha and beta are drawn from similar distributions
        sigma_alpha = pm.Uniform("sigma_alpha", .0, 20, testval=2.)

        sigma_beta = pm.Uniform("sigma_beta", .0, 4, testval=2.)

        # Simulate the alphas
        alpha = pm.Normal("alpha", alpha_this_category, sigma_alpha, shape=(1))

        # Simulate the betas
        beta = pm.Normal('beta', beta_this_category, sigma_beta, shape=(1, num_features))

        # Simulate the noise
        s = pm.Uniform("s", .01, 10, shape=1)

        yd = pm.Normal('y', mu=alpha[0] + T.sum(beta[0]*features, 1), tau=s[0] ** -2, observed=y)

        step = pm.NUTS()

    tr = pm.sample(int(1e4), step, model=hlm)

    mcmc_summary = pm.summary(tr)

    location_no_to_mcmc_models[location_no] = mcmc_summary

    print (location_id, "intercept:", mcmc_summary["mean"]["alpha__0"], "weight:", mcmc_summary["mean"]["beta__0_0"])


texts: Is there a way to find feature importance per each prediction?

texts:
I know that
X_train = [

[1, 2, 3, 4, 5, 6],
[7, 8, 9, 10, 11, 12],
[13, 14, 15, 16, 17, 18],
.....

]

clf.fit(X_train, y_train)
probas_test = clf.predict_proba(y_)

clf.feature_importances_

will give the feature importance of a classifier that has been fit to the Training dataset.
But, How can I use this trained model to give a similar kind of importance value when predicting against unseen data per INDIVIDUAL ROW. For example let,
X = [1, 2, 3, 4, 5, 2]

clf.predict_proba(X)

Then is there a way to get the importance of each column, which is specific to this particular row rather than considering a set of rows. Something like
clf.feature_importances_individual_

or even somthing like
clf.get_feature_importance(X)


texts: Confidence intervals in multivariate linear regression

texts:
I am fitting my data to a multivariate linear regression $Y = BX + \Xi$, where the response is bivariate $Y\in R^{n\times 2}$, and the predictor is uni-variate but elevated to the projective plane to account for the intercept $X\in R^{n\times 2}$.

Now, finding the best fit reduces to $\hat B = (X^T X)^{-1}X^T Y$.

But I am interested in finding a $0.7$ confidence region around $\hat B$. How do I do that?


texts: Logistic regression program does not give statistically correct result

texts:
I wrote a very simple and compact logistic regression program using theano. I am initialising my data randomly, and I have restricted the number of training steps to 1, so since the weights are also random, I should expect the result to follow a standard distribution. I have ten classes and 10 initial examples, so I am expecting my accuracy to be close to 10, most of the time, however I am often (too often) getting results that are quite far from 10 (like 0 or 49). I should expect to get those results sometimes, but not so often. Is there something wrong in my code?

import theano
import theano.tensor as T
import numpy as np
rng = np.random
#rng = np.random.RandomState(0)

ninputs = 100
nfeatures = 784
noutputs = 10
nhiddens = 50

#training rate
tr_rate = 0.1
training_steps = 1

D = (rng.randn(ninputs, nfeatures), rng.randint(size=ninputs, low=0, high=noutputs))

# Feed-forward
x = T.dmatrix('x')
t = T.ivector('t')

w = theano.shared(rng.randn(nfeatures, noutputs), name="w")
b = theano.shared(np.full(noutputs, 0.01), name="b")
sigma = T.nnet.softmax(T.dot(x,w) + b)
prediction = T.argmax(sigma, axis=1)

parameters = [w, b]

xent = -T.mean(T.log(sigma)[T.arange(t.shape[0]), t])          # Cost function

cost = xent.mean() + 0.01 * (w ** 2).sum()                     # Regularisation

gparameters = [T.grad(cost, param) for param in parameters]    # Compute the gradient of the cost

pars = zip(parameters, gparameters)

train = theano.function(
          inputs=[x,t],
          outputs=[xent],
          updates = [(param, param - tr_rate * gparam) for param, gparam in zip(parameters, gparameters)],
          allow_input_downcast = True )

predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    train(D[0], D[1])

result = predict(D[0]) - D[1]

N = ninputs

error = 0
for index in result:
    if result[index] != 0:
        error += 1

correct_guesses = N - error
accuracy = (N - error)*100/N

print
print "correct predictions = %f over %i examples" % (correct_guesses, N)
print "accuracy = %i%%" % accuracy


texts: Generate Knowledge graph from text paragraph as input using python

texts:
I want to plot a knowledge graph from text paragraph as input in python, So wanted to know is there any transfer learning model, machine learning method or python library which helps in generating knowledge graph directly.
Currently, I am using the spacy library to get dependency parsing tags of words and then I have categorised tags into the subject, relation, object buckets or as a node in the knowledge graph.
But this method is too static as it is difficult to extract info from any type of sentence i.e taking into consideration all types of grammar syntax and semantics in order to properly categorise them into nodes. so is there any alternate way of developing a knowledge graph so that it could work for any type of grammatical sentence it encounters and convert to a knowledge graph?


texts: Training Deep learning and validation loss

texts: I'm trying to replicate result of a paper. The paper is a U-net for De-noising of some images. So basically I have a simple U-net that I give noisy data as input and have denoised data as the wanted output (use l2/MSE loss) . So, in the paper and generally in most papers like this (deep learning applied to medical imaging), they say they run the model for something like 300 epochs or they say they run it like ~50 hours. My question is that aren't they supposed to put call backs for the validation loss, so once the validation loss stops improving they stop the training, otherwise the model will overfit? Also does the number of batches in an epoch matter. If because of memory constraints I use 2 batch sizes per epoch, would it be a bad practice?


texts:
I face a problem with using SHAP value to interpret the Tree-based model.
(https://github.com/slundberg/shapsd)
First, I have input around 30 features and I have 2 features that have high positive correlation between them.
After that, I train the XGBoost model(python) and look at SHAP values of 2 features the SHAP values have negative correlation.

Could you all explain to me, why the output SHAP values between 2 features don't have the correlation the same as input correlation? and I can trust that output of SHAP or not?

=========================

The correlation between input: 0.91788
The correlation between SHAP values: -0.661088

2 features are
1) Pupulation in province and
2) Number of family in province.

Model Performance
Train AUC: 0.73
Test AUC: 0.71

Scatter plot
Input scatter plot (x: Number of family in province, y: Pupulation in province)
SHAP values output scatter plot (x: Number of family in province, y: Pupulation in province)


texts: Does the type of y value affect the prediction power of model?

texts:
I am using the sunlight intensity time series data(X) to predict plant height(Y) in different locations using CNN model in Keras.

I am wondering if I change the group Y values into 2 categories: High and Low based on a threshold(so that the Y becomes dummy variable), will the model be less accurate in making predictions as compared to using the original numerical Y data? Thanks