texts: Are RNN or LSTM appropriate Neural Networks approaches for multivariate time-series regression?

texts:
Dear Data Science community,

For a small project, I've started working on Neural networks as a regression tool, but I am still confused about possibilities of some variants. Here's what I am aiming to do:

I have multiple input data time series $X(t)=[X_1(t), X_2(t), X_3(t),X_4(t)]$, and multiple target data time series which I want to modelize $Y(t)=[Y_1(t), Y_2(t)]$. All data are available for training.
I aim to train my model/regression on an interval $[t_0,t_n]$, and then be able to apply it on a larger different interval.
I know that relation between my $Y$ and $X$ are non-linear, but also that in need to take in account lag, or inertia. For example, $Y_1(t)$ is dependant of $X_1([t-dt_1,t])$ and $X_2([t-dt_2,t])$. All $dt_n$ are different, but I have an approximate idea of how 'far' I need to reach.

With this in mind, through some research I have been guided to focus on Recurrent Neural Networks (RNN) and Long Short Term Memory (LSTM) networks. I aim to use TensorFlow/Keras to work on this. However, after some reading, I'm getting confused with those solutions. Many people present them in prediction applications, which supposedely means that those networks use data from a time interval (either $X$ or $Y$ in my case) on an interval $[t_0,t_n]$ to predict $Y(t_{n+1})$. But my objective is to use $X([t_0,t_n])$ to modelize $Y([t_0,t_n])$ (on the same time interval). I am getting confused with this notion of "prediction".

Therefore, are RNN and LSTM networks appropriate solutions for my multivariate time series regression/model project? Or am I already going the wrong way?

As a beginner in this field, any reference or link to ressources/tutorial, or demo, is also gladly welcome.


texts: I am building a deep neural network to predict values 0 and 1 .My training data contains class labels 0 and 1 .Currently i am getting a cross validation loss around .25 . How good is the model performing?


texts: Predicting when a partition in Oracle database will be archived

texts:
We have an Oracle database in which the main table "A1" is partitioned by the hour of insertion of the row. A row once inserted, may be updated depending upon data in 6 other tables (B,C,D,E,F,G) in the same database.

A partition in table A1 typically gets updated for a few hours after it starts receiving data. After the updates to the partition are finished, the partition gets archived, i.e, it is moved from production to an archive database. The time of creation of a partition and the time of its archival are logged.

Now, my question is given the past history of creation and archival times of partitions, how can I predict when a partition created now in A1 will be archived? What kind of model (regression, etc.) is the best to answer this question? How can I set up a model that tracks the dependence of the archival time of a partition on the attributes in other tables (B,C,D,E,F,G)?


texts: can we learn a model to pre-process text?

texts:
I'm in a very dire situation where I have to preprocess the text but the text in the documents is very random. It is in the form of numerical points.

I want to remove a certain class of points  (like bullet points) from those documents but simple preprocessing is not helping as the points are appearing randomly anywhere in the document.

I was thinking of learning a model where the model identifies the points which I want to remove.

Is it viable? And If so what approaches should I look for.

[Natural language processing]
Thanks!


texts: Predicted probabilities of Multi Label Classification

texts:
I'm currently working on a Kaggle Competition wich objective is to predict probabilities of an ID belonging to each class. There are 4 posible classes.
The data is tabular and because it's a Kaggle 'just for fun' Competition, the features are already clean and the data is semi-ready to train the model.
After standarizing (MinMaxScaler()) the data and splitting into train, test and validation, the data is ready to train.
Ok, till there everything it's fine. We have our train data (X). For the target variable (Y) we use One Hot Encoder, so then we have a Nx4 matrix for Y (N: number of samples; 4 number of clases). I choose to encode the (Y) variable with One Hot Encoder because I need as output a matrix of shape Nx4 with the predicted probabilities for each sample of belonging to each class.

First approach was a Neural Network because with an output layer with softmax activation function it would give as output exactly what I wanted to. I tried with different architectures for the Neural Net but I can't make it work, the training and validation accuracy are divergent, so it doesn't generalize good.
After trying different Neural Nets, I tried with a Random Forest Classifier with Multi-Output, but in this case the predicted probabilities that it return are all for one-vs-rest models.
There is any way to solve a multi label classification with shallow machine learning algorithms so the output is like a softmax output layer?
Thank you for reading till here, any help will be very welcome.


texts: PCA and K means in the context of machine learning

texts:
I have a question regarding machine learning. I mostly do exploratory data analysis (EDA) of large 'omics datasets. I recently submitted a metabolomics based manuscript, and one of the reviewers was not happy with my analysis of the data. I think he as misunderstood the concept of machine learning, but I could be mistaken. I have added one of many examples where he refer to machine learning.
Reviewer: "PCA, K-means, and any other machine learning multivariate method of analysis must be performed on sex-specific groups"
I have always believed that machine learning is used for classifying and prediction purposes, but I dont seem to find any clear definition. Anyhow, I have three questions:

Can somebody define machine learning?
Is PCA machine learning? I have always thought its a method of dimension reduction that seeks to explain observed variance? It does not care about labels or groupings, nor does it predict or classify anything. Can somebody explain me where I got this wrong?
Is K means clustering per definition machine learning? This algorithm has been around since the late 60s, long before machine learning was even a thing. I understand that this algorithm is applied in machine learning, but so is linear regression and logistic regression - but we dont call that machine learning.

I would appreciate your input on this?


texts: How to decode encoded labels in Decision tree classifier

texts:
I have some dataset with procurements of organization where actually i'm working. The aim is to find most important features that describe why some processes of purchases is succesful, and why not succesful.
To solve this, I'm using Decision Tree Classifier.
First, I convert boolen columns to int
for col in ['flag_prequalification', 'flag_procurement_category_strategy']:
data.loc[:, col] = data.loc[:, col].astype(int)

Then, convert object type to string. This for label encoding
for cat_feature in categorical_features:
    data.loc[:, cat_feature] = data.loc[:, cat_feature].fillna('unknown').astype(str)
%%time
encoder = defaultdict(LabelEncoder)
data.loc[:, categorical_features] = (
    data
    .loc[:, categorical_features]
    .apply(lambda x: encoder[x.name].fit_transform(x), axis=0)
)

After, let's use log(sum) instead of sum
for col in ['sum_tru_no_nds (lot)', 'price (lot)', 'count (lot)']:
data[col] = (data[col] + 1).apply(np.log)

And, in the end plot the tree:
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus

import graphviz
# DOT data
dot_data = tree.export_graphviz(clf, out_file=None,
                                feature_names=None,
                                class_names=target_col,
                                filled=True)

# Draw graph
graph = graphviz.Source(dot_data, format="png")
graph

The question is, how can I explain this tree to others, more precisely how can I decode this labels in tree?


texts: oversampling plus down sampling using smote not working on random forests

texts:
I am trying to solve a classification problem on a highly imbalanced data set. I am using SMOTE to over sample the minority samples and down sample the majority ones. After creating a balanced data set, I applied the random forest model. But, the prediction error for the minority class is extremely high even after using a balanced data set. What could possibly be going wrong?

library(DMwR)
new.data <- SMOTE(Clicked ~ ., train, perc.over = 600, perc.under =  80)
table(new.data$Clicked)

rand.forest <- randomForest(Clicked ~., data=new.data, mtry = 7,
                            importance = TRUE, proximity=TRUE, ntree = 1000
                            )

#confusion matrix
table(yhat.rf, test$Clicked)

yhat.rf   0   1
      0 889  47
      1  57   7


texts:
When I create an InputLayer, a None dimension gets added. I don't know why this is.
    image_shape = ds_info.features['image'].shape
    input_shape = (ds_info.features['label'].num_classes, image_shape[0], image_shape[1], image_shape[2])

    print('input_shape: ', input_shape)

    ## Create the CNN
    # Create the input layer
    input_layer = Input(shape=input_shape)

    print('input_layer: ', input_layer.shape)

Results
input_shape:  (1623, 105, 105, 3)
input_layer:  (None, 1623, 105, 105, 3)


texts: What Machine Learning Technique can I use to judge boxing fights?

texts: I want to build a machine learning model that judges the fights based on the results of each round. Any suggestions on what techniques can I use?


texts: K-fold cross validation when using fit_generator and flow_from_directory() in Keras

texts:
I am using flow_from_directory() and fit_generator in my deep learning model, and I want to use cross validation method to train the CNN model.

datagen = ImageDataGenerator(rotation_range=15,width_shift_range=0.2,
                             height_shift_range=0.2,shear_range=0.2,
                             zoom_range=0.2,horizontal_flip=True,
                             fill_mode='nearest')

image_size = (224, 224)
batch = 32

train_generator = datagen.flow_from_directory(train_data,
                                              target_size=image_size,
                                              batch_size=batch,
                                              classes= classes_array)

I found this Youtube video and this Tutorial, But it is not use flow_from_directory().

Do you have any idea how do I use k-fold cross validation when using fit_generator and flow_from_directory() in Keras?


texts: Why accuracy scores reported by Keras are low and erratic while the loss on the validation set is decreasing?

texts:
I'm trying to build CNN to predict two-label classification problem. Unfortuenetely, I can't share my model architecture, but I compiled the model using:
model.compile(optimizer=optimizers.Adam(lr=1e-3),
              loss='binary_crossentropy',
              metrics=['accuracy'])

Once the training was finished, I plotted log-loss and accuracy scores on train vs validationd dataset. Log-loss looks like below:

so in my layman's eyes this looks perfectly fine. On the other hand, accuracy scores (see below, the labels are misleading) are erratic and Keras reports accuracy of 62.8% on the training set and 62.5% on the validation set at the end of last epoch.
This seems disturbingly low for me and the strong fluctuations on the validation set are concerning as well. I use a batch size of 32, so it may partly explain these flucutations but I don't think that's the reason behind the whole variance (although I may be entirely wrong...). However, when I calculated accuracy using sklearn accuracy_score function on my validation set:
from sklearn.metrics import accuracy_score
Y_prob = model.predict(X_test)
Y_pred = (Y_prob > 0.5).astype(float)
accuracy_score(Y_test, Y_pred)

I got a value of 0.99811817183879 and the contingency table of each of two labels looks like below:
import pandas as pd
pd.crosstab(Y_test[:, 0], Y_pred[:, 0], normalize=True)
col_0   0.0     1.0
row_0
0.0     0.969804    0.000680
1.0     0.000680    0.028837

pd.crosstab(Y_test[:, 1], Y_pred[:, 1], normalize=True)
col_0   0.0     1.0
row_0
0.0     0.981739    0.000296
1.0     0.000227    0.017738

I'm totally confused about this discrepancy. Therefore, I'd like to ask:

How Keras calculates this accuracy scores? Why is it so low if my post-analysis suggest otherwise?
Is there something wrong with my setup based on the training plots (particularly second one with accuracy)?


texts: Overfitting/Underfitting with Data set size

texts:
In the below graph,

x-axis => Data set Size
y-axis => Cross validation Score

Red line is for Training Data

Green line is for Testing Data

In a tutorial that I'm referring to, the author says that the point where the red line and the green line overlap means,

Collecting more data is unlikely to increase the generalization
performance and we're in a region that we are likely to underfit the
data. Therefore it makes sense to try out with a model with more capacity

I cannot quite understand meaning of the bold phrase and how it happens.
Appreciate any help.


texts:
I am training a CNN-LSTM-FC network with around 1 Million parameters network for spatial time series prediction but the validation loss does not even come close to the training loss after 1-2 epochs. Is this a clear case of overfitting? Any inputs would be appreciated!
I am using keras with tensorflow backend.


texts: Adding extra feature to word2vec embeddings for use in classification

texts:
I have a word2vec model that will transform my text into vectors, and I need to use them to classify the text. My data is essentially a timeseries of chat messages, so I think that the message frequency(Total messages / total time) holds a lot of information that should be used to classify.
I was thinking of taking the chat stream, combining them all into a single string, and then averaging all the word's word2vec representations together. But I also want the classifier to be able to use the message frequency.
Can I simply append the message frequency value onto the averaged word embedding vector?


texts: Future of deep learning (compared to traditional machine learning)

texts:
What do you think will be the future of deep learning? A lot of people talk about deep learning and I can see that it provides various possibilities. For instance, this question gives a nice explanation.

What's the future of deep learning? Will it become the main branch of machine learning or will the other branches of machine learning remain relevant as well? And what about different fields of research? Will it be more suitable for certain applications (like natural language processing) and less suitable for some other types of applications?

I'd love to hear your views on this, both from a practical as from a scientific point of view. Any practical experience or research that can help to resolve this question? More specifically: should we introduce deep learning in our company?


texts: Overcome memory limitation when downloading from database into Orange

texts:
I would like to run the association rule mining algorithm of the Orange library on a dataset that is stored in a PostgreSQL database. The table 'buildingset' contains the itemsets for each user, thus each record is related to a user, and each field is related to an item. The values are either 1 (smallint) or missing. The table has about 14,000 records and 31 fields.

When I try to run the algorithm on this dataset, I get the following error:

  ValueError Traceback (most recent call last):
    File "/opt/orange/orange3/Orange/canvas/scheme/widgetsscheme.py", line 722, in process_signals_for_widget
      handler(*args)
    File "/home/bdukai/.local/lib/python3.4/site-packages/orangecontrib/associate/widgets/owassociate.py", line 444, in set_data
      self.X = data.X
    File "/opt/orange/orange3/Orange/data/sql/table.py", line 353, in X
      self.download_data(AUTO_DL_LIMIT)
    File "/opt/orange/orange3/Orange/data/sql/table.py", line 333, in download_data
      raise ValueError("Too many rows to download the data into memory.")
  ValueError: Too many rows to download the data into memory.

Thus is there any way overcome this limitation, without upgrading the hardware?


texts: Building a machine learning model to predict crop yields based on environmental data

texts:
I have a dataset containing data on temperature, precipitation and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data.

Please note that the dataset has DAILY values for temperature and precipitation, but only 1 value per year for the yield, since harvesting of crop happens at end of growing season of crop.

I want to build a regression or some other machine learning based model to predict 2015 yields, based on a regression/some other model derived by studying the relation between yields and temperature and precipitation in previous years.

I am familiar with performing machine learning using scikit-learn. However, not sure how to represent this problem. The tricky part here is that temperature and precipitation are daily but yield is just 1 value per year.

How do I approach this?


texts: Examining a DocumentTermMatrix in RTextTools

texts:
I created a DocumentTermMatrix  for text mining using RTextTools. The rows for this DocumentTermMatrix  correspond to dataframe rows and matix columns correspond to words. My question is : How can I get the words (labels vector) for examining the DocumentTermMatrix  ? In other words, How can I get the vector of these 904 words?

require(RTextTools,quietly=TRUE)
data(USCongress)

doc_matrix <- create_matrix(USCongress$text, language="english",   removeNumbers=TRUE, stemWords=TRUE, removeSparseTerms=.998)

dim(USCongress)
[1] 4449    6

dim(doc_matrix)
[1] 4449  904


texts: How can I read in a .csv file with special characters in it in pandas?

texts:
I am trying to read in a .csv file containing some data. I only need to read in specific chunks of rows from the file, such as line 15- line 20, line 45-line 50, and so on. However, the file contains text and copy write information like, such as ©1990-2016 AAR,All rights reserved in several places. Such lines seem to be producing the error ValueError: No columns to parse from file, because when I just copy lines without such information using pd.read_csv(), it works fine. My goal is to automate the process of downloading these files from the web and reading them into pandas to grab chunks of rows and then do some processing with it, so I can't just manually specify the windows of text lacking such characters.

Here is what I tried:pd.read_csv("filename.csv",encoding=utf-8, skiprows = 14) and pd.read_csv("filename.csv",encoding=utf-16, skiprows = 15), after looking at similar answers in stack exchange, but this didn't work. Can anyone give me some guidance on this?


texts: cnn classifier failing on image after contour selecting

texts:
I have a trained mnist-based cnn classifier which is working fine when there is only one number in the image.For more than 1 number in the image I am taking contours and separating the numbers using ROI.
But my model is failing on the separated images
Original image

Contoured image

Here 5 is being classified as 3.
Here is my code:

import cv2
image = cv2.imread("im1.png")
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) # grayscale
ret,thresh1 = cv2.threshold(gray,127,255,cv2.THRESH_BINARY)
gray = cv2.bilateralFilter(thresh1, 11, 17, 17)
kernel = np.ones((5,5), np.uint8)
img_dilation = cv2.dilate(gray, kernel, iterations=1)
img_erosion = cv2.erode(img_dilation, kernel, iterations=1)
#edged = cv2.Canny(gray, 30, 200)
plt.imshow(img_erosion[:,:], 'gray')
im2,contours, hierarchy = cv2.findContours(img_erosion.copy(),         cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
contours = sorted(contours, key = cv2.contourArea, reverse = True)
# get contours
# for each contour found, draw a rectangle around it on original image
roi=[]
i=0
for contour in contours:

    # get rectangle bounding contour

    [x,y,w,h] = cv2.boundingRect(contour)
# discard areas that are too large

    if h>300 and w>300:

        continue
# discard areas that are too small

    if h<40 or w<40:

        continue
# draw rectangle around contour on original image
    i=i+1
    roi.append(image[y:y+h, x:x+w])
#cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,255),2)
# write original image with added contours to disk
im_names=["im1.png","im2.png","im3.png"]
i=0
"""for roi in roi:
    cv2.imwrite(im_names[i], roi)
    i=i+1"""
str1=''
for roi in roi:

   #im = cv2.bitwise_not(image)
    #cv2.imwrite("contoured.png", image)
    im=roi
    from skimage.transform import resize

    def rgb2gray(rgb):
        return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])

    #im = plt.imread("contoured.png")
    im = rgb2gray(im)
    im = resize(im, (28, 28))

    plt.imshow(im[:,:], 'gray')
    #plt.show()

    temp = np.zeros((1,28,28,1))
    temp[0,:,:,0] = im
    num=int(classifier.predict_classes(temp))
    str1+=str((num))
print(str1)

Here classifier is the trained cnn model


texts: Combine multiple features for text classification

texts:
Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. I'm trying to make my own classification algorithm (the text sends a positive/negative message) and I have already preprocessed the text and tested it using different approaches in order to find the one that works the best in my case. The best results were achieved by using Random Forest classifier with BOW - unigrams, but the accuracy can still be further improved (at the moment is somewhere around 0.73). What I want to do next is to create a Bag of emojis, try different sentiment analysis techniques (Vader, SentiStrength), count the number of negative/positive words in each of the  phrases. My question is how can I take into account all these different features? Should I create a table with the results given by all of these features and calculate the mean? For example (0 - classified as positive):

Classifier   Sentiment analysis   Bag of emojis               Pos/Neg words
  0 or 1           0 or 1             0 or 1        if numberPos > numberNeg consider positive

I'm kind of new at this and I was wondering if anyone could please guide me in the right direction?


texts:
I'm interested in interaction effects in Random Forest. I try to implement Friedman H'statistic basically developped for Gradient boosting Friendman, 2001.
But I have no idea how calculate (44) on the real data.

If anybody has some examples or explanations?


texts: How do we find the gradient and the back propagation error if we had a bias which just added a scalar value after the convolution calculation?


texts: Training Restricted Boltzmann Machines (RBMs) using gradient descent

texts:
Hey I am a little new to the whole RBM entropy/energy training thing, having some trouble understanding and trying to figure whether it is worth the effort needed to understand.

Can't RBMs quite easily be trained with a standard gradient descent algorithm? Is the current way just much faster?

Here is some sample Theano code of what I mean:

def rbm(inputs, shape, cost=0, last_cost=1):
    X = T.vector()                     # input
    W = theano.shared(np.zeros(shape)) # weights

    b = theano.shared(np.zeros( shape[1] )) # bias to hidden
    B = theano.shared(np.zeros( shape[0] )) # bias to reconstruction

    H = sigmoid(T.dot(X, W) + b) # hidden activation
    Y = T.dot(H, W.T) + B        # reconstruction

    L = ((X - Y) ** 2).mean()                    # loss function
    G = T.grad(L, W), T.grad(L, b), T.grad(L, B) # gradients

    a = .1                                                  # learning rate
    U = [(W, W - a*G[0]), (b, b - a*G[1]), (B, B - a*G[2])] # shared updates

    train = theano.function([X], L, updates=U) # training function

    while abs(last_cost-cost) > 0.01:
        last_cost, cost = cost, 0
        for input in inputs: cost += train(input)

    return W.get_value(), b.get_value()

And yes this code is very crude.


texts: How to write a code to display output value in every 60 minutes using panda python using csv file

texts:
I have temperature in a csv file. My file is updated with time . I have loaded this data to a pandas DataFrame. What I want to do is predict temperature value in next 60 minutes.

I changed the start time into 0 minutes.

After changed the start time into 0 minutes then I write the code for to display time in every 60 minutes.

date                  time   temp   humidity  wind
2018-10-03 06:00:00    0.0    63       0        0
2018-10-03 07:00:00   60.0    63       0        2
2018-10-03 08:00:00   120.0   104      11       0
2018-10-03 09:00:00   180.0   93       0        50
2018-10-03 10:00:00   240.0   177      0        2
2018-10-03 11:00:00   300.0   133      0        0
2018-10-03 12:00:00   360.0   70       0        0
2018-10-03 13:45:00   465.0   83       0        60
2018-10-03 15:00:00  540.0    127      0        50
2018-10-03 16:00:00  600.0    205      0        0
2018-10-03 17:00:00  660.0    298      0        0
2018-10-03 18:00:00  720.0    234      0        0
2018-10-03 19:00:00  780.0    148      0        20
2018-10-03 20:00:00  840.0    135      0        0
2018-10-03 21:30:00  930.0    100      0        50
2018-10-04 06:00:00    0.0    166      0        0
2018-10-04 07:00:00   60.0     0       0        0

After that I wrote the code for next output temperature value. mean my first output is 63 (temp) at 60 minutes.

time(t)        temp     humidity   wind     output(temp )(t+60)
 0              63        0          0         63
 60             63        0          2         104

Like this it should be run.

What I am trying to do is to display my output (next temperature value ) value in every 60 minutes. inputs will be added same as csv file . But output should be display in every 60 minutes.

I wrote the code for that. But It is not properly displaying my output.
Can anyone help me to solve this problem.

my code:

import datetime
data = pd.read_csv('data6.csv' , sep=',')
print(data)
data['date'] = pd.to_datetime(data['date'] + " " + data['time'], format='%m/%d/%Y %H:%M:%S')
lastday = data.loc[0, 'date']
def convert_time(x):
  global lastday
  if x.date() == lastday.date():
    tm = x - lastday
    return tm.total_seconds()/60
else:
    lastday = x
    return 0

data['time'] = data['date'].apply(convert_time)
data.set_index('date', inplace=True)

data = data.values

X = 1
n_out = 1

x,y=list(),list()
start =0

for _ in range(len(data)):
  in_end = start+X
out_end= in_end + n_out
if out_end < len(data):
    x_input = data[start:in_end]
    x.append(x_input)

    while y.append(data[in_end:out_end,1]):
        if data['time'] <= 60:
            print(y)
            data['time']+=1
start +=1

 x = np.asanyarray(x)
 y = np.asanyarray(y)

my output:

[[ 63.][104.][ 93.][177.][133.][ 70.][ 83.][127.][205.][298.][234.][148.[135.][100.][166.]]

But here the issue came that

After 12:00:00 I measured the value at 13:45. So after 12:00:00 my output is displaying according to the 13:45 (mean 465 minute) value. But I want to get my output in every 60 min , if no value in that time I want to put it as 0 at that time.

What I want to write for output is

time         temp     humidity    wind
360          70        0           0
420          0         0           0
480          0         0           0

inputs should be added like this

 time        temp  humidity  wind
 0 min        63     0         0
 0-60 min     63     0         2
    :
    :

Then here at

 12:00:00 ; 13:00:00

inputs :

 time (360:420 min )  70  0  0
 time (420:480 min)   83  0  0

This is the method that I am trying to do .But it didn't come nicely according to my code.

Can anyone help me to solve this problem?

csv file:
 My csv file.

Using resampling method, output:

Then the output:

Output after resampling:


texts: Why don't we gradually update the activation parameters in RNN from one activation to the next as the network is learning more?

texts:
I'm very new to (unidirectional, vanilla) RNN and sequence modeling in general, and all I understood about the motivation on having the connection between two successive hidden layers/activation is that: this connection is needed to reuse the information learnt from the $t$-th part of the $i$-th sequence, $x_i^{<t>}$ to learn about the $x_i^{<t+1>}$, i.e. the $(t+1)$ -st part of the (same) $i$-th sequence.

Correct me if I'm wrong, but I fail to see the motivation behind using the same activation-to-activation parameter set $\theta_{aa}$ for every connection between two successive hidden states, except of course that: we have less number of parameters to estimate while we minimize the cost. I can't help thinking that $\theta_{aa}$ should be gradually updated with each hidden state, as more information (=part of the same sequence, i.e. words in case of translation) comes in. See the example below.

Let's consider the example of machine translation from English to French for example: (EN) "I am a man" to (FR) "Je suis un homme". Here intuitively, the RNN should try to learn that "am" occurs with certain probability after "I" in English, and correspondingly in French, "suis" occurs with certain probability after "Je"; but given that it's already learnt that, the (conditional?) probability of the occurrence of "un homme" after "je suis" can be more effectively estimated when we know the probability of "a man" occurring after "I am". So intuitively, the RNN should be "better informed" when it knows more parts of the given sequence than lesser part of it, and hence the activation parameters should be gradually updated accordingly.

I must be missing something, but not sure what it is? I've only motivated myself using the machine translation example, but examples from other areas would also be appreciated.


texts: Is there potentially data leakage during imputation for time-varying sensor data?

texts:
I have a time-varying dataset that contains some missing data. I have sensors that continuously monitor some properties at evenly-spaced intervals and I would like to impute the missing values using basic interpolation for both the training and test set. This is a time-series binary classification problem (e.g., based on the entire time-series present, classify as either 1 or 0). I am concerned that taking data from the future to interpolate the missing value is a form of data leakage.
My reason for believing it is not is primarily based on the fact that I am not doing forecasting. I am not trying to predict future values of these sensors, just impute the missing dynamic variables with its most likely values (in fact, based on my domain knowledge and some experiments simple interpolation is very accurate at predicting the true values). If I were trying to predict future sensor values (e.g., forecasting) this would certainly be data leakage, correct?


texts: Capturing movement importance - logistic regression output

texts:
I'm studying some event for a set of objects that can be plotted on a square $[0, 100] ^ 2$. I have used logistic regression to calculate probabilities that event occur for different objects and the output can be plotted as below:

We can observe a high density around point $(100, 50)$ and that basically the farther the less probable the event is. A distance to this point is one of the predictors, but there are also few others. What I'm interested in is to capture a potential movement, i.e. how valuable is to move from point A to point B. But I would like to reward also those movements which are made far from the point $(100, 50)$, not only those near to the point. And for the former the absolute increase in probability will be small even if a movement is pretty significant (long distance), while for the latter we can observe a large increase in probabilities even with a tiny move. So I think what I would need is to somehow nullify/smooth effect of distance from $(100, 50)$, i.e. make it less significant for calculating our movement gain. And I don't really know how to accomplish that. I can't calculate the percentage gain, because this results in a really large gains. I think I need some monotonic transformation that would make a large probabilities lower and low probabilities higher. Any ideas what I could apply here?


texts:
One CLass SVM is a well known machine learning algorithm for novality detection, but it is applicable only for a set of vectors not for time series data.

I read a paper recently of how to use (One-Class SVM) OCSVM with time series data by first converting the time series into a phase space then apply the projection into OCSVM [1]

Is there an implementation or library for such an algorithm to handle OCSVM with time series data?

[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.653.2440&rep=rep1&type=pdf


texts:
I'm pretty new to predictive modeling, but am interested in generating predictions for credit card account spend. These are existing accounts.

The data I have available to me is Card Type (i.e. Platinum, Black, Gold) and spend/transaction data over the last year.

I have a two questions:

How do I decide whether to use linear regression or a decision tree/random forest? Or do I try many techniques?
I expect spend will differ considerably across Card Types; does this suggest I should build separate models by Card Types? If I had a another set of attributes that I thought might segment customers, but wasn't sure, how would I evaluate that?

If there are guides online that answer the above, that'd be much appreciated as well. Haven't found any good ones (or at least I think not)


texts: Sparse matrix in R based on the data frame

texts:
Suppose I have book ratings in the form of data frame (where 0 means no rating):

$\begin{array}{|c|c|c|}
\hline
\textbf{User.ID}& \textbf{ISBN} & \textbf{Book.Rating} \\ \hline
 276725 & 034545104X & 0 \\ \hline
 276726 & 0155061224 & 5 \\ \hline
 276725 & 3257224281 & 7 \\ \hline
 ... & ... & ... \\ \hline
\end{array}$

In what easiest way can I get the form as below (I want to use it to create a realRatingMatrix object) ?

$\begin{array}{|c|c|c|c|c|}
\hline
\ & \textbf{034545104X} & \textbf{0155061224} & \textbf{3257224281} & ...\\ \hline
 \textbf{276725} & . & 3 & 7 & ...\\ \hline
 \textbf{276726} & 5 & 5 & . & ...\\ \hline
 ... & ... & ... & ... & ...\\ \hline
\end{array}$


texts: Binary Search Tree with categorical and numeric values in python

texts:
I coded bst in python and it is working for both categorical and numeric values. Is it draws tree with ASCI value.? So in Isolation Forest algorithm how it is helping to get better accuracy than considering only numeric features and removing categorical features?

Thank you.


texts:
I have an unbalanced dataset X.
I split it between data and labels, then I standardize the data.
Then I use train_test_split to split between train and test data and I output the result.

Now I want to compare to see what I would get if I used smote to upsample the minority class, but I want to keep the comparison meaningful. To do that, I keep the same test data as before, and I add the new synthetic samples only to the train data. How should I handle the standardization?

Should I

Simply assume that since I used points from the already standardised data that the new synthetic samples will also be standardised? Therefore just add the synthetic samples to the train data and be done with it? (Without touching the test data)
Create the synthetic samples from the pre-standardised data, standardise the synthetic samples and add them to the train data? (Without touching the test data)
Create the synthetic samples from the pre-standardised data, add them to the train data, and standardise the whole set? (Without touching the test data)
Or ... ?

I am getting very different results doing these three techniques, what is the best way to get a meaningful result and comparison?


texts: Model for predicting duration based on categorical data

texts:
I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all independant), and some historic data might look like:
JobID   Manager     City        Design          ClientType      TaskDuration
a1      George      Brisbane    BigKahuna       Personal        10
a2      George      Brisbane    SmallKahuna     Business        15
a3      George      Perth       BigKahuna       Investor        7

Thus far, my model has been relatively basic, following these basic steps:

Aggregate the historical data based on each category, calculating the mean, and counting how many times it occurs. From the previous example, the result would be:

Category        Value           Mean    Count
Manager         George          10.66   3
City            Brisbane        12.5    2
City            Perth           7       1
Design          BigKahuna       8.5     2
Design          SmallKahuna     15      1
ClientType      Personal        10      1
ClientType      Business        15      1
ClientType      Investor        7       1

For each job in the system, calculate the job duration based on the above. For example:

JobID   Manager     City        Design          ClientType
b5      George      Brisbane    SmallKahuna     Investor

Category        Value           CalculatedMean      CalculatedCount     Factor (Mean * Count)
Manager         George          10.66               3                   31.98
City            Brisbane        12.5                2                   25
Design          SmallKahuna     15                  1                   15
ClientType      Investor        7                   1                   7

TaskDuration    = SUM(Factor) / SUM(CalculatedCount)
                = 78.98 / 7
                = 11.283
                ~= 11 days

After testing my model on a few hundred finished jobs from the last four months, I calculated average discrepancies ranging from -15% to +25%.
In my actual model I have 15 categories, and am drawing historical data from ~400 jobs.
I think the largest issue (amongst others) is the simplicity of my model. Are their better/well established methods for calculating a value based on categorical data? And if not, how can I improve my predictions?
Related question here.


texts: adding summary to tensorflow model error

texts:
I have a CNN model for classifying lung CT images, the code is written in tensorflow, I added some tensorflow summaries to my code to show graph, scalar, histogram, ... of my tensorflow model in tensorboard, in last step when I want to add_summary to the file writer it gives me an error, here is my code:

def train_CNN(input):
    train_predict = CNN_Model(x_img)
    with tf.name_scope("cross_entropy"):
        cost = tf.nn.softmax_cross_entropy_with_logits_v2(logits=train_predict, labels=y_label, name='cross_entropy')
        cost = tf.reduce_mean(cost, name='reduce_mean')
        tf.summary.scalar("cost", cost)
    with tf.name_scope("optimization"):
        optimizer = tf.train.AdamOptimizer(learning_rate, name='AdamOptimizer').minimize(cost)

    with tf.name_scope("accuracy"):
        correct_predict = tf.equal(tf.argmax(train_predict, 1), tf.argmax(y_label, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_predict, tf.float32))
        tf.summary.scalar("accuracy", accuracy)

    #tf.summary.image("input", x_img, 5)

    sess.run(tf.global_variables_initializer())
    log_train_path = 'C:/temp/tensorflow_logs' + '/train_{}'.format(datetime.now().strftime("%Y-%m-%d-%H%M%S"))
    summary_writer = tf.summary.FileWriter(log_train_path)
    summary_writer.add_graph(sess.graph)
    merged_summary = tf.summary.merge_all()

    all_time = 0

    for epoch in range(num_epochs):
        start_time = time.time()
        ep_loss = 0
        for data in train_data:
            X = data[0]
            Y = data[1]
            summary, _, c = sess.run([merged_summary, optimizer, cost], feed_dict={x_img: X, y_label: Y})
            ep_loss += c
            summary_writer.add_summary(summary, epoch)
        end_time = time.time()
        all_time += int(end_time-start_time)
        print('Epoch', epoch+1, 'completed out of',num_epochs,'loss:',ep_loss, 'time usage: '+str(int(end_time-start_time))+' seconds')

        print('Accuracy of this epoch:',accuracy.eval({x_img:[i[0] for i in val_data], y_label:[i[1] for i in val_data]}))

    print('Finall Accuracy:',accuracy.eval({x_img:[i[0] for i in val_data], y_label:[i[1] for i in val_data]}), 'time usage: '+str(all_time)+' seconds')

after running the model it give me error, can anyone tell me how to solve it?

here is the error:

InvalidArgumentError: Expected dimension in the range [-1, 1), but got 1
     [[Node: accuracy/ArgMax_1 = ArgMax[T=DT_FLOAT, Tidx=DT_INT32, output_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_y_label_0_1, accuracy/ArgMax_1/dimension)]]


texts: Creating radial basis for linear regression Python

texts:
I'm trying to do time series forecasting with linear regression like it's done in this video: Radial basis forecasting starting from 5:50.
I understand the basic idea of basis, but I don't think I understood the usage of it in time series data correctly. I have a Pandas dataframe with daily timestamps and target variables. I tried writing radial basis function
def radial_basis(x, month):
    alpha = 0.5
    coef = -1/(2*alpha)
    return np.exp(coef*(x-month)**2)

and calculating basis for every month (x is row number). This didn't work.
Any tips on how I should try to do this?


texts: TensorFlow: trainable_variables() is empty

texts:
I want to retrieve the list of trainable variables/weights in my model (wrapped in a tf.Estimator). However, tf.trainable_variables always returns an empty list, what am I doing wrong?

import numpy as np
import pandas as pd
import sys
import globals
import pkg_resources
import numpy as np

import tensorflow as tf

def cnn_model_fn(features, labels, mode):

    input_layer = tf.reshape(features["x"], [-1, 51, 13])
    input_layer = tf.cast(input_layer, tf.float32)

    # Convolutional Layer #1
    conv1 = tf.layers.conv1d(inputs=input_layer, filters=32, kernel_size=5, padding="same", activation=tf.nn.relu)

    # Pooling Layer #1
    pool1 = tf.layers.max_pooling1d(inputs=conv1, pool_size=2, strides=2)

    # Convolutional Layer #2 and Pooling Layer #2
    conv2 = tf.layers.conv1d(inputs=pool1, filters=64, kernel_size=5, padding="same", activation=tf.nn.relu)

    # Pooling layer #2
    pool2 = tf.layers.max_pooling1d(inputs=conv2, pool_size=2, strides=2)

    # flatten the feature map
    pool2_flat = tf.reshape(pool2, [-1, 12 * 64])

    # Dense Layer
    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)

    # Logits Layer
    logits = tf.layers.dense(inputs=dropout, units=5)

    predictions = {
        # Generate predictions (for PREDICT and EVAL mode)
        "classes": tf.argmax(input=logits, axis=1),
        # Add `softmax_tensor` to the graph. It is used for PREDICT and by the `logging_hook`.
        "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
    }

    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

    # Calculate Loss (for both TRAIN and EVAL modes)
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)

    # Configure the Training Op (for TRAIN mode)
    if mode == tf.estimator.ModeKeys.TRAIN:
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)

    # Add evaluation metrics (for EVAL mode)
    eval_metric_ops = {"accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])}

    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)

def main(unused_argv):

    total = pd.read_feather('testfile.feather')
    labels = total['labels']
    features = total.iloc[:, 16:679]

    mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, model_dir="/tmp/gait_convnet_model")

    # Log the values in the "Softmax" tensor with label "probabilities"
    tensors_to_log = {"probabilities": "softmax_tensor"}
    logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=50)

    # Train the model
    train_input_fn = tf.estimator.inputs.numpy_input_fn(x={"x": np.array(features)}, y=np.array(labels), batch_size=100, num_epochs=None, shuffle=True)
    mnist_classifier.train(input_fn=train_input_fn, steps=1, hooks=[logging_hook])

    temp_list = tf.trainable_variables()
    print(temp_list)

if __name__ == '__main__':
    tf.app.run()


texts:
I was following Tensorflow's own time series/LSTM tutorial, and there's something I don't quite understand about the whole process around Backpropagation Through Time (BPTT).
The resources I've found explain how prediction for a time in a future with an offset work (the "offset" in the tutorial, and the t+3rd time instance in the Wikipedia article)
What I don't understand is how does this generalize to where the input width is more than 1?
So in the Wikipedia article we see the case for inputWidth=1, outputWidth=1 and offset=3.
I think I understand if outputWidth>1 then you'd just start using not only the last, but also previous cell predictions as output. I also understand if offset>1 the you just unroll the network into offset number of steps (as shown on Wikipedia).
But what if inputWidth>1? Let's say inputWidth=k (k>1)!
In this case, do you unroll the same RNN multiple times in parallel, and somehow average the weights? Do you unroll the same RNN k times sequentially? What does that look like?


texts: Training with multi-series of different length with stateful LSTM

texts:
I'm training a stateful LSTM.  My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but City B I might only have 500. I have 20 features for each day.

Say I set the sequence size to 200; for City A I'd have a batch of 20 complete sequences, but for City B I'd only have 2.

With stateful LSTM I need to specify the batch_input_shape in the form [batch size, sequence size, features], which I think will cause me some problems because my batch size changes for each city I process.

As I see it, the only option available to me is to define the batch size for batch_input_shape as 1, and then just effectively issue a single 200-length sequence for each iteration. So batch_input_shape would be [1, 200, 20]

Obviously this isn't ideal because it's such a small amount of data that's being presented to the NN for each iteration (it's certainly mini-batch!) so it will be quite slow to train the network.

But am I right that due to the differing sizes of the data, I don't really have another option?  Besides slow train time, are there any inherent problems training with batch size of 1?


texts: Why does joint embedding of word and images work?

texts:
I often see some papers where the authors do point-wise multiplication of word and image embedding (e.g the image below).

Why does this  implementation works? I do not understand.


texts: Is it possible that I conduct a supervised machine learning like random forest after creating a target variable through clustering?

texts: I create 4 clusters by using k-mean cluster analysis. After that, I try to classify this target variable with random forest in order to confirm if the clusters are properly grouped. Is it possible?


texts: Best way to store financial data together with news

texts:
My question is if there is a DBMS that allows storing financial data (i.e. timestamp+value) together with small text fragments.

These small fragments will not be very long, but it might help an algorithm to predict changes in the future. In summary, I wonder if you can suggest me what it is the best way to massively store timestamps+numeric value+(optional)text fragment.

Thank you in advance.


texts: Reinforcement learning - generating a matrix of continuous values with varying size for test data generation

texts:
Currently, I am using RL A3C algorithm for test data generation, where for a set of 30 functions written in C (mostly basic algorithms like Prime number checks, triangle validity, etc.) I try to generate a set of test data to have 100% coverage.

Now for this, I need to generate a matrix of these test data. To explain, lets say that we have 3 execution paths for the function and the function takes two parameters of integer. This means, that to have 100% code coverage, I need 3 function calls with different function parameters, hence a matrix of test data 3x2. But in reality, I can have a matrix of 12x5 and more, as the size of the matrix is varying.

What I have done now is, that with the help of this implementation: https://github.com/stefanbo92/A3C-Continuous of A3C I can generate a continuous value for one of the parameters of the function calls. I iterate through the functions, process the ASTs using a convolutional layer and generate a value for one cell in the matrix using a LSTM cell. Then I create the tests using this matrix and retrieve the coverage value. If the coverage is 90%+, I return a reward of 1, otherwise I return 0. I have also tried returning the coverage value as a reward, but with little to no changes in results.

Right now, I try to change the value separately for one cell in the matrix at a time and then generate a test and then get the coverage value. I think that this doesn't allow the agent to efficiently train on the data, as the action is only associated to the one value of the matrix, not the matrix in general (or the current row of the matrix). The current results only confirm this, as after around 1200 training episodes the coverage and reward stagnated and even dropped from the initial value.

As such, I would like my agent to not generate a continuous integer value for just one cell of the matrix, but ideally the entire matrix as one action.

I have no clue how to do this for varying matrix size, which I require, and I thought of using some sort of encoding/decoding, but have no clue as to have it could be achieved.

Any kind of advice is appreciated as to how to deal with this issue.

Thanks in advance.


texts: What does a deep learning model look like in file?

texts: So after you train your model with millions of rows of data, what is the model being saved in disk look like? Is it file with millions of weights and biases that would do the best predictions?


texts: Possible?: Split pdf file on where object detection algorithm finds custom object

texts:
I have scanned documents in large pdf files consisting of many individual documents. Each document begins with a exhibit number sticker, much like this one: example. The files are scanned in greyscale.

I am looking to detect this numbered sticker object in the files. On top of that, my current understanding is that I would need to crop this sticker if I want to run another NN on it to recognize the handwritten digits on the sticker.
I wish to split the files on the pages where an object is detected and rename those individual files based on the handwritten digit on the sticker.

So far I know I must split the pdf pages into individual images with pdf2image like in this example, and if I am using a local algorithm like with opencv4, I should label the training data beforehand with something like labelimg. After reading, it looks like tensorflow and keras will suit my project fine.

My questions are as follows:

Assuming the image detection model generates a valid bounding box, how can I crop the bounding box?
What NN would best suit each need (detecting single object + handwriting OCR)
What output file should I expect from the model, xml? Should I expect any obstacles in using the output file as a basis to process the corresponding pdf files with something like PyPDF2?

The short of it is I'd want to do what's described in this article and edit my pdf file based on the results.


texts: How do I stack copies of a 1D tensor in keras?

texts:
I have a tensor x of shape (5, 1). I would like to stack 10 copies of it so that I get a tensor of shape (5, 10). In a sort, it would look like [x|..|x]. How to do it the keras way? I have tried the Concatenate layer, but I got an error.
Help is welcome!


texts: What kind of RAM to choose for data analysis?

texts:
I'm planning to upgrade my computer and also want to buy some new DDR3 RAM modules. Making some research I realized that there are 2 main characteristics of RAMs (beside capacity): frequency and latency.

I want to use my computer for data analysis as well (like Kaggle competitions). Usually I work in R and sometimes in Python.

I understand that there is a trade-off between latency and frequency (that is also reflected in the price sometimes). My question is which one is more important for data analysis purposes? How can it effect real world performance? How much difference should I expect between choosing a model with overall better and lower parameter values?

UPDATE: I don't ask which RAM is better in general. I need answer from users who know how memory management happens in R (or Python). How are the different ML algorithms managed in memory and which RAM characteristics are preferred to support it. I would also appreciate personal experiences on this field.


texts: operating on a dataset with 125,497,040 records

texts:
I'm trying to run add_datepart() which converts a df column from a datetime64 to many columns in place

Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear','Ismonthend', 'Ismonthstart', 'Isquarterend', 'Isquarterstart', 'Isyearend', 'Isyearstart' . etc

I'm using is Grocery Sales dataset. Total dates i believe in that are 125497040. What should I do to run this operation?

Every time I run this piece of code. The kernel dies(out of memory which is 17.2 GB RAM)

So I tried breaking down this data frame in smaller parts and then running add_datepart but still the same result

I wrote this code

def add_datepart_large(temp_df, size):
    list_df = [temp_df[i:i+size] for i in range(0,temp_df.shape[0],size)]
    for i in range(len(list_df)): add_datepart(list_df[i], 'date')
    return pd.concat(list_df)

final_df = add_datepart_large(df_all, 100000)

If after running this code, the kernel dies. What's wrong?


texts: How to create an ensemble in tensorflow using tf.estimator?

texts:
I have created a neural network using tensorflow's estimator API:

classifier = tf.estimator.DNNRegressor(feature_columns=feature_columns,hidden_units=[10,20,10],n_classes = 3)
classifier.train(input_fn=lambda: input_fn(X_train,y_train),steps = 1000).

I would like to create an ensemble of these neural networks, since there is some randomness in the output of an individual network, even if the RMSE is acceptable.

So far the only way I have found of doing so is to use a loop which trains the estimator and then generates predictions over a number of iterations, storing the output in an array and then averaging the values to get my ensemble prediction.

But there must be a better way to do than using tf.estimator.

How?


texts: Preparing text for modeling in dialogue structure

texts:
I'm working on implementing the DialogueGCN code from this paper. Its a model that classifies the 'emotion' from utterances of text within a conversation. As this model takes into account speaker context, the structure of the data is very important. It looks something like this:

c = conversation (each conversation is independent of the other
u  = utterances (ex.
                 u1 = "Hey how are you?"
                 u2 = "good, thanks, and you?"
                 u3 = "confused, my dog is cooking pasta")
Utterances are of variable length and conversations have variable amounts of utterances.
Each utterance has an 'emotion' label.

                               |    corpus    |
                              /        |       \
                             c1       c2       c3
                           / | \     / | \    / | \
                          u1 u2 u3 u1 u2 u3  u1 u2 u3

The paper uses a CNN to generate features from the text data. The CNN model comes from Kim (2014). There are several implementations of this paper. I am using this Keras implementation. The model architecture essentially boils down to this:

### the model input is 1 utterance

z = Embedding(len(vocabulary_inv), embedding_dim, input_length=sequence_length, name="embedding")(model_input)
z = Dropout(dropout_prob[0])(z)

# Convolutional block
conv_blocks = []
filter_sizes = [2,4,5]
for sz in filter_sizes:
    conv = Convolution1D(filters=num_filters,
                         kernel_size=sz,
                         padding="valid",
                         activation="relu",
                         strides=1)(z)
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
    conv_blocks.append(conv)
z = Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]

z = Dropout(dropout_prob[1])(z)
z = Dense(hidden_dims, activation="relu")(z)

### the output of the dense layer is a tensor for the utterance

IIUC, each utterance of each conversation is individually input into this CNN and 1 tensor per utterance is the output. Then, all utterances per conversation are concatenated. After processing the whole corpus, the result is a list of concatenated tensors. This is fed into the GCN model.

If you are still following me, my question is this:

I'm trying to use the GCN model to predict on new set of dialogues. The dialogues are structured in the same format, but are still in text (string) form. I need to vectorize them to feed them into the CNN feature generation model. The DialogueGCN paper doesn't say how this was accomplished. What is the optimal way to vectorize strings when they are structured in the dialogue format where words in each utterance are dependent within a conversation, but words across conversations are independent?

In other words, if I am converting each unique word to a unique integer, should I process the entire corpus as one (using the same word:integer mapping for all conversations), or should I process each conversation separately (unique word:int mapping for each convo) because they independent of one another. For instance, the word 'baseball' could mean something different in conversation 1 and conversation 2, depending on context (previous and future utterances).


texts:
I have logs of the form (e.g. from a gym login).. the representational case is so:
UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical
Ava, 5jan 12pm, 10 mins, 20 mins,
Bob, 5jan 2pm, 30 min, 20 mins,
Cecila, 6jan 10am, 40min, 0 mins
...

Now I've converted the above time column to HourOfDay and day of month to get:
UserName, DOM, HOD, #weights, #elliptical
Ava, 5, 12, 10, 20
Bob, 5, 14, 30, 20
Cecilia, 6, 10, 40, 0
..

I treat the first 3 columns as categorical data and the last two as numerical, and I run K-Prototypes with N=2 (anomalous or non-anomalous). The final predictions I get can be filtered on each user to find anomalies specific to the username. The anomalous cluster is the one with lesser elements.
However, for some of the users, the cluster partitions on the Login time (HOD/DOM).. E.g. everything before 12am is one cluster and everything after 12am is another one. That doesn't convey any information.
What is the best way to handle these scenarios?
Is there a better way to do anomaly prediction on these kinds of dataset?
Update:
Type of anomalies I'm looking for:

Ava did 20 mins in elliptical, that she never used before. This individually can be done simply by using some form of outlier analysis, or K-means (on dataset filtered by 'Ava')
Ava did elliptical on Monday morning (Samething as above but filtered on Ava & time of the day).

Individually I can create models for each dataset with reasonable success, but how do I create one model that handles both of them.
If I use an actual clustering algorithm like DBSCAN/HDBSCAN, how do I not have it partition on the time? (or some other categorical variable)


texts:
ML and Data Science world.

I am a newbie to CNNs, but do possess a basic understanding of ML and Neural Networks.

I wanted to create my own CNN that works on the Cats and Dogs Dataset. I preprocessed the data and built my network, but when I fit the model with the data, I am not able to get more than 55% accuracy, which means the model isn't learning anything.

Can anybody explain what I am doing wrong here?

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(3,3),padding='SAME', input_shape=X[0].shape))

model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))

model.add(Conv2D(filters=64, kernel_size=(3,3), padding='SAME'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))
model.add(Dropout(rate=0.4))

model.add(Conv2D(filters=128, kernel_size=(3,3), padding='SAME'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))
model.add(Dropout(rate=0.35))

#model.add(Conv2D(filters=64, kernel_size=(3,3), padding='SAME'))
#model.add(Activation('relu'))
#model.add(MaxPooling2D(pool_size=(2,2), dim_ordering='th'))

model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))

model.add(Dense(2))
model.add(Activation('softmax'))

And this is the optimizer part:

opt = keras.optimizers.SGD(lr=0.0001, decay=0.0)
model.compile(optimizer=opt, loss='binary_crossentropy', metrics['accuracy'])
print(model.summary())

model.fit(X, np.array(Y), validation_data=(test_x, np.array(test_y)), epochs=30, verbose=2)

I've been stuck on this for the past 2 days.
Any and all help is appreciated.

Thanks.


texts:
Knowing the nature of my time series problem, I am using a stateful LSTM to forecast one step ahead.
My question is quite straightforward.
Do I need to explicitly save and pass the hidden cell in every prediction step in deployment, or Keras model already saves, updates and uses the states between predictions, and what I will do is simply predict as new data arrives?
Thanks in advance.


texts: Is there any APIs for crawling abstract of paper?

texts:
If I have a very long list of paper names, how could I get abstract of these papers from internet or any database?

The paper names are like "Assessment of Utility in Web Mining for the Domain of Public Health".

Does any one know any API that can give me a solution? I tried to crawl google scholar, however, google blocked my crawler.


texts:
I want to create a neural net that can obtain some specific words from a pdf document into JSON or XML. For example let's assume that I have a pdf containing some information about countries and i want to recuperate the countries name and population to obtain  something like this :

<countries>
  <country>
    <name>
      France
    </name
    <population>
      70m
    </population
  </country>
.
.
.
</countries>

Should I build a neural net and train it myself? If so can you give a good  tutorial to follow please, or is there an already trained one that I can use?


texts:
The common way of learning word embeddings is based on BOW, and Skip-gram models.

Is it possible to train a RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings? Basically, we train a network with positive and negative samples, and back-propagate into word vectors. What are the drawbacks of this technique?

Any reference to similar works is highly appreciated.


texts: How to correct mislabeled data in dataset?

texts:
I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly labeled.

Is there any way of finding out which elements are wrong, so I would be able to discard them?


texts: Is it a good idea to train with a feature which value will be fixed in future predictions?

texts:
I am facing a regression problem and I have one feature that has some relevant correlation with the output. The value of this feature will be fixed in ALL the predictions I will use this model for.

Should I keep it in my model or not?

Thanks


texts: Reading the literature around deep learning adversarial attacks it appears to be wholly concentrated on attacks of image classification models. Are there papers that describe attacks on non image data ? Searching archive for deep learning adversarial attacks appears to contain results that are just related to image classification field.


texts: the size of training data set in the context of computer vision

texts: Generally speaking, for training a machine learning model, the size of training data set should be bigger than the number of predictors. For a neural network, or even a deep learning model, the number of parameters are usually tens of thousands or even millions. It seems that in practice, the number of training data set, i.e., the number of images, is usually less than the number of parameters. How to explain this? I know, we can claim that the pre-trained model may remove the requirement of having that many images. Is this the only reason, or we should use number of pixels multiplied by the number of images to measure the size of training data set.


texts:
I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear and another to appear in the data as a whole.  Occasionally, a widget will just disappear and so the dataset is incomplete, invalidating the whole dataset for that day.

What I am looking for is a method of comparing the current set of widgets with another set of widgets to detect if any widgets are missing.  I am not trying to create the values, just identify that they are missing.  I could do time-series, but that feels like overkill on so many widgets and there are multiple attributes on which data might be missing.  I was hoping for something more set based that might account for the regular changes in widgets but detecting the unusual dropouts.  I am sure I just need to adjust the way I am thinking about the problem.

Any ideas would be much appreciated?


texts: How to propagate error back to previous layer in CNN?

texts:
I have a convolutional layer (link) with an input 5x5x2 (width, height, depth):

The layer has 3 filters with dimensions 3x3x2, it produces an output with dimensions 3x3x3.

I have completed the forward pass:

In a backward pass I calculated gradients of weights $dL/dw$, but I am not sure how to calculate $dL/dh$ (gradients for the previous layer).

First or all, is my calculation of $dL/dw$ valid ? (please refer to external spread sheet linked to the question) And most importantly, how do I propagate deltas to the previous layer ?


texts:
I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one author for a document). I wanted to find out the topics interested by each author by looking at documents they write. I'm currently looking an LDA variation (labeled-LDA proposed by D Ramaga: https://www.aclweb.org/anthology/D/D09/D09-1026.pdf .). I'm using all the documents in my dataset to train a model and using the model to predict the labels of an author(here I use the aggregation of all documents belonging to an author to represent him, treat authors like documents).

I'm using python and I found some implementation available on Github:
https://github.com/JoeZJH/Labeled-LDA-Python
https://github.com/shuyo/iir/blob/master/lda/llda.py

I tried the first one and tested it on a small data. It worked but when it came to a big data, the memory required grew a lot (For initialisation, this algorithm generalizes a zero matrix of size #labels times #terms/#vocabulary). I think I need to find a way to reduce the dimension of my corpus. I thought about directly reducing the size of my vocabulary, but it will reduce the length of my texts. And LDA does not work for short text. I’m afraid that if I do this, my texts will not be enough and some information will be lost in the process.

So here are my questions:

Is my method theoretically correct? (I will appreciate it if someone can propose other models suitable for this task)
If there is any method to reduce the dimension while keeping as much information as possible. For instance, is TF-IDF an option for the reduction?

I do appreciate it if you can give some advice on those two questions.


texts:
Tl;dr: I would like to read video streams coming from all the participants from this https://tfvideo-5708-dev.twil.io/video.html in python
Details: I have a live tutoring website that uses Twilio Video API to carry out the video class. I want to run the object detection and other computer vision models on the input stream. I am stuck at how to get this input video stream on which i can run ml models

using Python. I can read by open cv2.VideoCapture(....) method and then run my inference using some pre-trained model or using inbuilt computer vision models from open-cv. I want to understand what should be the parameter under brackets of cv2.VideoCapture().
using Tensorflow.js API - by integrating the tf.js code in the website's Javascript and running the inference on that stream. Is it feasible?

I am having difficulty fetching that stream data to use for my Ml models
To test, I have created a web chat application that leverages Twilio Video . here is the link to the original page and here is the github repo. Following is the link that you can join for video web chat here if you open this link, you can join the room (I will have to deploy first before you can join) and then my requirement is to get access to the streams of all the participants that join.
Here are some links that I have gone through:
https://github.com/twilio/twilio-video.js/issues/333
https://www.twilio.com/blog/build-video-chat-application-python-javascript-twilio-programmable-video
https://www.infoq.com/articles/video-stream-analytics-opencv/
Can anyone help me with this? Is there any other way?
If the question is not clear, let me know what else is needed. I can give more details.


texts: Intuition Behind Restricted Boltzmann Machine (RBM)

texts:
I went through Geoff Hinton's Neural Networks course on Coursera and also through introduction to restricted boltzmann machines, still I didn't understand the intuition behind RBMs.

Why do we need to compute energy in this machine? And what is the use of the probability in this machine? I also saw this video. In the video, he just wrote the probability and energy equations before the computation steps and didn't appear to use it anywhere.

Adding to the above, I am not sure what the likelihood function is for?


texts: I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?


texts: Pytorch: How to create an update rule the doesn't come from derivatives?

texts:
I want to implement the following algorithm, taken from this book, section 13.6:

Here, the neural networks' outputs are $V(S, w)$ and $\pi(A|S,\theta)$, parameterized by $w$ and $\theta$ respectively.

For this question, I will refer only to $V(S,w)$ for simplicity.

The update rules for w and for theta, the two networks' parameters, depend on Zw and on Ztheta, which in turn depend recursively on themselves, as well as on the networks' outputs' derivatives.

Simplifying the equations to create a loss term [that pytorch can handle] that could be derived using loss.backward() would yield

$w \leftarrow w + \alpha^w \delta z^w_{new}$

$w \leftarrow w + \alpha^w \delta [\gamma \lambda^w z^w_{old} + I \nabla_wV(S,w)]$

$w \leftarrow w + \alpha^w \delta \nabla_w[\gamma \lambda^w z^w_{old} w + I V(S,w)]$

thus the loss would be

$Loss = \delta[\gamma \lambda z^w_{old}w + I V(S,w)]$

A few things I am struggling with:

How to obtain $w$ for the loss term, Meaning, accessing they network's weights directly somehow?
How to write an iterative update rule without the mathematical manipulations?
This is prefered for cases when such manipulations are difficult, or impossible, or for cases when the update rule is given in the above form (like in the book), and the hassle of going to the loss form could maybe be avoided. This would also prevent mistakes, by just taking the final form from the (tested) book.


texts:
Hello PyTorch experts:
Sentences and documents both can be variable length.
Lets say, we have following 2 docs:
doc1=[torch.tensor([1,2,3,4]),torch.tensor([4,5,6]),torch.tensor([7,5])]
doc2=[torch.tensor([1,2,3]),torch.tensor([4,5])]

(Here, each tensor is a sentence and each number in the tensor is an index of embedding matrix)
doc1=pad_sequence(doc1,batch_first=True)
doc2=pad_sequence(doc2,batch_first=True)

batch=pad_sequence([doc1,doc2],batch_first=True)

This will throw an error because the length of the longest sentence in the first document is greater than the length of the longest sentence in the second sentence.
So, we need to pad each sentence to the longest sentence of the batch.
One solution would be, find the length of the longest sentence in each batch and pad each sentence to that fixed length. But the pad_sequence function of PyTorch does not support that.
Am I missing something? Is there any other PyTorch way to do this?


texts: is it possible to write machine learning algorithm in octave or python for application created in android

texts:
We planned to create an Android app which predicts the present student in class through face detection by camera in the class, my job is to write algorithm for face detection and the problem is I can do it for octave and Python but never did for Java or other programming language.

Can I use the octave or Python script for Android app, is there any possible ways to connect the dots between Android app and Python/octave script?


texts:
The question is very simple, yet I can't find a quick confirmation on the web. It might seem obvious - by design, will Dropout always result in stochastic-looking gradient descent? (SGD)

I've built a system which converges nicely with momentum at 0 and learning rate 0.01, even with 100 layers stacked. With dropout, the error decreases, but jumps up and down due to dropout knocking-out certain neurons?

Is it usual to see fluctuations of error during backprop with dropout, similar to SGD?

Would that  imply I can train on huge batches instead of minibatches, without the fear of overfitting by default?
For example, using Resilient Propagation (rProp) - in my test-example the LSTM trains in just 40 iterations using rProp, instead of 1000 using SGD


texts: What are the possible ways to detect skin while classifying diseases?

texts:
I am working on a skin disease classification problem where I have successfully created a classifier ( TensorFlow + Keras ) which can classify images of two skin diseases.

The sample image needs to be classified in this manner :

Whether the sample is an image of the skin.
Does the skin have any two of the diseases ( Melanoma or Psoriasis )
If a disease is found, to which class does it belong (CLASS1: Melanoma or CLASS2: Psoriasis

  How can create Classifiers which could carry out the following tasks?
  Do I need the image localization or CNNs or something like YoLo?
  What steps should I implement?

I have created a classifier ( with an accuracy of 96% ) which can classify an image of the two diseases efficiently. But it can't detect the presence of the disease ( Step 2 in the above task list ).


texts:
Can anybody explain Apache Flume for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

Please

As you would explain for a non-technical person


texts:
What algorithms should I research to sort pie charts by similarity?

The real-world problem I'm working with is a collection of race data for a few dozen cities — each race listed as percent of the city's population.

I'm interested in selecting a single city and finding other cities with similar racial proportions.


texts:
The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. I am confused about the first step for doing this because there are many datasets (4). If the first step is to preprocess the data, which dataset will be preprocessed first? I am using the R language.

Thank you.


texts: What should be the best CNN model for Feature extraction of images for Image-Image search engine using LSH?

texts:
I have images something like the below:

And like this:

I have a huge data around 20M or so am I want to apply de-duplication and improve the search for this Image data. What I have in mind is to use Train and Extract Features from a CNN based Network and apply LSH: Locality Sensitive Hashing or Approximate Nearest Neighbour: ANN. I want to know that which model should I use to extract the features. I am thinking about training Siamese network (Maybe ResNet based architecture, if possible) or any of the pre trained Imagenet classifiers like pre trained VGG, ResNet, MobileNet etc. Which one should I use because I think I have a different use case because it is more like an OCR kind of approach rather than Classifier. For training purposes, I think I can have around 100K-1M images.
Also, can I use U-Net or AutoEncoders?


texts:
Not sure, if this is the right forum so redirect me if it wrong.

I have started on an NLP problem in kaggle. There i have word embeddings from google news, wiki, glove
in a zipped folder. I want to use one  of them, say glove, without unzipping the zipped file.
This is beecause if I try to unzip, it exceeds the 4.9 Gb space limitation and throws error and stops.

Any idea on how to deal with it?

I found out the way around for it.

   import io
import zipfile

dim=300
embeddings1_index={}

with zipfile.ZipFile("../input/quora-insincere-questions-classification/embeddings.zip") as zf:
with io.TextIOWrapper(zf.open("glove.840B.300d/glove.840B.300d.txt"), encoding="utf-8") as f:
for line in tqdm(f):
values=line.split(' ') # ".split(' ')" only for glove-840b-300d; for all other files, ".split()" works
word=values[0]
vectors=np.asarray(values[1:],'float32')
embeddings1_index[word]=vectors


texts:
I'm working on a consulting project for a tech client, and caught myself scratching my head about what the best way to present advanced analytics workflow is. What will be shown to the panel will focus on results, but in this particular case it is warranted to show a visual for what the process behind the scenes is.

Specifically, I need to show the following:

1) Some raw data file is used as input to a cleaning script which performs imputation and adds/drops variables based on some criteria. Some are added with joins on external files.
2) The cleaned data is passed to a script which creates variable subsets dependent on remaining granularity and subsets of interest. This outputs many cleaned data sets.
3) The new data sets are passed to an external computing cluster, which builds a model on each data set, and outputs a table with performance metrics.
4) Top performing models are re-run locally and score observations.

Currently, I am leaning toward UML, as the panel is likely to be familiar with it and appreciate its use. Still, I would like to know if there is any convenient workflow diagramming standard which exists in data science.

Some perspective:

Information systems have historically been well established in diagramming both on a conceptual and hardware level with ERD's and data flow diagrams of all kinds. Software development has also had a heavy emphasis put on this through flowcharts and UML diagrams. Business analytics relies on BPMN to show business processes.

However, when trying to find a diagramming standard in the realm of data science, all I could find were references to the many beautiful and clear visualizations of the findings we make ourselves. We certainly don't shy away from standard methods in visualization (and often for good reason). So, is there such a method for documenting our own workflow, as there is in other fields?


texts:
I would greatly appreciate if you could let me know how to fix the following issue:

I used sklearn.metrics.fowlkes_mallows_score to compute G-mean score for my binary classification problem, but it produces nan:

AUC: 0.960674
Accuracy: 0.949137
Cohen Kappa: 0.484456
F_measure: 0.508772
G-mean: 0.833217
G-mean1: nan
Matthews: CorrCoef 0.510088
Precision: 0.391892
Recall: 0.725

However, it should be: 0.533031

Thanks in advance.


texts: Why does my Keras Conv Net only return 1?

texts:
model = Sequential()
   model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(img_h_w,img_h_w,1)))
   model.add(Conv2D(32, (3, 3), activation='relu'))
   model.add(MaxPooling2D(pool_size=(2,2)))
   model.add(Dropout(0.25))
   model.add(Flatten())
   model.add(Dense(128, activation='relu'))
   model.add(Dropout(0.5))
   model.add(Dense(1, activation='softmax'))
   model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01),metrics=['accuracy'])
   model.fit(x_train, y_train, batch_size=100, epochs=3, verbose=1)
   score = model.evaluate(x_test, y_test, verbose=0)

Running this returns

2500/2500 [==============================] - 5s 2ms/step - loss: 7.5694 - acc: 0.5252
Epoch 2/3
2500/2500 [==============================] - 5s 2ms/step - loss: 7.5694 - acc: 0.5252
Epoch 3/3
2500/2500 [==============================] - 6s 2ms/step - loss: 7.5694 - acc: 0.5252

Which shows the accuracy is at 50%, and when I look at the predictions made, I see it only outputs 1. It is meant to classify images into 0 or 1.


texts:
I read a few tutorials on neural network backpropagation and decided to implement one from scratch. I tried to find this single error for the past few days I have in my code with no success.

I followed this tutorial  in hopes of being able to implement a sine function approximator. This is a simple network: 1 input neuron, 10 hidden neurons and 1 output neuron. The activation function is sigmoid in the second layer. The exact same model easily works in Tensorflow.

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)

def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))

x_data = np.random.rand(500) * 15.0
y_data = [sin(x) for x in x_data]

ETA = .01

layer1 = 0
layer1_weights = np.random.rand(10) * 2. - 1.
layer2 = np.zeros(10)
layer2_weights = np.random.rand(10) * 2. - 1.
layer3 = 0

for loop_iter in range(500000):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    layer2 = layer1_weights * layer1

    # layer 3
    layer3 = sum(sigmoid(layer2) * layer2_weights)

    # error
    error = .5 * (layer3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2
    error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)
    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1
    error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights

    if loop_iter % 10000 == 0:
        print(error)

The unexpected behavior is simply that the network doesn't converge. Please, review my error_wrt_... derivatives. The problem should be there.

Here's the Tensorflow code it works flawlessly with:

x_data = np.array(np.random.rand(500)).reshape(500, 1)
y_data = np.array([sin(x) for x in x_data]).reshape(500, 1)

x = tf.placeholder(tf.float32, shape=[None, 1])
y_true = tf.placeholder(tf.float32, shape=[None, 1])
W = tf.Variable(tf.random_uniform([1, 10], -1.0, 1.0))
hidden1 = tf.nn.sigmoid(tf.matmul(x, W))
W_hidden = tf.Variable(tf.random_uniform([10, 1], -1.0, 1.0))
output = tf.matmul(hidden1, W_hidden)

loss = tf.square(output - y_true) / 2.
optimizer = tf.train.GradientDescentOptimizer(.01)
train = optimizer.minimize(loss)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(500000):
    rand_index = np.random.randint(0, 500)
    _, error = sess.run([train, loss], feed_dict={x: [x_data[rand_index]],
                                              y_true: [y_data[rand_index]]})
    if i % 10000 == 0:
        print(error)

sess.close()


texts: Definitions/meanings of UCI Census-Income KDD dataset variables

texts:
Where can you find more detailed information on the variables in the UCI census income dataset? (Available here).
For example, what do the codes mean in the variable: industry code (AMJIND)?
Looking on the website, it is hard to find anything related to this dataset (1994/1995).


texts: what are Latent model and Latent Varibles?

texts:
what are latent models and why they use Expectation maximization instead of gradient descent to optmize the parameters ?
How to interpret and understand whether a hidden varible is present in model or not ?


texts: KL divergence loss goes to zero while training VAE

texts:
I'm trying to train a variational autoencoder to perform unsupervised classification of astronomical images (they are of size 63x63 pixels). I'm using an encoder with 2 convolutional layers and a dense layer, and a similar structure for the decoder. I'm performing Xavier initialization of gradients. I'm using the Adam optimizer with a learning rate of 1e-4.
I observe that the KL divergence starts at very small values (roughly of the order of 1e-4) and suddenly vanishes after a few epochs while training, while my reconstruction loss reduces normally (I use MSE as the reconstruction loss). What could be the reason? Should I perform scaling of the losses separately?
This is my model.
initializer = glorot_normal()

def sampling(inputs):
  z_mean, z_log_var = inputs
  epsilon = k.random_normal(shape=(k.shape(z_mean)[0], 2), mean=0., stddev=0.1)
  return z_mean + k.exp(z_log_var) * epsilon

input_images = Input(shape = (63,63,1))
conv1 = Conv2D(16, (3,3), activation = 'relu')(input_images)
conv2 = Conv2D(8, (3,3), activation = 'relu')(conv1)
flattened = Flatten()(conv2)
x = Dense(4, activation = 'relu')(flattened)

z_mean = Dense(2, name = "z_mean")(x)
z_log_var = Dense(2, name = "z_log_var")(x)
z = Lambda(sampling, output_shape = (2,))([z_mean, z_log_var])

encoder = Model(input_images, [z_mean, z_log_var, z], name = "encoder")

latent_inputs = Input(shape = (2,))
x = Dense(59*59*8, activation = 'relu')(latent_inputs)
x = Reshape((59,59,8))(x)
conv4 = Conv2DTranspose(8, (3,3), activation = 'relu')(x)
decoded = Conv2DTranspose(1, (3,3), activation = 'softmax')(conv4)

decoder = Model(latent_inputs, decoded, name = "decoder")

z_mean, z_log_var, z = encoder(input_images)
vae_decoder_output = decoder(z)
vae = Model(input_images, vae_decoder_output, name = "VAE")
vae.summary()

This is the loss function that I'm trying to implement.
recon = MSE(input_images, vae_decoder_output)
recon = k.mean(recon)
kl_loss = 1 + z_log_var - k.square(z_mean) - k.exp(z_log_var)
kl_loss = k.sum(kl_loss, axis = -1)
kl_loss *= -0.5
vae_loss = k.mean(kl_loss*10**3+recon)
vae.add_loss(vae_loss)


texts: What exactly negative/positive value of Captum's Integrated Gradient mean?

texts:
I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ...

Positive attribution score means that the input in that particular
position positively contributed to the final prediction and negative
means the opposite. The magnitude of the attribution score signifies
the strength of the contribution. Zero attribution score means no
contribution from that particular feature.

But what exactly positively/negatively contribution mean in human language ?
I have to explain these to my co-worker.
Does negative mean that value of feature have less informative to prediction of that class ?
When I average score of all values as graph below.
What exactly the most negative(or most positive) score mean in human language ?


texts: Classifying time series data that overlap

texts:
I am working with Time-Series Data that has to be classified into two classes (Blue and Red) or at least Classify the data as one class (Red), I'm unable to come up with features that distinctly separate the data.

Please advise as to how do I need to approach this problem.


texts: Transformation of non-categorical discrete feature

texts:
Goal: Predict a performance score of a place of interest in a given city based on (amongst others), the number of restaurants within 200m. $\\$

  Dataset: $D$ with a feature $x$ indicating the $\textit{number of restaurants within 200m}$ of some place of interest (target varible, $y$)

  $\textbf{Question}$: How can x be tranformed, such that it fits e.g a gaussian distribution for better modelling?

I am not sure how this is handled. If it was a categorical variable, it could just be a one-out-of-k encoding.

Are there any standard approaches to this? A logarithmic transformation does not yield anything useful.


texts: sklearn : missing pruning for decision trees

texts: Why pruning is not currently supported in scikit-learn? How can we tune the decision trees to make a workaround?


texts:
Any suggestion on what kind of dataset lets say $n \times d$ ($n$ rows, $d$ columns) would give me same eigenvectors?.

I believe it should be the one with same absolute values in each cell. Like alternating +1 and -1. But it seems to work otherwise.

Any pointers?


texts: How do Bayesian methods do automatic feature selection?

texts:
Someone asked me this question and I do not know I answered it correctly.
I answered the question in the following way: One type of Bayesian method is Bayesian inference and feature selection has to do with ${L}^{1}$ regularization because it is used extensively for this purpose. So, for ${L}^{1}$ regularization, the penalty $\alpha \Omega (\boldsymbol{w}) = \alpha \sum_{i} |w_{i}|$ used to regularize a cost function is equivalent to the log-prior term that is maximized by MAP Bayesian inference when the prior is an isotropic Laplace distribution.

But my question is this an automatic feature selection? ${L}^{1}$ regularization finds the specific subset of the available features to be used. Also is my answer correct to this question? I am just curious to know if my line of thinking makes sense or if it does not. If it does not, please let me know why. Thanks


texts: Is there wights of voice or audio for VGG or Inception?

texts:
I want to use VGG16 (or VGG19) for voice clustering task.
I read some articles which suggest to use VGG (16 or 19) in order to build the embedding vector for the clustering algorithm.
The process is to convert the wav file into mfcc or plot (Amp vs Time) and use this as input to VGG model.
I tried it out with VGG19 (and weights='imagenet').
I got bad results, and I assumed it because I'm using VGG with wrong weights (weights of images (imagenet))

So:

Are there any audio/voice per-trained weights for VGG ?
If not, are there other per-trained audio /voice models ?


texts: How to pre-process the name String of a customer?

texts: I implement logistic regression to predict if a customer is a business or a non-business customer with the help of TensorFlow in Python. I have several feature candidates like name, street, zip, longitude and latitude. At them moment I am thinking of how to use the name field. The name often has repeating parts like “GmbH” (e.g. “Mustermann GmbH”) which in this context has a similar meaning to Corp. which is an indicator that the customer is a business customer. This information is useless in combination with the other parts of the name because then the name will be unique. So my question is: how should I pre-process this field so that only repeating parts will be used to predict the classification?


texts: How to interpret feature importance in text classification using Fasttext?

texts: Once the text is converted into a vector of size(1,100), how can we interpret and backtrace a word's importance which helped in classification?


texts: Causes of vanishing and exploding gradient

texts: I know the concept between the vanishing and exploding gradient. I would like to know the possible causes of these phenomena. I think I read somewhere on the internet about the activation functions. Could someone please clarify? Any help will be greatly appreciated.


texts: incremental learning vs transfer learning

texts: Can anyone explain me how incremental learning differs from transfer learning with example? Also does Transfer learning limited to neural networks?


texts:
I have made a model to predict some values. In real time, it works perfectly fine and yields prediction within a fraction of a second. The algorithm I am using is PassiveAggresiveClassifier to predict on multiple classes.

However, when I load tested it, it lagged in performance. I tried on passing 500 requests at a time and it essentially bogged down.

Is there a way to enhance this aspect of a model program-wise? Can python solve this? Is there another way?

Thank you.


texts:
is there cross validation for xgb classification for multi labels?
I have been search but can not find any cross validation for xgb classifier
is using cross validation for xgb or xgb classifier worth?
and is there any example I can find?


texts: Appending DataFrames to lists in a dictionary - why does it seem like the list is being referenced by each new DataFrame?

texts:
I have a DataFrame that pairs one or more labels to a sample group and id, for a given sample stored in a database at SampleGroup/SampleID:

There are ~100 labels. I want to create binary models to do classification on each label, and then run these models in parallel to do multi-class classification. To store these models, I am creating a dictionary of form

{label_1:[df_1, model_object_1],
 label_2:[df_2, model_object_2],
...,
label_n:[df_n, model_object_n]
}

Where each df is a DataFrame of the form above, except that the value of the 'Labels' column is replaced with a 1 or 0, depending on whether dictionary key 'label_i' is in the original label list for that row. Here's the code that (should) do that, that has been giving me some trouble:

models = dict.fromkeys(target_labels, [])

for label in target_labels:
    label_list = []
    for multi_label_list in df['Labels']:
        if label in multi_label_list:
            label_list.append(1)
        else:
            label_list.append(0)

    data = {
        'SampleGroup':df['SampleGroup'].copy(),
        'SampleID':df['SampleID'].copy(),
        'Labels':label_list
    }

    models[label].append(pd.DataFrame(data=data, index=df.index))
    print(len(models[label]))

When I run this, each new binary label_list that is created for a label gets appended to every model in the dictionary, as if I'm creating a reference to the same label_list (similar to how df2 = df would create a reference to df, instead of a copy). The output of the above code tells the story clearly:

[

I managed to hack a fix for this by assigning each new DataFrame to the key instead of appending it to the key's value list:

models[label] = (pd.DataFrame(data=data, index=df.index))

What property of DataFrames (or perhaps native Python) am I invoking that would cause this to work fine, but appending to a list to act strangely?


texts: Scale out to between 0 and 1 before computing loss fucnction in keras

texts: Is there a way to scale the output of a CNN before computing the loss function in keras. It would have to work for batches as well. Specifically I am training images and the output is a probability map of each pixels for the entire batch. Then I compute the loss function with the ground truth data. The problem is that the predicted output during training at some intermediate stage lies between 0.02 to 1e-4 and the ground truth pixels are between 0 and 1 so the loss function doesnt penalize pixels which dont look similar to the ground truth. I get this by visualizing the squared error between the output at some intermediate stage and the ground truth data.


texts: Which text clustering algorithms should I use to group free text fields in Python?

texts: I have a data set in which users have input free text (think "APPLE", "Apple", "apple", "appl") and I'm trying to group the input by response type (e.g. organize all of the preceding examples by "Apple"). What's the best way to do this in Python?


texts: Improve Precision of a binary classifier - Decision Tree in Python

texts:
Currently, I am working on a project. The dataset is balanced roughly in the ratio of 50:50. I created a decision tree classifier. I am achieving decent accuracy (~75%) on validation data but the precision for the target variable is biased. For class=0 it is approx. 98% while for the class = 1 it is just 17%.

I have tried scaling the data using MinMaxScaler still no luck.

model = tree.DecisionTreeClassifier(class_weight={1:30}, min_samples_leaf=160, max_depth=10)

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=10)

min_max_scaler = preprocessing.MinMaxScaler()
X_train_scaled = min_max_scaler.fit_transform(X_train)
X_test_scaled = min_max_scaler.fit_transform(X_test)

model = model.fit(X_train_scaled, y_train)

prediction = model.predict(X_test_scaled)

print metrics.accuracy_score(y_test, prediction)
print classification_report(y_test, prediction)

Size of x_train_scaled = 12600 and x_test_scaled = 5400
Accuracy: 75%
Precision: {0:100%, 1:17%}
Recall: {0:74%, 1:100%}
F1-Score: {0:85%, 1:29%}

How can I improve the precision of class=1 while still maintaining the overall precision and accuracy?