texts: Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier?

texts: I'm new to ML and training classifiers in practice, so I was just wondering what the difference was between the built-in sentiment tools of packages such as NLTK and TextBlob as compared to manually creating a classifier (training, testing, etc). I think I read in a comment somewhere that Textblob/NLTK's existing sentiment analysis tools basically just tokenize the text and count the number of positive/negative words to determine an overall sentiment rating (not sure how accurate this is). Does anyone know if using a custom classifier would, in general, be a better way to doing sentiment analysis of text (I'm looking at analyzing the sentiments expressed in hotel reviews)?


texts: Heatmap of large 2D array using datashader and plotly

texts:
I’m trying to show a heatmap of a large 2D array (160x250000 entries). This should go into a dash app so I'm using plotly to deal with graphics and my idea was to use datashader for performance but I’m having troubles getting it right. However, independently of dash I’m already having problem with plotly + datashader (see code below). There is probably something very basic I’m not understanding in this process. It would be great if someone could tell me what I’m doing wrong.
Below you can find a minimal example to reproduce the issue (using offline.plot since I’m using spyder to run it).
If I’m not wrong, the code is defining a grid of 900x300 and the value represented in each element of the grid is the mean of the original 2D array entries falling in that grid element. Now, I’m expecting the figure to keep the resolution fixed as I zoom in and to show more fine-grained details - at some point, if I zoom in enough, in each grid element there should be just one entry of the original 2D array.
However when I zoom in in the produced figure the aggregation does not change.
I'm using plotly 4.12.0 and datashader 0.11.1. Please let me know if further information is needed.
Any help understanding this would be much appreciated.
import plotly.express as px
from plotly.offline import plot
import numpy as np
import datashader as ds
from datashader import reductions as rd
import xarray as xr

pw_s = np.random.randn(150, 25000)
pw_s[:, 10000:] += 3
pw_s = xr.DataArray(pw_s, coords=[('y', np.arange(150)),
                                  ('time', np.arange(25000)/2000)])
cvs = ds.Canvas(plot_width=900, plot_height=300,
                x_range=(0, 25000/2000),
                y_range=(0, 150))
agg = cvs.raster(pw_s, agg=rd.mean())

fig = px.imshow(agg)
plot(fig)
```


texts: Python Seaborn: How can one obtain a box plot (box and whisker plot) for the non-diagonal variable pairs in a pairplot()?

texts:
I tried using kind="box" but am not sure if that works. For the example I tried, I got blank subfigures in the non-diagonal parts of the PairGrid.

Edit: Details.

The function I am trying to use is seaborn.pairplot. As mentioned in the link, by default, this function will create a grid of Axes such that each variable in the data parameter will be shared in the y-axis across a single row and in the x-axis across a single column. The diagonal Axes are treated differently, drawing a plot to show the univariate distribution of the data for the variable in that column.

I am looking for a way to make sure that the kind parameter leads to a box plot (seaborn.boxplot) for the non-diagonal parts of the PairGrid.


texts: How do reshape an image to fit my Mnist Convolutional model?

texts:
I have done research but cannot seem to find what's wrong here
I have created this model for Mnist digit clasification :
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D , MaxPool2D,Dropout,Flatten
from tensorflow.keras.datasets import mnist

model = Sequential()
model.add(Conv2D(filters = 32 , kernel_size = (3,3),activation ='relu' , input_shape = input_shape ))
model.add(Conv2D(64,(3,3),activation = 'relu'))
model.add(MaxPool2D(2,2))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128,activation = 'relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation ='softmax'))

and I have this image to feed into the model:

So i reshaped the image like this:
from skimage import data
from skimage.transform import resize
image = image_gray
image = resize(image, (28, 28,1)).shape
image

But when i try to predict using this image i get this error:
ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (3, 1)

I have tried multiple ways using skimage but I can't seem to get the model to fit. What can I do?


texts: We have a time series data.Is it possible to create a RNN such that there is only dimenson(or feature) in data(as shown in fig1).If yes,will it identify the pattern in time series data correctly? The confusion arises because only neuron (yellow to blue in figure 2) information is passed to the output.Is it sufficient to predict a pattern in time series data?


texts:
I have a data set consisting of 100 features each of which are ternary: values of -1 if it exists in one category, 0 if it doesn't exist, and 1 if it exists  in the second category. For example

F1 F2 F3 ... F90 F91 F92 ... F99 F100
0  0  0  ... 1   -1  0   ... 0   -1
0  -1 0  ... -1   0  1   ... 0   0

The data is very sparse, ~20 of the 100 features have values of -1 or 1 for each row of data. I want to find similar rows of data through a heatmap visualization and dendrogram but I was confused on whether to use Euclidean distance or Cityblock distance. I'm quite new to data mining and while reading the scipy pages, I found many distance measures which I have no idea means what. Is there a good distance measure for my type of data set?


texts: Is my approach for a loss function that adds more importance to negative samples reasonable?

texts:
For my current project I'm using XGBoost Regression to predict values y_pred with mean = 0 and std = 1. I want my model to place more emphasis on predicting samples right, where the true value y_true is negative. Would the follwing loss function be a reasonable approach?
L(y_true,y_pred) = (y_pred - y_true) ^ 2 * exp(-y_true)


texts: How can I restart model fit after x epochs if loss is high?

texts:
I am having trouble sometimes getting a fit on my data, and when I restart fit (with shuffle=true) then I sometimes get a good fit.

See my previous question:

Why does my model sometimes not learn well from same data?

As a work around, I want to automatically restart the fitting process, if loss is high after x epochs. How can I achieve this?

Here is a simplified structure:

def train_till_good():
    # Some loop around train()

def train():
    load_data()
    model = VerySimpleNet2();

    checkpoint = keras.callbacks.ModelCheckpoint(filepath=images_root + dataset_name + '\\CheckPoint.hdf5')

    myOpt = keras.optimizers.Adam(lr=0.001,decay=0.01)

    model.compile(optimizer=myOpt, loss='categorical_crossentropy',  metrics=['accuracy'])

   LRS = CyclicLR(base_lr=0.000005, max_lr=0.0003, step_size=200.)

    tensorboard = keras.callbacks.TensorBoard(log_dir='C:\\Tensorflow', histogram_freq=0,write_graph=True, write_images=False)

    model.fit(train_images, train_labels, shuffle=True, epochs=num_epochs,
              callbacks=[checkpoint,
                         tensorboard,
                         LRS],
              validation_data = (test_images, test_labels)
              )

def VerySimpleNet2():
    model = keras.Sequential([
        keras.layers.Dense(112, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.4),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])
    return model

  EDIT 1

Here is another question that describes the data:

Why does adding random pixels stop my model learning in cnn?


texts: What is the classical way to visualize 3D filters in convolutional neural networks?

texts:
Suppose that at a layer $N$ within a CNN, my "image" is a 200$\times$200$\times$10 array. Thus, if I convolve such an array with, for example, 15 filters of size 3$\times$3$\times$10, I will end up with a new "image" whose shape is $A\times{B}\times{15}$ where $A$ and $B$ depend on the stride of the convolution.

I am thus wondering how such 3D (3$\times$3$\times$10 in my toy example) convolution filters are usually displayed as 2D images (i.e. sum or mean, ...)?


texts:
I have this formula for the density estimation.
$$p_n(x) = \frac{k_n / n}{V_n}$$

I have been told that with a Parzen window approach you can specify $V_n$ as a function of $n$. So if $V$ decreased when $n$ increased it is clear that it is a fixed volume.

I have also been told that with a knn approach you specify $k_n$ as a function of $n$. So if you increase as $n$ is raised, it is clear that the volume is dependent of the volume.

So can anyone explains me the above statements. I think it is somewhat clear to me how knn and Parzen works. (knn count the $k$ nearest neighbor, at the new sample is assigned to the class which has most votes. In Parzen the volume is fixed).

I also do not understand the two formulas in the figure. The figure illustrated two methods for estimating the density at a point $x$ at the center of
each square. The top knn, bottom Parzen


texts: How to avoid resampling part of pipeline on test data (imblearn package, SMOTE)

texts:
I am using the imblearn package to resample some data before applying other transformation/prediction techniques.  Specfically, I am using SMOTE in a slightly unconventional way, as a data augmentation technique to upsample all classes rather than just the minority.

I would like to do a pipeline of SMOTE -> PCA -> SVM on data separated into training and test (most likely K-fold).  However, consider the following snippet of code from the imblearn pipeline doccumentation:

pipeline = Pipeline([('smt', smt), ('pca', pca), ('knn', knn)])
X_train, X_test, y_train, y_test = tts(X, y, random_state=42)
pipeline.fit(X_train, y_train)
y_hat = pipeline.predict(X_test)

In the last line, it seems the test data will be resampled, which doesn't make sense -- only the training data should be resampled using this technique, or you will be testing on artificially generated data.  The documentation for pipeline.predict() implies samplers are applied during prediction as well.  Is there a way to avoid this, besides simply removing SMOTE from the pipeline?


texts: Explaination or Description of clusters after clustering

texts:
After clustering, is there a way to explain the clusters? Or get the boundaries of the clusters?

For example:
If we have a data set of people's spending habits with columns for their spend in different categories like groceries, clothing, transportation, rent etc. And we applied a clustering algorithm (like k-means or agglomerative clustering) on it.
Can we get descriptions of clusters, like:

Cluster 1 contains people who spend

More than \$500 on groceries
Less than \$200 on transportation

Cluster 2 contains people who spend

Less than \$100 on rent
Less than \$300 on transportation
More than \$50 on transportation

Basically I need an explanation which is meaningful to a layman user.


texts: Why I am getting prediction score 1 i.e. 100%

texts:
I am reading few parameters and trying to predict target value using Linear regression and GB. Surpicingly I am getting score = 1 on test data. How come? Can anyone tell me whats wrong with this code?

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

dataset = pd.read_csv("prod_data_for_ML.csv",header = 0)

#Data Pre-processing
data = dataset.drop('organization_id',1)
data = data.drop('status',1)
data = data.drop('city',1)

#Find median for features having NaN
median_zip, median_role_id, median_specialty_id, median_latitude, median_longitude = data['zip'].median(),data['role_id'].median(),data['specialty_id'].median(),data['latitude'].median(),data['longitude'].median()
data['zip'].fillna(median_zip, inplace=True)
data['role_id'].fillna(median_role_id, inplace=True)
data['specialty_id'].fillna(median_specialty_id, inplace=True)
data['latitude'].fillna(median_latitude, inplace=True)
data['longitude'].fillna(median_longitude, inplace=True)

#Fill YearOFExp with 0
data['years_of_experience'].fillna(0, inplace=True)

#Start training

labels = dataset.location_id
train1 = data
reg = LinearRegression()
x_train , x_test , y_train , y_test = train_test_split(train1 , labels , test_size = 0.20,random_state =1)

# x_train.to_csv("x_train.csv", sep=',', encoding='utf-8')
# x_test.to_csv("x_test.csv", sep=',', encoding='utf-8')

reg.fit(x_train,y_train)
reg.score(x_test,y_test)
```


texts: How to design n-dimensional feature descriptor similar as the input image?

texts:
I am re-writing the H-Net code in Keras for cross-domain image similarity. The network architecture is described in the attached paper. I wrote the encoder and decoder parts but unable to get similar dimenstions from decoder. Following is the code. Please, point me out where I am making mistake.

import tensorflow as tf
from keras.models import Model
from keras.layers import Input, Conv2D, Conv2DTranspose, BatchNormalization, Activation, MaxPool2D

# pe: Photo Encoder, PFM: Photo Feature Map, pd: Photo Decoder

# Input shape for encoder
input_img = Input(shape=(265,256,1))

pe = Conv2D(96, kernel_size=11, strides=(4,4), padding = 'SAME',name='conv1')(input_img)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)
pe = MaxPool2D((3, 3), strides=(2, 2), padding = 'VALID', name = 'pool1')(pe)

pe = Conv2D(256, kernel_size=5, strides=(1,1), padding = 'SAME', name='conv2')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)
pe = MaxPool2D((3, 3), strides=(2, 2), padding = 'VALID', name = 'pool2')(pe)

pe = Conv2D(384, kernel_size=3, strides=(1,1), padding = 'SAME', name='conv3')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)

pe = Conv2D(384, kernel_size=3, strides=(1,1), padding = 'SAME', name='conv4')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)

pe = Conv2D(256, kernel_size=3, strides=(1,1), padding = 'SAME', name='conv5')(pe)
pe = BatchNormalization()(pe)
pe = Activation('selu')(pe)
pe = MaxPool2D((3, 3), strides=(2, 2), padding = 'VALID', name = 'pool3')(pe)

pe = Conv2D(1024, kernel_size=7, strides=(1, 1), name = 'con2vector', padding = 'VALID')(pe)
pe = BatchNormalization()(pe)
p_encoder = Activation('selu')(pe)

pfm = tf.reshape(p_encoder,[-1, 1024])

'''
Output of each conv2d and pooling layers for Photo-Encoder

Block 1 convolution:  (?, 64, 64, 96)
Block1 maxpooling:  (?, 31, 31, 96)
Block 2 convolution:  (?, 31, 31, 256)
Block 2 convolution:  (?, 15, 15, 256)
Block 3 convolution:  (?, 15, 15, 384)
Block 4 convolution:  (?, 15, 15, 384)
Block 5 convolution:  (?, 15, 15, 256)
Block 5 maxpooling:  (?, 7, 7, 256)
Block 6 convolution:  (?, 1, 1, 1024)
Shape of photo feature map:  (?, 1024)
'''

'''
Following code is the implementation of photo decoder
'''
pd_input = tf.reshape(p_encoder, [-1, 2, 2, 256])

pd = Conv2DTranspose(128, kernel_size=5, strides=(2, 2), padding='SAME', name="pdcnv1")(pd_input)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(64, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv2")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(32, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv3")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(16, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv4")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(8, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv5")(pd)
pd = Activation("selu")(pd)

pd = Conv2DTranspose(4, kernel_size=5, strides=(2, 2), padding='SAME',name="pdcnv6")(pd)
pd = Activation("selu")(pd)

p_decoder = Conv2DTranspose(3, kernel_size=5, strides=(2, 2), padding='SAME', activation='sigmoid', name="pdcnv7")(pd) # (?, ?, ?, 3)

"""
Output of each Conv2DTranspose layer
Photo decoder input shape:  (?, 2, 2, 256)
Photo deconv block 1:  (?, ?, ?, 128)
Photo deconv block 2:  (?, ?, ?, 64)
Photo deconv block 3:  (?, ?, ?, 32)
Photo deconv block 4:  (?, ?, ?, 16)
Photo deconv block 5:  (?, ?, ?, 8)
Photo deconv block 6:  (?, ?, ?, 4)
Photo deconv block 7:  (?, ?, ?, 3)
"""

autoencoder = Model(input_img, p_decoder)


texts: Simple feed forward neural network on Tensorflow question

texts:
I'm building a simple feedforward neural network in Tensorflow and something seems to be broken.  I have adopted the basic structure from http://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/ and have tried to adapt it to spit out 1 output unit instead.  The input is 10 numerically represented players ex:(5, 17, 19, 54, 110, 3, 112, 78, 60, 63), 5 on the 'North' team and 5 on the 'South' team, and the output is a 1 or 0 (if the 'North' team won or not).  3 hidden layers (200 parameters each).  With how this is set up, the algorithm trains to constantly predict 1 (North victory).  What can I fix that allows this to train to predict which team is going to have a higher chance of winning?  My expected outcome is ex: [.54] (slight edge to the north team) or [.40] (edge to the south team) etc.

import numpy as np
import tensorflow as tf
import sqlite3

def get_matches(start, stop):
    connection = sqlite3.connect('5v5.db')
    cursor = connection.cursor()

    result = cursor.execute('''SELECT * FROM matches WHERE rowid>=? AND rowid<=?''', (start, stop))

    return result.fetchall()

def init_weights(shape):
    return tf.Variable(tf.random_normal(shape, stddev=0.01))

def model(X, w_h1, w_h2, w_h3, w_o):
    h1 = tf.nn.sigmoid(tf.matmul(X, w_h1))
    h2 = tf.nn.sigmoid(tf.matmul(h1, w_h2))
    h3 = tf.nn.sigmoid(tf.matmul(h2, w_h3))
    return tf.nn.sigmoid(tf.matmul(h3, w_o))

if __name__ == "__main__":
    TRAINING_GAMES = 2500 #number of games to train on
    TEST_GAMES = 500 #number of games to test the results on

    NUM_DIGITS = 10 #input units
    FIRST_HIDDEN_LAYER = 200
    SECOND_HIDDEN_LAYER = 200
    THIRD_HIDDEN_LAYER = 200
    BATCH_SIZE = 150

    training_matches = get_matches(1, TRAINING_GAMES)
    trX = np.array([match[1:11] for match in training_matches])
    trY = np.array([[match[11]] for match in training_matches])

    X = tf.placeholder("float")
    Y = tf.placeholder("float")

    w_h1 = init_weights([NUM_DIGITS, FIRST_HIDDEN_LAYER])
    w_h2 = init_weights([FIRST_HIDDEN_LAYER, SECOND_HIDDEN_LAYER])
    w_h3 = init_weights([SECOND_HIDDEN_LAYER, THIRD_HIDDEN_LAYER])
    w_o = init_weights([THIRD_HIDDEN_LAYER, 1])

    py_x = model(X, w_h1, w_h2, w_h3, w_o)

    cost = tf.square(Y - py_x)
    train_op = tf.train.GradientDescentOptimizer(0.05).minimize(cost)

    predict_op = py_x

    with tf.Session() as sess:
        tf.initialize_all_variables().run()

        for epoch in range(10000):
            p = np.random.permutation(range(len(trX)))
            trX, trY = trX[p], trY[p]

            for start in range(0, len(trX), BATCH_SIZE):
                end = start + BATCH_SIZE
                sess.run(train_op, feed_dict={X: trX[start:end], Y: trY[start:end]})

                print(epoch, sess.run(predict_op, feed_dict={X: trX, Y:trY}))

        test_matches = get_matches(TRAINING_GAMES+1, TRAINING_GAMES+TEST_GAMES)

        teX = np.array([match[1:11] for match in test_matches])
        teY = sess.run(predict_op, feed_dict={X: teX})
        print(teY)


texts: Training multiple keras models and combining outputs to determine losses

texts:
I'm trying to predict the future states of a 1D travelling wave (square, triangle and sawtooth) using a deep learning setup in Keras. The waves are discretised in a 1024 data points. As this gives a quite high input dimensionality for an RNN, the 1024 inputs are reduced to a low dimensional latent space of 10 dimensions using a convolutional autoencoder setup. The RNN is given 5 previous time step latent inputs to predict the next time step. I have trained the autoencoder and RNN seperately and obtained pretty decent results, but ideally could be improved. The following figure gives an example of how I trained the models in a decoupled manner.

My idea to improve this model by coupling the training process. Here the autoencoder is trained with the same loss as the RNN. The following picture gives an idea of what i'm trying to achieve.

My hope is that the latent space will be more suitable for time predictions using this setup, as the autoencoder is penalised for creating a latent space which is hard to do time predictions with.

My problem is that I don't know how to implement the combining of the models in this manner in Keras. Could anyone help me with this?

P.S.

I have asked another question about this topic in the following link which gives some examples of my project and on the decoupled training results.

Other question on Data Science Stack Exchange


texts: Data augmentation for recommendation systems

texts:
I have a user-item matrix that I use to train a denoising autoencoder to predict the top-k items to recommend to the different users.
The idea is to corrupt the matrix by erasing a percentage p of the items that each users bought and train the autoencoder to reconstruct the uncorrupted matrix.
Following the implementation of this paper, I am currently erasing 20% of the bought items.
I was wondering if it is legit to augment the dataset by first erasing the p=20% to create the "noised" matrix and, successively, use for instance p=40% and concatenate the two noised matrices and trin the autoencoder to reconstruct a stack of two uncorrupted matrices.
Is it reasonable or is it just an invitation for overfitting?


texts: Music Generation LSTM not learning (Keras)

texts:
I am trying to train a RNN in keras to produce music but I am having difficulty training it. The loss seems to remain fairly high and constant despite me changing the hidden size and number of layers and when it does sometimes train it often outputs the same note for every input, outputting a song of just one note repeated. Is this a problem with my data or how can I fix my model? The notes are extracted from the midi file and then one hot encoded.

def read_midi(path,trans='C'):
    songs = glob(path)
    notes = []
    offsets = []
    durations =[]
    for file in songs:
        print(file)
        try:
            midi = converter.parse(file)
        except:
            continue
        parts = instrument.partitionByInstrument(midi)
        if parts is not None:
            notes_to_parse = parts.parts[0].recurse()
        else:
            notes_to_parse = midi.flat.notes
        for element in notes_to_parse:
            if isinstance(element,note.Note):
                notes.append(str(element.pitch))
                offsets.append(element.offset)
                durations.append(element.quarterLength)
            elif isinstance(element,chord.Chord):
                notes.append('.'.join(str(n) for n in element.normalOrder))
                offsets.append(element.offset)
                durations.append(element.quarterLength)
    return notes,offsets,durations

In this case I am just using the pitch just to see if I can get the RNN to train but I also planned to use the duration and change in offset as another feature by one-hot encoding combinations of the 3. The sequences are generated as follows

pitchnames = sorted(list(set(data)))
seq_length = 100
classes = len(pitchnames)
num_class = {i:ch for i,ch in enumerate(pitchnames)}
class_num = {ch:i for i,ch in enumerate(pitchnames)}
input_data = [class_num[x] for x in data]

input_temp =[]
output_temp=[]
for i in range(0,len(input_data)-seq_length,1):
    input_temp.append(input_data[i:i+seq_length])
    output_temp.append(input_data[i+seq_length])

sequences = len(output_temp)

x = np.reshape(input_temp,(sequences,seq_length,1))
x -= int(np.mean(x))
x = x/classes
y = keras.utils.to_categorical(output_temp)
assert classes == y[0].shape[0]

music_model = Sequential()
music_model.add(CuDNNLSTM(512,input_shape=(seq_length,1),return_sequences=True))
music_model.add(Dropout(0.5))
music_model.add(CuDNNLSTM(512))
music_model.add(Dense(256))
music_model.add(Activation('relu'))
music_model.add(Dropout(0.4))
music_model.add(Dense(classes))
music_model.add(Activation('softmax'))
music_model.compile(loss='categorical_crossentropy',optimizer='RMSprop')
filename = './ML/keras_rnn/music_weights.{epoch:02d}-{loss:.2f}.hdf5'
callbacks = [ModelCheckpoint(filename,monitor='loss',save_best_only=True)]
music_model.fit(x,y,callbacks = callbacks,epochs=1000,batch_size=64)

The loss seems to stop decreasing at around 5.03 and doesn't decrease further over the epochs. Any help would be greatly appreciated


texts:
Suppose we have predefined list of tags Tag #1, Tag #2, ..., Tag #N
and we want to assign tags to sample texts based on keywords or semantic analysis. A text can have multiple Tags. I don't want to consider it as classification problem assuming each tag as a separate class. What algorithm we can use here, Any suggestions would be of great help.


texts: How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

texts:
I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently see two different kind of codes:

Type 1 is where no TimeDistributed applied like this:

model=Sequential()

model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1], border_mode="valid", input_shape=[1, 56,14]))
model.add(Activation("relu"))
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=pool_size))

model.add(Reshape((56*14,)))
model.add(Dropout(0.25))
model.add(LSTM(5))
model.add(Dense(50))
model.add(Dense(nb_classes))
model.add(Activation("softmax"))

Type 2 is where TimeDistributed is applied like this:

model = Sequential()

model.add(InputLayer(input_shape=(5, 224, 224, 3)))
model.add(TimeDistributed(Convolution2D(64, (3, 3))))
model.add(TimeDistributed(MaxPooling2D((2,2), strides=(2,2))))
model.add(LSTM(10))
model.add(Dense(3))

My questions are:

Is my assumption correct that Type 1 is the one-to-many kind and Type
2 is the many-to-many kind? Or TimeDistributed has no relevance in
this aspect?
In either case of one-to-many or many-to-many is the last dense layer
supposed to be 1 node "long" (emitting only one value in turn) and
the previous recurrent layer is responsible to determine how many
1-long value to emit? Or the last dense layer is supposed to consist
of N nodes where N=max sequence length? If so, what is the point of
using RNN here when we could produce a similar input with multiple
outputs with N parallel "vanilla" estimators?
How to define the number of timesteps in RNNs? Is it somehow
correlated with the output sequence length or is it just a
hyperparameter to tune?
Inn case of my Type 1 example above what is the point of applying
LSTM when the model emits only one class prediction (of possible
nb_classes)? What if one omits the LSTM layer?


texts: Found input variables with inconsistent numbers of samples: [11232, 5616]

texts: I don't know what is the reason for the error please guide me and help me out. I am at a learning stage.


texts: Hyperparameter Tuning with Simulated Data

texts:
I'm trying to create a SVM classifier which can predict some fault, and to train it I'm using simulated examples of the fault. Of course, the simulations are not perfect, but they appear to be good enough since I get reasonable results when predicting on the real examples.

What is the best way to tune the hyperparameters in order to maximise the performance on the real data? Using GridSearchCV does a good job of maximising the accuracy on a subset of the simulated data, but that doesn't always mean good results on the real data.

Any general tips on working with simulated training data are also welcome.


texts: Algorithm to construct similarity structure from hash lookup table

texts:
I have constructed a lookup table using locality-sensitive hashing for comparing nearly similar documents/records. If two records (columns) have the same hash value in a row, they are considered to be similar. For example, the structure is

   R1 R2 R3 R4 R5
b1 a1 a2 a3 a2 a5 .....
b2 a2 a4 a1 a4 a4 .....
b3 a3 a5 a3 a7 a4....

Since each band holds similar records, similar record sets will be

S1 = {R2,R4}
S2 = {R2, R4, R5}
S3 = {R1,R3}

And the merged similarity structures will be

S1' = {R2,R4,R5}
S2' = {R1,R3}

I want to span the matrix and map all similarity structures so I have buckets of similar records. The dimensionality of the hash table is large, hence a naive approach is not likely to work. What types of algorithms should I consider to achieve this efficiently?

Edit 1:
Question updated to clarify more on the objective


texts: Why is sklearn.metrics.roc_auc_score() seemingly able to accept scores on any scale?

texts:
I had input some prediction scores from a learner into the roc_auc_score() function in sklearn. I wasn't sure if I had applied a sigmoid to turn the predictions into probabilities, so I looked at the AUC score before and after applying the sigmoid function to the output of my learner. Regardless of sigmoid or not, the AUC was exactly the same. I was curious about this so I tried other things like multiplication by arbitrary numbers and applying arbitrary log or exp functions and the score was still the same.

Assuming I haven't made some other error somewhere, why is the sklearn function for ROC AUC able to work on any scale of scores?


texts:
As the title says:

Where does the random in Random Forests come from?


texts: Cross validation for anomaly detection using autoencoder

texts:
I am using autoencoder for anomaly detection in warranty data. I don't have any ground truth labels to confirm whether the anomalies detected by the model is really an anomaly or not. Since I don't have the labels, I would like to know if there is a way to do cross validation in that case?

Any help is much appreciated


texts: Paper/Idea to get the most attractive chapter in a novel

texts:
I face a problem as below:

There is a novel app and need to find the most attractive chapter in previous 30 chapters of each novel
So that we can limit a user continue unless he make payment

I didn't find such paper/idea through google .
What I thought is :
Set different payment point in different chapter, show to random user and record the CTR .
Then we can use it to get payment distribution to set proper payment point .
But we don't have many user now, and cost would be large .
So I wonder is there any NLP way to detect most attractive chapter ?


texts: How to extract location related terms from raw text in python

texts:
I want to extract location related keywords from raw text in python.
I have already tried spacy but the results were not good and I just got names of countries while I want fine-grained location mentions like streets or neighborhoods in a city.
I also have tried stanford NER but the problem is it is too slow and I need to produce my results with a good speed.
Is there any package or library for python which can solve my problem?
Also if there is any other suggestions which are not for python I will be glad to hear them.


texts:
So, I've been trying to implement my first algorithm to predict the (sales/month) of a single product,  I've been using linear regression since that was what were recommended to me. I'm using data from the past 42 months, being the first 34 months as training set, and the remaining 8 as validation.

I've been trying to use 4 features to start:

Month number(1~12)
Average price that the product was sold during that month
Number of devolutions previous month
Number of units sold previous month

Here are images with graphs comparing the Real Data x Predicted Data and a Error x number of elements graph:

So far the results are not good at all (as shown in the images above), the algorithm can't even get the training set right. I tried to use higher degrees polynomials, and the regularization parameter, it seems to make it worse.

Then, I would like to know if there is a better approach for this problem, or what could I do to improve the performance.

Thanks a lot in advance!


texts: Counting non-overlapping objects in a semantic segmentation prediction mask

texts:
What is a good way to count the number of roofs detected by a CNN in the following output on the right (produced using keras/tensorflow):

I need to count the discrete shaded areas and estimate their area in pixels. I'm a little stumped. It is not strictly an image classification problem.


texts: How to implement Brown Clustering Algorithm in O(|V|k^2)

texts:
I am trying to implement the Brown Clustering Algorithm.

Paper details: "Class-Based n-gram Models of Natural Language" by Brown et al

The algorithm is supposed to in O(|V|k^2) where |V| is the size of the vocabulary and k is the number of clusters. I am unable to implement it this efficiently. In fact, the best I can manage is O(|V|k^3) which is too slow. My current implementation for the main part of the algorithm is as follows:

for w = number of clusters + 1 to |V|
{
   word = next most frequent word in the corpus

   assign word to a new cluster

   initialize MaxQuality to 0

   initialize ArgMax vector to (0,0)

   for i = 0 to number of clusters - 1
   {
      for j = i to number of clusters
      {
         Quality = Mutual Information if we merge cluster i and cluster j

         if Quality > MaxQuality
         {
            MaxQuality = Quality

            ArgMax = (i,j)
         }
      }
   }
}

I compute quality as follows:

1. Before entering the second loop compute the pre-merge quality i.e. quality before doing any merges.
2. Every time a cluster-pair merge step is considered:
    i. assign quality := pre-merge quality
   ii. quality = quality - any terms in the mutual information equation that contain cluster i or cluster j (pre-merge)
  iii. quality = quality + any terms in the mutual information equation that contain (cluster i U cluster j)  (post-merge)

In my implementation, the first loop has approx |V| iterations, the second and third loop approx k iterations each. To compute quality at each step requires approx a further k iterations. In total it runs in (|V|k^3) time.

How do you get it to run in (|V|k^2)?


texts: What is the difference between Concatenate() and concatenate() layers in Keras / TensorFlow 2.0?

texts: I am learning TensorFlow 2.0, whose layer functions are based on Keras. What is the difference between the Concatenate() and concatenate() layers?


texts: Get row wise frequency count of words from list in text column pandas

texts:
I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences

words = ["rain", "buy new house", "tornado"]

What I need to do is create a column in the data frame which checks these words in the text column row by row and if it presents then update the column with word and it's frequency. For example first row text

"I was going to buy new house last week but it was raining since then. Once the rain stops I'll go and buy new house"

the column should read

{"buy new house",2}, {"rain",2}

or may be create a duplicate row and add the comma part in next row.

How to proceed in this as I am fairly new.


texts: Can a siamese neural network distinguish expected from unexpected changes?

texts:
Please redirect me to another stack exchange if this isn't the appropriate forum. I am interested in finding a neural network architecture that can detect distance between two inputs. As an example, say I want to detect pairs of signatures. A pair would consist of a person writing their signature with their left hand and their right hand. The model would then detect a forgery if the input were a pair of signatures from two different people both using their right hand.
I was interested in using a siamese neural network (SNN) for its continuous value prediction, but I am not sure if the SNN is appropriate for handling this problem of trying to differentiate an expected change (switching from right to left hand) from an unexpected change (different people writing).
Is the SNN still appropriate here? Or is there a model that would be more appropriate, preferably one that also outputs a continuous distance metric? My actual input is not images, but 2-by-N numeric matrices


texts: Testing independence of random variables in Python

texts: Are there any tools available in Python that allow for testing of independence of two random variables (data columns)? I have two columns of data $X$ and $Y$. They can be both discrete, with values $\{0,1\}$ or one of them can be continuous. I would like to perform some statistical test to be sure they are independent. I am using Python so it would nice to have some ready-to-use tool implemented. I my also use R if it is not something difficult to do.


texts: CNN training data size for determing the winner of tic-tac-toe

texts:
I'm trying to learn machine learning with tensorflow and wrote a program that uses CNNs to determine game results for a given tic-tac-toe board. Its inputs and outputs are -

Input  - An array of 9 elements that represents the board (0=empty, 1='X', 2='O')

Output - An array of 3 elements ([1,0,0]=draw, [0,1,0]=player1 won, [0,0,1]=player2 won).

Below is the tensorflow graph part of the program (it is a modified version of tensorflow tutorial at - https://www.tensorflow.org/get_started/mnist/pros).

baseFeatureSize = 64
x = tf.placeholder(tf.float32, shape=[None, 9])
x_image = tf.reshape(x, [-1, 3, 3, 1])
W_conv1 = nn_utils.weight_variable([2, 2, 1, baseFeatureSize])
b_conv1 = nn_utils.bias_variable([baseFeatureSize])
h_conv1 = tf.nn.relu(nn_utils.conv2d(x_image, W_conv1) + b_conv1)
W_conv2 = nn_utils.weight_variable([2, 2, baseFeatureSize, baseFeatureSize * 2])
b_conv2 = nn_utils.bias_variable([baseFeatureSize * 2])
h_conv2 = tf.nn.relu(nn_utils.conv2d(h_conv1, W_conv2) + b_conv2)
W_fc1 = nn_utils.weight_variable([3 * 3 * baseFeatureSize * 2, baseFeatureSize * 4])
b_fc1 = nn_utils.bias_variable([baseFeatureSize * 4])
h_pool2_flat = tf.reshape(h_conv2, [-1, 3 * 3 * baseFeatureSize * 2])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
W_fc2 = nn_utils.weight_variable([baseFeatureSize * 4, 3])
b_fc2 = nn_utils.bias_variable([3])
y_ = tf.placeholder(tf.float32, shape=[None, 3])
y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

def conv2d(x, W): return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x): return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)
def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

Questions - The above model works well but requires to train with over 700k inputs to get output prediction accuracy up to 80%. But the total possible permutations of the game board are less than 300K. If I feed just all of the unique permutations, then the accuracy worsens.

Is it normal for CNNs to require training data size to be much larger than all possible permutations of the inputs? I'm assuming this is not true or else they'd not be able to play chess and go etc, so what am I doing wrong?
Based on your experience, how big a training sample size would typically make sense for this setup?


texts: In Python, why subsetting with or without square bracket is different?

texts:
Suppose I have a data frame called quoteDF

quotesDF

              volume  shares
2017-01-03   2934300     100
2017-01-04   3381400     120
2017-01-05   2682300     140
2017-01-06   2945500     160
2017-01-09   3189900     180
2017-01-10   4118700     200

If I do,

> quotesDF.loc[1, 'shares']
120
> quotesDF.loc[1, ['shares']]
shares 120
Name: 1, dtype: object

Why the first one retunrs 120, the second one retures shares 120? In my mind, they are the same thing, except I put the second one in a vector. But, the first one is a vector stands by itself. It's just that I didn't put the square bracket on it.

Why Python give me such a confusing time?


texts: What is the explanation of discretized log logistic in the following code? Is there any reference regarding this piece of code?

texts:
I have taken the following piece of code from the OpenAI code of GitHub from the following link:
https://github.com/openai/iaf/blob/master/tf_utils/distributions.py#L28
def discretized_logistic(mean, logscale, binsize=1 / 256.0, sample=None):
         scale = tf.exp(logscale)
            sample = (tf.floor(sample / binsize) * binsize - mean) / scale
          logp = tf.log(tf.sigmoid(sample + binsize / scale) - tf.sigmoid(sample) + 1e-7)
           return tf.reduce_sum(logp, [1, 2, 3])
I am unable to understand mathematical intuition or explanation of the above code. Even, I am unable to find the reference for this. Could you please explain?


texts: What is the easiest way to identify a gender for a noun (in french)?

texts:
I am working on an app where in order to process some data, I need to be able to identify the gender for some selected words.
My data is in French.

The feature I am looking for should be able to tell me that "les garçons" is male plural, "chienne" is singular female.

What is the easiest word to do so?
If you have a link to a library do not hesitate.


texts: Why do Isolation Forest implementations turn it into a supervised learning problem (with random values for the target?)

texts:
I am looking at various implementations of the Isolation Forest in python and R.
Both sklearn in python and solitude in R use a y variable with the ExtraTrees regressor.
Since, Isolation Forest is unsupervised, I am wondering why it is being turned into a supervised problem? Wouldnt this be an issue when scoring on previously unseen data sets?
For example sklearn (python) line 248 has this.
And in solitude line 144 as well.


texts: RandomForest (RandomForestRegressor) returns weird predictions

texts:
I wish to see how different algorithms perform when predicting stocks (using technical indicators as features). When modeling the randomForest (and looking at the graph) I get very bizarre results.
For the first 5500 or so data points the predictions are just a straight line, after that, it predicts the price rather well (probably overfitted but regardless).

(blue line is the test and red is the prediction)
I was wondering if anyone knows why it's behaving so strangely.


texts: KNN Variance using a high value of K and cross-validation

texts:
it has come to my understanding, that a value of K=1, gives a high variance because we are only using only one data point, hence we are very likely to model the noise in that training example.

Bias: It will take the value of point 3 as it’s the closest one. It looks much better here (I know not the best example but you can consider another example such as house price and axis to be locality and price). It makes more sense that the point will be similar to the closest points compare to the point far away.

Variance: Let’s say point 11 was at age 32, then the closest point would have been 9 and hence the new predicted value would have been much different than the current one. Hence, it has a high variance.

They used a different data point from the test data, but why is point 11, at age 32?, what makes for the two data points to be related, because surely if I use age 50, I do not expect
a prediction close to the prediction at age 30, do they assume they have the same height, if then, why are we using our dependent variable to predict our dependent variable
Can someone provide me answers

Much appreciated


texts: How to Get Live Data Updates & Graphing with Tableau Public

texts:
I'm a big fan of Tableau's easy-to-use data visualization tools. I tried building dashboards with Python's dash package, but it was too complex for me and Tableau offered an easy way to building beautiful data visualizations.

The only downside to Tableau is that it's free version (Tableau Public) does not allow you to connect to a live data source and thus make live graphs and dashboards (only exception is Google Sheets once every 24 hours), for this you'd need to fork out USD 70 a month at least.

Hence my question: Has anyone found a workaround? Specifically, has anyone figured out a way to get live data updates using just Tableau Public?


texts:
I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as
$$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{
2/π}(x + 0.044715x^3)])$$

Could you simplify the equation and explain how it has been approximated.


texts:
I'm looking for an algorithm, that classifies a webpage. But not the content. I know the readability algorithm (by arc90) to extract a main text from an webpage. That for instance can decide, wether a webpage contains an article (is a news or blog article). Thus it can decide wether a webpage is an article page or not.

But I want to go further and classify, wether a non-article-webpage is a product page or a landingpage promoting a product or (!) wether a page has some purpose to motivate the visitor to take some action.

The third case has to be anything else. Whatever: It could be an overview site or else.

Does anyone know such a classifier or has any idea how to analyse the DOM-tree of a webpage in terms of similarity of striking container constellations?


texts: How to find the probabilities of certain events occurring in a defined sequence?

texts:
Good day, everyone!

I have a problem where I have to program something like this:

I have some arbitrary number of events, let's call then Event A, B, C, D, E, F, ... and so on.

Now they occur in well defined sequences, just like below:

[A, A, C, A, A, A, F, B, A]

That's just a random sample, I have a data of thousands of such event sequences, and there are going to be 9 events in every sequence.

What I'm trying to do is to calculate the probability of each individual event happening. Like:

what's the probability of event A occurring first
what's the probability of event B occurring, given that the previous event was A
what's the probability of event C occurring, given that the previous two events were A and B

So far I've gone through the maths behind conditional/joint probabilities, markov chains, but I can't get to come up with the programming part. Are there any leads to where I can get my hands on some framework, library, or a tool which will help me run the simulation with all of my sequences, and give any finalised probabilities?

Thanks!

P.S: For those who are aware of MLB, I'm just trying to find the probabilities of certain scores being made in all the 9 innings based on past scores. If I calculate the simple probability of a score being made, I get a very high probability of zero, which is not the case. I only have the data for the box scores.


texts: When steps_per_epoch is set, does the model only take the front of the data?

texts:
steps_per_epoch parameter adjust maximum batches of samples in a epoch. I want to use it because the whole training is too huge.
The dataset was set with dataset.repeat(3).map(data_augmentation_func).shuffle(100).batch(batch_size).

When using steps_per_epoch of model.fit of Tensorflow2 with tf.dataset, does the model only take the front of the dataset or randomly take the data from it?


texts: Evaluation metric for Information retrieval system

texts:
I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to
maximize Recall@100 and Mean Average Precision (MAP).
According to Introduction to Information Retrieval, Precision (P) is the fraction of retrieved documents that are relevant:

Recall (R) is the fraction of relevant documents that are retrieved:

I want to know how to come up with ground truth(relevancy label) if it's not available? In other words, if I want to calculate precision or recall for the Semantic product search and if we don't have relevancy label available for input product query. In that case, how researchers calculate precision and recall? or how do they generate it?


texts: How to combine human-labelled data with user behavior data?

texts:
I am working on a supervised learning problem for a web-search task, where I have access to a relatively small set of human-labeled examples and lots of user-behavior data.
Now, user behavior data is biased, because of presentation bias, position bias etc. So it's likely that its' distribution will be different from human-labeled data.
I am planning to use both to train a Neural Network model.
Now I am confused about how to combine both datasets?


texts:
If I want to suggest a course path for a student who wanted to be a chemical engineering where each degree has to go through certain mandatory courses like math ,physics chemistry . Again to complete these mandatory courses there is various pre-requisite courses . Again I have to suggest a best course path by breaking the course in season like fall,winter and summer etc according credit points rule of university .

So just wanted understand which algorithm or approach i should choose to solve this kind of problem .

I thought of choosing heuristic approach find the best path but there is various mandatory courses and student has to complete each mandatory courses and heuristic A* path finding algorithm works if you know the source and destination  .

Again I thought of using Travelling Salesman problem approach where student has to travel mandatory course but in this case problem is each mandatory courses has again various path . so how do approach about this .

Thanks in advance , Please guide me friends . appreciate your earl reply.


texts: I cant understand this backpropagation code

texts:
Below is the code for backpropagation for 1 epoch in a class called NeuralNetwork where:
sigmoid_deriv: Method to find derivative of sigmoid function.
error = A[-1] - y
D = [error * self.sigmoid_deriv(A[-1])]

for layer in np.arange(len(A) - 2, 0, -1):
    derivative = D[-1].dot(self.W[layer].T)
    derivative = derivative * self.sigmoid_deriv(A[layer])
    D.append(derivative)

1.
In the image:
dw2 = dz2 * a1.T
Where as in the code:
D = error * sigmoid_derivative(a2)     (if we assume there are only 2 layers)
My questions are:

How is above relation equal.
Is D derivative of loss with respect to w.
How is the derivative in the code equals the equations of dw1 in the image.


texts: If i no longer have access to feature in a data set should i retrain my model?

texts: based on naive Bayes theorem if I no longer have access to the address book to tell whether the email author is known. Should I re-train the model, and if so, how?


texts: What is the vector value of [CLS] [SEP] tokens in BERT

texts:
In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector?

I wanted to replace the proper nouns like names, buildings, locations with similar approach. How should i go about masking the same?


texts: Oversampling/Undersampling only train set only or both train and validation set

texts: I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set and then split it to train and validate set or first split into train and val set and then perform sampling(over/under) on the train set only


texts:
I am attempting to build a predictive model based on the past historical data. I have details of specific machine failure based on the past year data. I have data from some months of 2016 and from 2017 January to November. I am attempting to predict if the machine fails in December. I have attached the past historical data based on the data transformation I performed. I am stuck at a point on how to build a model to identify if the machine throws up any repair or replace in next one month.

I need to build a classification model which will identify if the machine fails. I don't understand on how to convert this multiple rows to single row record for each machine. Do I need to create categorical for day and month. How would I represent each sensor data column. Essentially look at the right way of representing data so that it is easier for classification to predict for next one month.

Any sort of guidance would be of great help dataset.csv
 train.csv

train.csv contains the 1 month data of December for the machines which are already predicted. The target column is to predict in the month of December, if the machine requires a repair or replacement or no issue.


texts: Applications and differences for Jaccard similarity and Cosine Similarity

texts:
Jaccard similarity and cosine similarity are two very common measurements while comparing item similarities. However, I am not very clear in what situation which one should be preferable than another.

Can somebody help clarify the differences of these two measurements (the difference in concept or principle, not the definition or computation) and their preferable applications?


texts: What is the difference between model hyperparameters and model parameters?

texts:
I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. Consider a machine learning model, an SVM/NN/NB based classificator or image recognizer, just anything that first springs to mind.

What are the hyperparameters and parameters of the model?
Give your examples please.


texts: How many features can we input for a SVM to classify?

texts:
I am new to SVM classifiers. I read on the internet that SVM are binary classifiers and also many SVMs, as described in research papers, only take 2 features as the input.

(e.g. https://scikit-learn.org/stable/auto_examples/svm/plot_iris.html)

My question is, does it have to be 2 input features? Can we use more than 2? If so, how do we write this code in python?


texts: What is the difference between the training and testdata set?

texts: What is the difference between the test and training data sets? As per blogs and papers I studied, what I understood is that we will have 100% data set that is divided into 2 sets (test data set is 30% and reaming 70% is training data set). I would like to know more points and use of differentiating the 100% data set to test and training data sets.


texts: speech accent recognition data augmentation and training

texts:
I am using a Kaggle dataset to learn more about using sound with Deep Learning.

I have extracted the mfcc features using this cool library - librosa.

However, I want to try feature extraction using CNN's and then train on LSTM.
(Any advice here would be good, if I am on the right track)

Should I just put the raw wav file into CNN or should I do some data augmentation before that?

Link to the dataset


texts: Reinforcement Learning with varying state space

texts:
I would like to ask a general question on reinforcement learning.
Let $S = ([x_0, y_0], [x_1, y_1])$ be the states, where $lower \leqslant x_i, y_i \leqslant upper$.
Is it possible (after training), to increase the state space, i.e., $S' = ([x_0, y_0], [x_1, y_1], [x_2, y_2])$ without training the model again? In other words, can we reuse the knowledge obtained in the previous step (S states)?


texts: How to find the related columns between column?

texts:
Suppose I have a data frame with columns car_id,number_car,bike_id,number_bike.
df =
Product_id   Category order_id   order   customer_id  customer Name
 pd01         car      or01      service   cus01       Jack
 pd02         bike     or02      repair    cus02       Swan
 pd03         car      or3       repair    cus03       Tom
 pd04         car      or04      cleaning  cus04       ram

Where we can make out that Product_id has a relation with Category, order_id has a relation with order and customer_id has a relation with customer name. But I am just curious about finding this by using nltk or any other way. Is it possible to find the relation between the columns using nltk word2vec? Just like sending a sentence as a list can we send the column as a list?
If not possible by nltk or is there any other way of doing this in data science, or any other method not related to data science. What is the best way to do it?
Basically, I am trying to find the dimensional tables without any human interference


texts: Imputation missing values other than using Mean, Median in python

texts:
I heard that Mean, Median isn't the best way to impute the missing values, why would that be?

In my scenario, I have data like this

Brand|Value
A|2,
A|NaN,
A|4,
B|8,
B|NaN,
B|10,
C|9,
C|11

if using mean imputation the data would be

Brand|Value
A|2,
A|7.3,
A|4,
B|8,
B|7.3,
B|10,
C|9,
C|11

which does make sense for brand B to be 7.3 but doesn't make sense if brand A 7.3 because the value of Brand A has its tendency somewhere around 2 and 8
is there any other way to fill the missing values based on the Brand?

This is an example of data with only 2 features, with 1 feature that may has pattern for missing values, what if there are like 20 features, and there would be more than one features that may have pattern to better define the missing values.

How to apply this in Python?


texts: Derivative of the function of random variable

texts:
Suppose we have a function $\phi(X)$  of random variable $X$. Suppose both of $\phi(X)$ and $X$ are random variables. If $\phi$ is differentiable, how to calculate the derivative of $\phi(X)$ w.r.t. $X$? for example, if $\phi(X)=stand(X)=\frac{X-E(X)}{std(X)}$ (suppose $E(X)$ and $std(X)$ exist), how to calculate $\frac{d\phi(X)}{dX}$? I knew that if we assume the approx. distribution of $X$ exists using LLN or CLT, we can just using mean and std of Gaussian distribution. But how to get a general form of the derivative $d(stand(X))/dX$, $d(EX)/X$ and $d(std(X))/dX$? how to define the limitation and rules of derivatives in this case?
Background of this problem:
Suppose I trained a neural network with standardisation of the data following $(X-EX)/std(X)$. The input is $x(t)$ and output is $y(t)$. How can I calculate the sensitivity of this trained network (basically the $dy(t)/dx(t)$)? if I multiply $(1+\epsilon)$ to $x(t)$, the $1+\epsilon$ will disappear after standardisation. So what should I do? I tried to add a perturbation to the standardized input $X'$ which is $(X-mean(X))/(std(X))$. But after the perturbation what I got is $dy(t)/d(standard(x(t)))$ rather than $dy/dx$. So I also need the ratio of $dx/d(standard(x))$ where $standard(x)$ is $(x(t) - mean(x(t)))/std(x(t))$.
We can assume $x(t)$ and $y(t)$ is a $N$-order Markov process.
So how to calculate the $dx/d(standard(x))$?


texts:
Where Can I find a Bootstrap Template for a Dash app?

Are there implemented examples? I cannot find nothing about this.

I am building a Dashboard with Dash/Plotly but I want to improve the style of the application.


texts: Graphical results of Q-Learning: is improvement possible by parameter tweaking?

texts:
From left to right:

Maximum Q value for action selection (averaged)
Train error (averaged)
Reward from environment (averaged)

I run double Q-learning.

A behavioral policy is ε-greedy, ε constant decaying to 0.1.

I used learning rate decay as follows:

learning.rate = start_learn_rate / log(counter + 1), #decaying learning rate

optimizer = 'sgd'
start_learn_rate = 0.001

I observe degradation of the average reward after it hits its maximum, with a clear oscillation behaviour from high to low.

Is this chart typical to TD (Q) learning routine?

Could you advise on what parameters could be changed to stabilize the result?

Why the models behave like that? Don't they ought to expose a stable behaviour when Q is stabilized?

Environment, agent actions, and reward structure

My task is quite specific. I model sequences of actions applied to a process generating time series for the purpose of making something like predictions.

Observe a sine function (timeseries). Agent gets information about lagged values (and some engineered features) of that process from it's PAST. Another part of environment consists of a series of PAST actions (i.e., past 10 actions).

Actions are:

buy (green)
hold (black)
sell (red)

A return (basis for the reward) is generated when conditions meet:

buy -> sell (close buy position and immediately enter sell)
sell -> buy (close sell position and immediately enter buy)
buy -> hold (close buy and wait for next action)
sell -> hold (close sell and wait for next action)

The return is maximum (equals 2) when the agent keeps the series of one-type actions starting at hill and down to valley, and vice versa for other sign positions.

At this chart you could see that in the bottom plot the agent learned to guess the direction right in most cases, but the return is not maximized to the value of 2.

My goal is not just to observe the leraning statistics, but to find an optimal policy to be applied to an out of sample data, with further exploration and policy refinement.

My intuitive guess is that the learning rate is too high when theagent is close to optimality and is forced to escape the region:

UPDATE

I get almost perfect solution of the optimal policy in my toy task, however, the neural network is not quite stable, and I have to manually choose the moment to stop learning.

This is how the system evolved during training:

Again the optimality seems to be reached at some point (with later fluctuations).

The learning behaviour of the agent:

Note that after aprx. 30 000 iterations the anent starts to make clear sequences of actions leading to maximized reward.

At test (with greedy action selection) the behaviour is that:

And test trades are also pretty clear (not perfect though):

There are 2 points I want to improve:

Stopping criteria for learning
Stability of max-Q and averaged return (and sum of returns) during
test.


texts:
The explanation from the official doc2vec paper "Distributed Representations of Sentences and Documents" for PV-DBOW is as follows:

  Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector.

According to the paper the word vectors are not stored and PV-DBOW is said to work similar to skip gram in word2vec.

Skip-gram is explained in word2vec Parameter Learning. In the skip gram model the word vectors are mapped to the hidden layer. The matrix that performs this mapping is updated during the training. In PV-DBOW the dimension of the hidden layer should be the dimension of one paragraph vector. When I want to multiply the word vector of a sampled example with the paragraph vector they should have the same size. The original representation of a word is of size (vocabulary size x 1). Which mapping is performed to get the right size (paragraph dimension x 1) in the hidden layer. And how is this mapping performed when the word vectors are not stored? I assume that word and paragraph representation should have the same size in the hidden layer because of equation 26 in 2


texts: GPU computing: how much VRAM do I need for mini batch gradient descent?

texts:
I want to do some GPU computing with an NVIDIA card, and am deciding between having a GTX 960 with a 2GB or 4GB ram. Which one should I take? How much difference would these make in terms of the batch size I can use for mini batch gradient descent? Would this difference be significant?

Thank you for answers.


texts: how to train custom word2vec embeddings to find related articles?

texts:
I am beginner in machine learning. My project is to make search engine based on AI which shows related articles when  we search on website. For this i decided to train my own embedding.
I found two methods for this:

One is to train network to find next word( i.e inputs=[the quick,the quick brown,the quick brown fox] and outputs=[brown, fox,lazy]
Other method is to train with nearest words(i.e [brown,fox],[brown,quick],[brown,quick]).

Which method should i use and after training how should i convert the sentence to a single vector to apply cosine similarity means sentence- the quick brown fox will return 4 vectors how should i convert it to feed for cosine similarity(which takes only one vector) with another sentence.


texts: Forecasting Consumption for Multiple Products for Multiple Regions

texts:
Came across a very interesting Real-World Time Series Forecast Problem.

Can you please help me understand the right track to resolve the below Time Series problem:

Input Data Sample:

and we want to predict the consumption of Product at each location by type for next 6 months.

I have tried doing it by building a simple ARIMA model to get monthly values and then multiplied by the factors calculated for consumption by Product, Type etc.

But can this problem be solved by any other Time Series Technique?
Can we use SARIMAX? If yes, then how can we deal with exogenous regressors being Categorical Variables?

Few observations in data:

Tea might have been consumed by a country in 2017 have now been replaced by Tea2 (Tea type 2) for next years, so how does time series maintain the continuity in data.

Any leads and solution will be highly appreciated.


texts: high precision and recall but low cross validated accuracy

texts:
I'm using a random forest classifier (scikit learn). When I do a 5-fold cross validation, the average accuracy score is around 70%. However, when I look at precision_score and recall_score, they are both 1. Is that possible? If it's wrong, what could be the problem? Here is the code:

clf = RandomForestClassifier(n_estimators=100)
clf.fit(X, Y.values.ravel())
scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")
Y_pred = clf.predict(X)
precision = precision_score(Y, Y_pred)
recall = recall_score(Y, Y_pred)


texts: Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

texts:
In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link:
https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec) and I was wondering why instead of transforming single words, they transform entire sentences. Isn't word2vec purpose to embed single words into a vector space? Why they embed entire sentences? How can one properly train word2vec and then applying to single words in Spark?


texts: How to model user choice probability: binary model vs multi class model

texts:
Let's say Morpheus has multiple users to offer colored pills(from an infinite set of colored pills), there are in total 3 unique colored pills(red, blue, green) Morpheus can offer. The trick is, Morpheus can offer only one pill to a user and the user has a choice to either take the pill or deny it. (Also, user's decisions are independent of each other)
Now Morpheus wants to be smart about his offer and wants to model the user such that the user selects the pill he is offering.
The users are moody and there is some uncertainty that they would randomly take a choice.
Rejection can be because of multiple unknown reasons such as I didn't like the color of the pill, I will choose the pill later, I want to understand more about this pill, Show me other pills before I decide
Now there are two ways I can think of modeling this:

Treating this as binary classification
Treating this as multi-class classification

When I treat this as binary classification, I pass pill color as feature with other user features to the model, and my output is the probability of user taking or rejecting a pill given the pill color.
Morpheus can then offer the pill color with the highest probability. This will use both Accept and Reject decisions of a user while modeling, but there is some uncertainty and the same type of users can accept or reject randomly.
When I treat this as a multi-class classification, where I try to predict the pill color itself. I would not use the rejected case in my training and would only consider cases when the user chose something.
In this way, I can reduce uncertainty in this case but would have to completely ignore rejected cases. Morpheus then can either use softmax or sigmoid for each class and take argmax to get the best choice to offer.
I am not sure if there are other ways to model this problem, but out of these two which can be a better way?


texts: Should the different layers of deep learning models have same size or they should be changed based on a rule

texts:
I see a lot of people varying the width of each layer in a deep neural network. ie. Input->50->100->150->Output. I'm curious what, if any, are the advantages of this structure over static layer widths ie. Input->50->50->50->Output.

Given the following three model structures what would be the real-world implications and differences between them and how they interpret data?

Example 1:

  Input -> 100 -> 100 -> 100 -> Output

Example 2:

  Input -> 50 -> 100 -> 150 -> Output

Example 3:

  Input -> 150 -> 100 -> 50 -> Output


texts: Does CART algorithm takes into account in the order of the set of attributes?

texts: when using matlab command 'fitctree' for classification purpose,  and I change the order of the attributes I do not find the same Tree and thus the same classificaiton error? why? CART algorithm does take account on the attributes firstly introduced ?


texts: What does it mean when the coefficient of the squared term is negative in regression?

texts:
I am reading a research paper which models a regression model where the returns are regressed on the number of ad exposures.

the equation looks something like this:

$Returns = beta_1*nExp + beta_2*nExp^2$

nExp: The number of times the user looks at the ad.

Returns: The revenue from the user

The result is that: $beta_1$ is positive (and statistically significant), which means that the ad exposures has a $+ve$ relation with the revenue.

However, $beta_2$ is negative. And the research paper says:

  term for the square of the number of exposures, which has a negative
  coefficient suggesting diminishing returns to ad exposure.

What does it really mean? What does it mean when the term's coefficient is $+ve$ but the coefficient of it's squared term in the equation is $-ve$?


texts: Scale (frequency) invariant cross-correlation

texts:
I am looking for cross-correlation of two 'signals' that would be invariant of scale(frequency, or sampling).

To give you example what I mean:

Standard cross-correlation of sin(x) and cos(x) find match after pi/2, because sin and cos are the same functions, just phased.

I am looking for some algorithm (I searched but didnt find anything) that would find perfect correlation between sin(x) and sin(2*x) or sin(x) and cos(2x). Same functions, but different frequency scale (or call it sampling).

Image instead of words:

Case 1 ( search sequence in signal, sequence can be of different scale/sampling )

Case 2 (Same curve, different scale)

Any idea? (I am looking for something very fast as far as computation is concerned)
Thanks


texts:
In the original Skipgram/CBOW both context word and target word are represented as one-hot encoding.

Does fasttext also use one-hot encoding for each subword when training the skip-gram/CBOW model (so the length of the one-hot encoding vector is |Vocab| + |all subwords|)? If they use it, do they use it in both context and target words?


texts: How to reduce pipeline footprint in scikit-learn?

texts:
I had a post on stackoverflow. As it is related to scikit-learn, I am hoping that I can obtain some assistance from data scientists in this forum. https://stackoverflow.com/questions/38640815/python-running-out-of-memory

In general, however, I would like to seek opinions on the reduction of memory use when one uses scikit-learn, as this can be a daily problem to deal with as a data scientist. Thank you very much.


texts: Suggestion on tag clustering visualization

texts:
I have a database of tags given by users to the product. For example

user; product; tag
1; A; Tag1
1; A; Tag2
2; A; Tag1
2; B; Tag1
..
..

I am trying to cluster tags which are given together to any product. At the end I want to visualize them like PCA plot where I can see clusters of tags which are 'closer' (tendency of user to assign these tags together) to each other. So far I can think of applying t-SNE or simply PCA to get some kind of clustering with available tags.

For this purpose, I made frequency matrix like following

first_tag; second_tag; occurrence
Tag1; Tag2; 2393
Tag1; Tag3; 38
Tag2; Tag3; 8393
..
..

I am stuck here. I don't know how to proceed with clustering. I simply tried visualizing it with networkx library of python where I made edges with 'first_tag' and 'second_tag' and 'occurrence' as its weight. But it was futile exercise, I couldn't get anything out of it. I thought of using sklearn.decomposition PCA but I am struggling to convert this data-set into proper matrix which can be fed to such algorithms. I can always make n x n matrix by making rows and columns equal to number of tags but it was causing very slow and sometimes out of memory errors. Any other elegant solution for this?


texts:
New to stats and R and was wondering if anyone can help me understand the output of the following code

pwr.t2n.test(n1=7561, n2=11531, d=NULL, sig.level = 0.05, power = 0.8)

output:

t test power calculation

         n1 = 7561
         n2 = 11531
          d = 0.04148028
  sig.level = 0.05
      power = 0.8
alternative = two.sided

When I was going through the documentation it says d value of 0.2, 0.4, 0.8 corresponds to small, medium and large.. However, I get d as 0.04 when I run the above? What does it mean? Does it mean that I will be able to see and effect size of 0.0414 when I perform my tests?

On the other hand when I run the below code i.e. setting d = 0.8

pwr.t2n.test(n1=7561, n2=11531, d=0.8, sig.level = 0.05)

The output of power is 1 meaning it will be able to find the Type II error
    t test power calculation

         n1 = 7561
         n2 = 11531
          d = 0.8
  sig.level = 0.05
      power = 1
alternative = two.sided

I am getting confused as I thought I will not be able to get a power of 1 for such high d value as my previous line gave a small value for d when power was 0.8

Can someone pls. explain and clear my understanding.

Thanks!


texts: What is difference between a Data Scientist and a Data Analyst?

texts:
https://www.datacamp.com/community/tutorials/learn-data-science-infographic
https://www.datacamp.com/community/blog/data-engineering-vs-data-science-infographic

These links contain almost everything but not the difference between data science and data analytics.

Is data analytics a part of the data science workflow? Is data analytics a subset of data science?


texts: Dynamic Regression Prediction with Flask(or sth. similar)

texts:
as a self learner Data Scientist I am working on deploying my model in order to get regular predictions. And I need your guidance about deployment section, in which I know nothing about and there is nobody around to guide me. There is not any line of code etc, so I am just asking for your theoretical opinions.
My process will go on like this:
1- Every 2 hours, system will run a query and save a csv file to a specific location. It will overwrite of course. And this task is on me, I am not asking about it. This csv file is the prediction dataset actually, which will be retrieved by ML model periodically to make predictions on it.
2- I have to make predictions, lets say, every 2 hour. My service should retrieve this csv file automatically from the local path I will define.
3- Then it should make predictions based on the model I will define for it in pickle form.
4- Finally it should download the results or final dataframe in csv format again into the same path as predictions.csv for example. It should overwrite of course.
For this process, I am making my research on Flask Web Services. But I must ask you, is this process possible with flask applications? As you can see, there is not any manual trigger in this process, everything is automatic, no one needs to visit a web page or hit a run button etc. If you can suggest me any resources or web pages that you think may be beneficial for me, it is highly welcome.
Have a nice day


texts: Tokenisation with Spacy - how to get tokens to the left/right of token

texts:
I am using Spacy for text tokenisation and getting stuck with it:

import spacy
nlp = spacy.load("en_core_web_sm")
mytext = "This is some sentence that spacy will not appreciate"
doc = nlp(mytext)

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_, token.shape_, token.is_alpha, token.is_stop)

returns something that seems to me to say that tokenisation was succesful:

This this DET DT nsubj Xxxx True False
is be VERB VBZ ROOT xx True True
some some DET DT det xxxx True True
sentence sentence NOUN NN attr xxxx True False
that that ADP IN mark xxxx True True
spacy spacy NOUN NN nsubj xxxx True False
will will VERB MD aux xxxx True True
not not ADV RB neg xxx True True
appreciate appreciate VERB VB ccomp xxxx True False

but on the other hand

[token.text for token in doc[2].lefts]

returns an empty list. Is there a bug in lefts/rights?

Beginner at natural language processing, hope I am not falling into a conceptual trap. Using Spacy v'2.0.4'.


texts: SVM behavior when regularization parameter equals 0

texts:
I read on this Wikipedia page the following about soft-margin SVM:

  "The parameter $λ$ determines the trade-off between increasing the
  margin size and ensuring that the $x_i$ lie on the correct side of the
  margin. Thus, for sufficiently small values of $λ$, the second term in
  the loss function will become negligible, hence, it will behave
  similar to the hard-margin SVM, if the input data are linearly
  classifiable, but will still learn if a classification rule is viable
  or not."

I can't understand why in the case that λ=0 the algorithm will behave like hard-margin SVM. If λ=0, it seems to me that the algorithm won't have any reason to perform any optimization on the margin. Doesn't it just become a perceptron in that case, since the algorithm only "cares" about classifying all the train data correctly, while not reaching any optimal solution regarding the margin?

I'll appreciate a clarification about the issue, please.


texts: I am working on embedding, i would like to know which if Mean Squared error (MSE) is better to make comparison between two embedding vector than Co-sinus similarity. In which situation use one or another


texts: Why don't we want Autoencoders to perfectly represent their training data?

texts:
From Ian Goodfellow's Deep Learning Book:

  If an autoencoder succeeds in simply learning to set g(f(x)) = x everywhere, then it is not especially useful. Instead, autoencoders are designed to be unable to learn to copy perfectly

I don't understand this part. g is the decoder, and f is the encoder. Why is it undesirable for the encoder and decoder to perfectly represent the input data x?

Another way to frame this question is - why do autoencoders require regularization? I understand in predictive machine learning, we regularize the model so that it can generalize beyond the training data.

However, with a sufficiently massive training set (as is common in Deep Learning), there should not be a need for regularization. To me, it seems desirable to learn g(f(x)) = x everywhere, and I don't understand why the author says otherwise.


texts:
I've been working on neural network for a while and I built simple network from scratch with python but before using TensorFlow, I would like to have a complete understanding of it.

Here is my question :

Lets say you have 3 layers you have 3 weights to update :

1) --> the weight between the outputlayer and the hiddenLayer2

2) -->  the weight between the hiddenLayer2 and the hiddenLayer1

3) -->  the weight between the hiddenLayer1 and the inputLayer

For the 1) the calculation is quite simple we got :

weight_3 += LEARNING_RATE * ((2*(target - output)) * sigmoid'(output) * layer2)

For the 2) the calculation is more complicated  and we got :

weight_2 += LEARNING_RATE * ((2*(target - output)) * sigmoid'(output) * weight_3) * sigmoid'(hiddenLayer2)

I need help for the 3rd part, I tried to calculate and find on internet but not a lot of people uses 2 hidden layer when they work from scratch.

I also tried to resolve the chain rule but its too long and I can't resolve.

Does someone know the formula to get the weight between the hiddenLayer1 and the inputLayer ?

Thank you so much in advance


texts: Mathematical bias and weight vs machine learning bias and weight

texts:
I am a little confused about the term Bias and Weight with respect to machine learning.
Say we want to predict the heights of people whose weights are given. So plot weights to x-axis and height to yaxis. To find out the linear relationship between height and weight we draw a straight line that shows the relationship between height and weight.

Using the equation for a line, you could write down this relationship
$y= mx+b ...i)$ more specifically in the machine learning terminology it could be $y= b+ w_1x_1...ii)$
So here b is the bias as per machine learning. However, b is the y-intercept as per mathematics. By defination b can be defined in machine learning A value indicating how far apart the average of predictions is from the average of labels in the dataset. Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).
Now another confusion if this is the case then what is loss? As per defination loss is A measure of how far a model's predictions are from its label. Then what is the difference between loss and bias?
Now, for weight, here, weight(m)means slope as per equation i). Mathematically slope can be define as
$m = \frac{rise}{run} =\frac{y_2 - y_1}{x_2 - x_1}$
However, weight$(w_1)$ can be defined in machine learning as A coefficient for a feature in a linear model. So my confusion is that is the procedure of finding weight is same as finding slope in mathematics?


texts: Jupyter Notebook not importing pandas module

texts:
I am running Jupyter on a server on a virtual environment. I then tunnel my connection so I can access Jupyter on my browser.

When I SSH into the server, I can use the Panda module in both Ipython and Python3.

I ran this code in Ipython

 import pandas as pd

In [2]: print(pd.__file__)
/home/ubuntu/.local/lib/python3.6/site-packages/pandas/__init__.py

Then I tried adding it to my path in Jupyter with the code below, still no luck.

import os
os.getcwd()
import sys
sys.path.append('/home/ubuntu/.local/lib/python3.6/site-packages/pandas/__init__.py')

import pandas as pd

I used these instructions to setup the Virtual Environment https://www.digitalocean.com/community/tutorials/how-to-set-up-jupyter-notebook-with-python-3-on-ubuntu-18-04

Thank you very much in advance, this will help me with my school studies a lot.


texts: What is the range of values of the expected percentile ranking?

texts:
I'm currently reading

  Hu, Koren, Volinsky: Collaborative Filtering for Implicit Feedback Datasets

One thing that confuses me is the "expected percentile ranking", an function the authors define to evaluate the goodness of their recommendations. They define it in the Evaluation methodology on page 6 as:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}$$

where $u$ is a user, $i$ is an item (e.g. a TV show), $r_{ui} \in [0, \infty)$ is the amount how much user $u$ did watch show $i$. $\text{rank}_{ui} \in [0, 1]$ is the percentage rank of item $i$ for user $u$. For example, it is 0 if for user $u$ the item $i$ has the highest $r$ value and 1 if the item $i$ for user $u$ has the lowest $r$ value.

I'm not super sure if I understood it correctly.

The authors write that lower values of $\overline{\text{rank}}$ are more desirable and for random predictions would lead to an expected value of $\overline{\text{rank}}$ of 0.5.

Examples

Assume there is only one item. In this case $\text{rank} = 0$. Makes sense, as there cannot be any predictions.
Assume there is only one user and two items with $r_{1,1} = 1$ and $r_{1,2} = 2$. Then:

$$\overline{\text{rank}} = \frac{1 \cdot \text{rank}_{1, 1} + 2 \cdot \text{rank}_{1, 2}}{1+2}$$

This means $\overline{\text{rank}} \in \{2/3, 1/3\}$.

If there is only a single user and all $|I|$ values of $r_{ui}$ are the same, then $\overline{\text{rank}} = \sum_{ui} \text{rank}_{ui} = \frac{|I|}{2}$

Questions

Is my understanding of the metric correct? Especially my last example and the statement by the authors that $\overline{\text{rank}} \geq 50\%$ indicated an algorithm is no better than random seem off.
What is $t$?


texts:
Suppose that I am trying to build a random forest by subsampling the data and choosing a single feature per tree randomly. For example, suppose there is some dataset,
$D = \{(x_{1},y_{1}), ......(x_{N},y_{N})\}$ where we have that $x_{i} \in \mathbb{R}^{D}$ and $y_{i} \in \mathbb{R}$ for $ i= 1,....n.$. We are trying to construct the tree as follows:

First we randomly sample one feature index $j \in \{1,....D\}$
Then we draw some sample of the data $\tilde D_{k}$ of size $M \le N$ with replacement. These datapoints will then have indices $k = k_{1},....,k_{M}$
Keep only the $j^{th}$ feature of the M samples:
$\tilde D^{j}_{k} = {(\tilde x^{(j)}_{(k_{1})},y_{(k_{1})}),......(\tilde x^{(j)}_{(k_{M})},y_{(k_{M})})}$
Then we build a decision tree on $\tilde D_{k}^{(j)}$.
Then average R of these random trees to create a random forest

We were asked for which class of conditional distributions $Y|X=x$ are very random forests unbiased? I am wondering what is meant by the "class" of conditional distribution? Could someone shed some light on this please?
Also, how does the bias and variance of this RF vary with a traditional RF? I assume that I will need to look at the generalization bounds? I am not sure. Could someone please shed some light on this?


texts: Python Pandas Plot: Using two columns to generate one y-axis

texts:
I have a pandas dataframe with 3 columns: condition, trial and time. I would like to create a scatter plot where the x-axis represents the time column and the y-axis represents the condition. The special part is that each condition has n trials. So in the dataframe there are n rows with the same condition marked with different trial indices (1-n) in the trial column. In the plot I would like to display each trial in a separate row (y-position), i.e. that each condition on the y-axis should be subdivided into n trials.
Currently, I plot the dataframe as follows:
df.plot(x = 'Time', y = 'Condition', kind = 'scatter', s = 1)
However, this plots all trials for each condition on the same horizontal line. Do you have any suggestions how to incorporate another column into the y-axis? I would be very grateful!
Greetings and thanks,
sonicycle


texts: How to sort two dimensional array in Java?

texts:
I have a two dimensional double array. I need to sort my based column 1 in descending order. How can I sort my two dimensional array in descending order.

Sample data and code are the following:

package mypro.com;

public class SortDoubleArrary {
    public static void main(String agrs[]) throws Exception{

        double[][] testdatset=new double[5][2];

        testdatset[0][0]=20.0;
        testdatset[0][1]=0.20;

        testdatset[1][0]=100.0;
        testdatset[1][1]=0.50;

        testdatset[2][0]=10.0;
        testdatset[2][1]=0.95;

        testdatset[3][0]=220.0;
        testdatset[3][1]=0.35;

        testdatset[4][0]=140.0;
        testdatset[4][1]=0.10;
    }
}

Expected output looks like the following

 10.0    0.95
 100.0   0.5
 220.0   0.35
 20.0    0.2
 140.0   0.1


texts: How to divide MNIST training dataset into subparts?

texts:
Background: I am trying to train different sub-samples of MNIST dataset(1 model for 10,000 , another model for 20,000 samples, and so on until a model for all 60,000 training samples) using Caffe.

Question: How do I sub-split the training dataset of 60,000 samples keeping the format of the training files same(intact) as originally published on http://yann.lecun.com/exdb/mnist/


texts: h2o, different stopping metric leads to different optimal for hyperparameters

texts:
I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package

# create hyperparameter grid
hyper_params = list(ntrees = c(10,20,50,100,200,500),
                    max_depth = c(5,10,20,30),
                    min_rows = c(5,10,20,30),
                    learn_rate = c(0.01,0.05,0.08,0.1),
                    balance_classes=c(T,F))

# random subset of hyperparameters
search_criteria = list(strategy = "RandomDiscrete", max_models = 192, seed = 42,
                       stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = "mean_per_class_error")

# run grid
gbm.grid <- h2o.grid("gbm", grid_id = "gbm.grid", x = names(td.train.h2o)[!names(td.train.h2o)%like%"s_sd_segment"], y = "s_sd_segment",
                     seed = 42, distribution = "multinomial",
                     training_frame = td.train.hyper.h2o, nfolds = 3,
                     hyper_params = hyper_params, search_criteria = search_criteria)

# sort results
gbm.sorted.grid <- h2o.getGrid(grid_id = "gbm.grid", sort_by = "mean_per_class_error", decreasing=FALSE)

hyperparameters_gbm <- as.data.table(gbm.sorted.grid@summary_table)

This gives as optimal combination for the hyperparameters

learn_rate  max_depth   min_rows    ntrees
0.08    10  5   200

Then I am trying to do the same but with different stopping_metric. So in the above i use mean_per_class_error, and in the following i use logloss, so i run the following code:

# create hyperparameter grid
hyper_params = list(ntrees = c(10,20,50,100,200,500),
                    max_depth = c(5,10,20,30),
                    min_rows = c(5,10,20,30),
                    learn_rate = c(0.01,0.05,0.08,0.1),
                    balance_classes=c(T,F))

# random subset of hyperparameters
search_criteria = list(strategy = "RandomDiscrete", max_models = 192, seed = 42,
                       stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = "logloss")

# run grid
gbm.grid <- h2o.grid("gbm", grid_id = "gbm.grid", x = names(td.train.h2o)[!names(td.train.h2o)%like%"s_sd_segment"], y = "s_sd_segment",
                     seed = 42, distribution = "multinomial",
                     training_frame = td.train.hyper.h2o, nfolds = 3,
                     hyper_params = hyper_params, search_criteria = search_criteria)

# sort results
gbm.sorted.grid <- h2o.getGrid(grid_id = "gbm.grid", sort_by = "logloss", decreasing=FALSE)

hyperparameters_gbm <- as.data.table(gbm.sorted.grid@summary_table)

which gives as optimal combination for hyperparameters

learn_rate  max_depth   min_rows    ntrees
0.1 20  5   500

I know that i use as argument strategy = "RandomDiscrete", but still for instance the optimal combination for gbm using stopping_metric = "mean_per_class_error" is the "50th optimal combination" for the gbm using stopping_metric = "logloss", and the "2nd optimal combination" for gbm using stopping_metric = "logloss", is the "14th optimal combination" for gbm using
stopping_metric = "mean_per_class_error"

Why could that happen ?


texts: What is "lag" in time series forecasting?

texts:
I'm studying machine learning (e.i. time series analysis). I encountered an Azure tutorial, Retail Forecasting.

https://gallery.azure.ai/Experiment/Retail-Forecasting-Step-2-of-6-train-time-series-models-3

Here, they introduce new features called "lag", which I don't understand what it means. In 3.4, they said "The module selects the best lag of this index based on maximum correlation." What does it mean? Could you please let me know some references to learn background knowledge?


texts: Does someone have a low level inplemenation of tensorflow's ConvLSTM2D

texts:
Right now I am using the ConvLSTM2D module of tensorflow and it works. But now I would like to feed an additional input between the convolutional stage and the lstm module. Therefore I would need a low-level implementation of this?

Does someone know, how to do that? can I do it with Conv2 and LSTM? But LSTM is takes no 2D input?

Thanks


texts: Non-mutually exclusive classification sum of probabilities

texts:
So I have the following problem: I realized (while writing my master thesis) that I am still not sure/have vague descriptions of some of the machine learning principles. I already asked one question regard definitions that can be found here.

Now I stumbled over another definition Problem.
Here is an excerpt from my thesis (this is in particular about neural-network classification):

  If the classes are mutually exclusive (i.e. if a sample $x^{j} = C_{0}$, $x^{j} \neq C_{i}\setminus~C_{0}$ ), the probabilities of all classes add up to one like \begin{equation}
\sum_{i} P(x^{j}=C_{i}) = 1.
\end{equation}
  In this case the best practice is to use a softmax activation function for the output neurons.
  If the classes are not mutually exclusive it would suffice to use a sigmoid output activation function, as the sigmoid function gets independent probabilities for each class \begin{equation}
\sum_{i} P(x^{j}=C_{i}) \geq 1.
\end{equation}

I already found the following link regarding this topic.
However I know that in practise if you don't use softmax activation function in your output layer, the value can be larger than 1 but can a probability be larger 1? Isn't that against its definition?

Is a non-mutual classification really a common case? Can somebody may be linking some cases (paper?) were they needed non-mutual classification?


texts: What model should I choose for Jobs recommender system?

texts:
Now, I have enormous number of resumes of someone who search for next jobs.
There are infomation about how old they are, what schools they graduated,and
what jobs they experienced.etc...

I want to make a recommender system using those data.
In the system, if I get new resume, I want to recommend certain jobs for him.

I have some idea what models will I use, for example,
collaborative filtering, recurrent neural network, word2vec etc...

But I couldn't find best method for this problem.
If someone Knows good idea, please tell me that.

This is an example of one data,

[age,education,skills,past_job1,job1's occupational category, past_job2,job2's occupational category,
...,]

All data is stored in string.


texts: Does resizing the image size, reduce the quality of the images in CNN?

texts:
I am doing a project for cancer recognition. My data set has hunderds of images but not of equal size. I wanna resize them to the size of the smallest image. But I am wondering do you think using the cv2.resize will effect the quality of images and so the result of  my work? If yes what I should do. If no please let me know.
Width disturbuation

higth disturbuation


texts: Target data values are not evenly distributed

texts:
Data nature:

I have features with 10 numeric type, and other 10 categorical, with a lot of  values, at the end, using one-hot encoding I got a matrix of 600 columns. My problem is with accuracy which is 0.7, knowing that other peers got more that 0.9.

Problem:

Target data is binary, and is not evenly distributed at all. Trying blindly after pre-processing from sklearn.linear_model import LogisticRegression and sklearn.svm scored using roc_auc_score:  .7 and .75.

Back to basics, I run this

train['cible'].value_counts() / train['cible'].count()

and got

1    0.970791
0    0.029209
Name: cible, dtype: float64

Quite interesting I think, but how can I improve accuracy. Any hints ?

Note: I will edit and add False Positive Rate and True Positive Rate as I lost output, after scaling, missing data imputation and retraining the model which takes couple of hours.


texts: Publicly available social network datasets/APIs

texts:
As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside with a link to the dataset/API, characteristics of the data available were added. Such information should be, and is not limited to:

the name of the social network;
what kind of user information it provides (posts, profile, friendship network, ...);
whether it allows for crawling its contents via an API (and rate: 10/min, 1k/month, ...);
whether it simply provides a snapshot of the whole dataset.

Any suggestions and further characteristics to be added are very welcome.


texts: Word taxonomies for Facebook likes categories

texts:
I query the Facebook graph API to get some users' likes, which come with a "category" field, which can be, for instance, Italian Restaurant or Health & Wellness website and so on.

I need to draw a profile of the user so I was thinking of retrieving the top level root in a taxonomy graph of the aforementioned categories in such a way that the first example falls into something like Food and the second one into something like Health. This way I'd know, by a majority rule, who is a sporty person, who likes fashion etc.

Is there a free public API I could use to get word taxonomies? I found Alchemy, an IBM project which is pretty cool but unfortunately limited to 1000 API requests per day and I'd need way more than that given the big amount of users and categories I have.

I know I'm probably asking for too much (a free and unlimited tool!) so the question might become "do you have better ideas about how to solve this"?