texts:
I tried to make simple neural network layers as the following, including forward and backward propagation. Here is my reference.
Firstly I assume an one layer FC:
$Y = X \cdot W + B$
X is input, which size is (N, D), N=batch-size, D=features; W is weight, which size is (D, M), D=features, M=next-layer features
then I stack it as two layer FC:
$Y_1 = X_1 \cdot W_1 + B_1$
$Y_2 = Y_1 \cdot W_2 + B_2$
Backpropagation:
Gradient w.r.t. $W_1$:
$\frac{\partial Y_2}{\partial W_1} = \frac{\partial Y_2}{\partial Y_1} \frac{\partial Y_1}{\partial W_1} = X_1^T \cdot W_2^T$ ( I move $w_2$ to the right side to follow the order in forward prop; also, I perform transpose to sum the features along batch axis.)
Gradient w.r.t. $W_2$:
$\frac{\partial Y_2}{\partial W_2} = Y_1^T$
The Problem:
I used the following snippet to verify above equations, but the results from Tensorflow are not the same as my assumption, specifically the gradients of $W_1$ and $W_2$.
Which part am I wrong?
    import tensorflow as tf
    tf.executing_eagerly()
    tf_batch_x = tf.constant(x_batch_array)
    l1_tf_batch_weight = tf.Variable(l1_weight_array)
    l1_tf_batch_bias = tf.Variable(l1_bias_batch_array)
    l2_tf_batch_weight = tf.Variable(l2_weight_array)
    l2_tf_batch_bias = tf.Variable(l2_bias_batch_array)
    with tf.GradientTape(persistent=True) as tape:
        tf_out = tf.matmul(
            tf_batch_x,
            l1_tf_batch_weight) + l1_tf_batch_bias

        #tf_out = tf.nn.relu(tf_out)

        tf_out = tf.matmul(
            tf_out,
            l2_tf_batch_weight) + l2_tf_batch_bias
        #tf_out = tf.nn.relu(tf_out)
    tf_grad = tape.gradient(tf_out, [l1_tf_batch_weight, l1_tf_batch_bias, l2_tf_batch_weight, l2_tf_batch_bias])
    print("tf_out", tf_out)
    print("tf_grad", tf_grad)


texts: Creating custom datagenerator in Keras with variable input size

texts:
I am currently working on an image classification problem. Each label that I have is determined from 4 grayscale images of variable size (ranging from 30px to 300px in height and width). Hence, I imagine my input shape to be something like (None, 4, None, None, 1). The first axis will be the batch size, and the third and fourth axis being the height and width respectively.

I am facing difficulty with creating a custom data generator for my use case. I am unable to resize the images as I need to retain the size of certain feature in my images. I understand that padding is a viable solution but it might lead to majority of the image being black pixels due to the large difference of sizes between each set of 4 images. Is there any solution other than padding?


texts: Modelling feature interactions in LSTM network

texts:
I created a binary model from 28 features, each sequence is 10 samples long. I tried these two models:
def create_model(lstm_size = 16, mask_value=-9999, num_lstm=3, do=0.5, model_interactions_lstm=True):
    inputs = [keras.Input((window_size, 1)) for x in feature_names]
    mask = [layers.Masking(mask_value=mask_value)(x) for x in inputs]
    if not model_interactions_lstm:
        to_concat = [layers.LSTM(lstm_size)(x) for x in mask]
    else:
        to_concat=mask
    concat = layers.Concatenate(axis=-1)(to_concat)
    blstm = layers.Dropout(do)(concat)
    if model_interactions_lstm:
        # Adding blstm
        for i in range(num_lstm):
            blstm = layers.Bidirectional(layers.LSTM(lstm_size,return_sequences = True))(blstm)
            blstm = layers.Dropout(do)(blstm)
        blstm = layers.Bidirectional(layers.LSTM(lstm_size))(blstm)
        blstm = layers.Dropout(do)(blstm)
    # Final layer
    outputs = layers.Dense(1, activation="sigmoid")(blstm)
    #Creating model
    model = keras.Model(inputs, outputs)
    model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001,) ,loss="binary_crossentropy",metrics=['AUC'])
    return model

The first model is created by invoking the function with model_interactions_lstm=False, and the second with model_interactions_lstm=True
Both models are trained with the same 15% validation set (data from one-two subjects), early stopping and reduce LR on plateau. I tried a bunch of learning rates and network sizes for each model. Test set is leave-one-subject out. I have about 78K instances and almost equally distributed target.
The second model allows feature interactions in the LSTM layer and the first does not - it allows interactions in the final layer only.
Theoretically, the second model can mimic the first model.
However, the first model always outperforms the second one. The best performance I got for each model are AUC of 0.88 for the first vs 0.82 for the second.
I am not trying to get to state-of-the art results, just to demonstrate that LSTM are good fit for the problem, however, I do not have explanation for this difference, and this should go into a paper.
Any ideas regarding this result?


texts: Automation of finding a starting point of measurement in a large dataset

texts:
I am looking for a way to automatically find a starting point of rising in my signal in Python. The data are collected with the frequency 10k (0.0001 s each) so the differences between each point are very small, lost in the noise. I found this point (black dot) manually using data analysis software before but I have multiple files and the manual process is not gonna work well. I was trying to think of something to do with derivative (red dots) or rolling variance (green dots) but it's a dead end for me now. Here's how manual point was chosen:

I pick a point that looks to me that is the closest one to rising signal but is still in the middle of noise before rising. Chosing it manually is just my rough estimation but I don't mind being one or two points wrong from the "correct" starting rising point. I will use it to offset my signal so that rising starts more or less at X = 0.
And now I wanted to find it using python. The full signal looks like this:

The derivative:

The rolling variance:

So they're all close to the interest point (black dot) but I don't know what to do with them next. If I change the limits it all looks like this:

Any ideas how to solve my problem? The simple code sample is below (plotting excluded)
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy.optimize import curve_fit
import scipy.signal as sig

#reading dataset
signal = pd.read_csv('dataset.txt', delimiter=' ' )
signal.columns = ['time','current']

#calculating derivative, finding max and min indices of derivative
signal_derivative = np.gradient(signal,axis=0)
signal['derivative'] = pd.DataFrame(signal_derivative[:,1])
index_derivative_max = signal['derivative'].index[signal['derivative'] == signal['derivative'].max()]
index_derivative_min = signal['derivative'].index[signal['derivative'] == signal['derivative'].min()]

#calculating rolling variance, range 50 points, finding indices of peaks
signal['rolling_var'] = signal['current'].rolling(window=50,center=False).std()
index_rolling_max = signal['rolling_var'].index[signal['rolling_var'] == signal['rolling_var'].max()]
index_rolling_2nd_max = signal['rolling_var'].index[signal['rolling_var'] == signal['rolling_var'][:100000].max()]


texts: What is the purpose of a 1x1 convolutional layer?

texts:
SqueezeNet uses 1x1 convolutions. I try to understand it in this simple example: if the input is one MNIST digit, i.e. of shape 1x28x28x1 (I use Batch x Height x Width x Channel).

Then applying a Conv2D(16, kernel_size=(1,1)) produces an output of size 1x28x28x16 in which I think each channel 1x28x28xi (i in 1..16) is just the multiplication of the input layer by a constant number. Is that right?

More specifically: Output[channel i][x,y] = InputLayer[x,y] * alpha_i for x,y in 1..28, where alpha_i is a constant for each channel.

Is this correct?

It's like going from 1 channel to 16 identical channels (except that they are multiplied by one global constant per channel).

What is its purpose?

Note: I have already read How are 1x1 convolutions the same as a fully connected layer? and 1x1 Convolution. How does the math work? but here it's slightly different.


texts:
This is my first data-science project and I would love to get some guidance to know how to get started.
My problem is the following:
I want to count objects that are in a picture. This picture has a special characteristic, it's going to be "objects" over a white background.
For example:

seeds
coffee beans
painted black dots
etc.

The input for example would be:

The output:
Objects: 9

I've been doing some research already and I found many places that they recommend the use of an R-CNN to locate and classify my "items" in the picture.
This sounds like a good idea but the problem I see here is that I do not know what I'm going to have in the picture. I know there are going to be "repeated items" of the same type and class but it could be anything.
As I don't know what I'm going to have in the picture I don't know what to use to train my model.
How should I proceed?


texts: Convolutional Network for Text Classification

texts:
I am trying to train a convolutional neural network with Keras at recognizing tags for Stack Exchange questions about cooking.

The i-th question element of my data-set is like this:

id                                                         2
title                    How should I cook bacon in an oven?
content    <p>I've heard of people cooking bacon in an ov...
tags                                 oven cooking-time bacon
Name: 1, dtype: object

I have removed tags with BeautifulSoup and removed punctuation too.
Since questions' content are very big I have decided to focus on titles.
I have used sklearn CountVectorizer to vectorize words in titles. However they were more than 8000 words (excluding stop words). So I decided apply a part of speech tagging and retrieve only Nouns and Gerunds.

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words='english')
titles = dataframes['cooking']['title']
pos_titles = []
for i,title in enumerate(titles):
    pos = []
    pt_titl = nltk.pos_tag(word_tokenize(title))
    for pt in pt_titl:
        if pt[1]=='NN' or pt[1]=='NNS' or pt[1]=='VBG':# or pt[1]=='VBP' or pt[1]=='VBS':
            pos.append(pt[0])
    pos_titles.append(" ".join(pos))

This represents my input vector. I have vectorized tags too and extract dense matrixes for both input and tags.

tags = [" ".join(x) for x in dataframes['cooking']['tags']]
Xd = X.todense()

Y = vectorizer.fit_transform(tags)
Yd = Y.todense()

Split data into train and validation set

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(Xd, Yd, test_size=0.33, random_state=42)

Now I am trying to train a Conv1D network

from keras.models import Sequential
from keras.layers import Dense, Activation,Flatten
from keras.layers import Conv2D, MaxPooling2D,Conv1D, Embedding,GlobalMaxPooling1D,Dropout,MaxPooling1D

model = Sequential()

model.add(Embedding(Xd.shape[1],
                    128,
                    input_length=Xd.shape[1]))
model.add(Conv1D(32,5,activation='relu'))
model.add(MaxPooling1D(100,stride=50))
model.add(Conv1D(32,5,activation='relu'))
model.add(GlobalMaxPooling1D())
model.add(Dense(Yd.shape[1], activation ='softmax'))

model.compile(optimizer='rmsprop',
          loss='categorical_crossentropy',
          metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=32,verbose=1)

But it gets stucked on a very low accuracy and it shows a barely increasing loss along the epochs

Epoch 1/10
10320/10320 [==============================] - 401s - loss: 15.8098 - acc: 0.0604
Epoch 2/10
10320/10320 [==============================] - 339s - loss: 15.5671 - acc: 0.0577
Epoch 3/10
10320/10320 [==============================] - 314s - loss: 15.5509 - acc: 0.0578
Epoch 4/10
10320/10320 [==============================] - 34953s - loss: 15.5493 - acc: 0.0578
Epoch 5/10
10320/10320 [==============================] - 323s - loss: 15.5587 - acc: 0.0578
Epoch 6/10
 6272/10320 [=================>............] - ETA: 133s - loss: 15.6005 - acc: 0.0550


texts: What will be the input_shape of tf.keras.layers.Conv3D be for these inputs

texts:
I have many videos, and each video is made up of 37 images (there are 37 frames in the whole video). And the dimension of each image is (100, 100, 3).... So the shape of my dataset is [num_of_videos, 37, 100, 100, 3]
If I want to pass these videos thorugh the tf.keras.layers.Conv3D()... what will be the input shape for the conv3D be:
Conv3D(32, 3, padding='same', activation='relu', input_shape=[What will the input shape here be])


texts:
My friend was asked this question in an interview for analytics and I cannot figure out the answer so I would like to see how could this data science problem be solved. Here's the problem:

Let's say the keyword "Superman" has a search volume of 250k per
month. Other related search terms can be "Superman Logo" (55k),
"Batman vs Superman" (60k), "Superman returns" (50k) etc. using which
users search for a certain topic in Google in a country (say US).
However, there may be thousands of keywords with the word "Superman"
in it, and we don't always have exact data for many of these keywords
with low volume (less than 1000). In order to find the volume of these
topics we use predictions based on the data that we already have for
high volume keywords.  For example, we may say that the term "Future
Superman" will not be searched by a lot of people and its volume can
be medium (close to 300).
Similarly, terms "Superman image" and "Superman jacket" will have
medium volume and terms that people rarely search will have low volume
like "Long Sleeve Superman Shirt" (100) and then there are terms like
"cheap superman shirt" or "Superman college" which will have very low
volume. Can you think of an approach on how you can classify keywords
into "Medium", "Low" and "Very Low" by using some logic when we only
know the high volume keywords?
Your answer should be descriptive and backed by reason. (Hint- your
approach may be to classify certain words which if present will make
the keyword "low" volume or "very low" volume. Or your approach can be
based on number of words in a keyword, spellings, where exactly a term
appears in a phrase and other common human psychology that works
during a search).


texts: reprocessing steps for images before training classification models

texts:
I have a data set of images for classification task.
I read some articles about image reprocessing (before training CNN models) which summarize in those steps:

scale image values (img / 255.0)

remove noise (using Gaussian blur)

morphology

I'm not sure when to use each step and what is the right order of those steps:

Do we need to remove noise before scaling images ? (or it dosn't matter) ?

I didn't found many articals about morphology step:
2.1 When will we use morphology ? (Is it always right to use this reprocessing step ?
2.2 What is the right order ? use morphology after scaling and removing noise ?


texts:
I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the dataset. An example would look like:

New sentence: "I opened a new mailbox"

Prediction based on dataset:

Sentence                       | Similarity
A dog ate poop                   0%
A mailbox is good                50%
A mailbox was opened by me       80%

I've read that cosine similarity can be used to solve these kinds of issues paired with tf-idf (and RNNs should not bring significant improvements to the basic methods), or also word2vec is used for similar problems. Are those actually viable for use in this specific case, too? Are there any other techniques/algorithms to solve this (preferably with Python and SKLearn, but I'm open to learn about TensorFlow, too)?


texts:
I am digging into finding a solution for background subtraction and one of the requirements is to not loose in quality of input image. Found that there is a specific type of CNN like Super Resolution CNN.

I was looking to find something when specifically this type of CNN is used and what is usually its accuracy. But everything that I find is about its structure.

Can someone explain me what are the pros and cons of SRCNN vs regular CNN?


texts: In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?

texts:
In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices?

I understand that I will not get frequencies by reconstructing the original matrix, but is the non-zeros after reconstruction valid?


texts: I have a retail customer transactional data set with features such as customer ID, product, date, number of products bought, ZIP code of customers, amount of the transaction. There already exists a business rule-based segmentation to segment customers if they are interested in coke or not based on their buying.  I have to do a binary classification to add more customers into the coke segment. Any suggestions on the kind of algorithms and approach that would suit this data set best?


texts: Anomaly score calculation for multidimensional data set

texts:
I'm currently using anomaly score calculation methods which work on single dimensional data sets (Timestamp & Value). I would like to calculate the same single anomaly score for multidimensional data set (Timestamp & Value 1 & Value 2 & ..... & Value N).

What method could compare all the values and calculate anomaly score based upon the relationship between them?


texts:
I am working on a project which aims at determining whether a patient has cervical issues or not, based on a certain movement (for instance, turning the head from left to right and so on).

For each patient, I have one or more sets of coordinates acquired with a VR headset. One problem is, all the patients are assumed healthy and I cannot compare their data with patients who actually have cervical issues.

I am currently working with two coordinates at a time, not the three, and I am considering two approaches: the first is to use approximation (splines...), the second is to use concave hulls. I am a bit more inclined to use hulls and I thought that I could calculate the distance between two hulls A and B as follows: area(A\B)+area(B\A). Note that the curves are parametric.

I have two questions:

Is it possible to "classify" patients using a data set that only consists of healthy patients? Or to find a "descriptor" for them ?
If so, or assuming I can get data on unhealthy patients, what tools can I use to classify the curves? I did not find anybody who worked on a similar problem.


texts:
I am trying to determine a person's emotions from their speech. This immediately rings Machine Learning bells and the first step in any ML problem is getting and processing data. My first question is, how do I know what data to provide a machine learning algorithm? Would I supply the entire sound file annotated with a score of "urgency"? Or would I break the sound file down into pitch and periodicity etc.? I guess what I am asking is, how do I quantify the level of significance of certain elements/dimensions of a piece of data in training a machine learning algorithm?

My second question, is what sort of machine learning algorithm would I use? I am relatively new to ML although I have some experience with TensorFlow, so even just a general concept would be helpful. The machine learning algorithm also needs to be applicable to real-time audio and return results within around 2000ms.

Thanks for the help, please let me know if I should update any component of my question with greater specificity!


texts: How to run a pyspark application in windows 8 command prompt

texts:
I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] as an environment variable and called spark-submit command in the cmd prompt. I believe that it is finding the spark context, but it produces a really big error. Can someone please help me with this issue?

Environment variable path: C:/Users/Name/Spark-1.4;C:/Users/Name/Spark-1.4/bin

After that, in cmd prompt: spark-submit script.py


texts: Similar output from random forest and neural network

texts:
I used a training data set to train both a random forest and a neural network (one hidden layer). Then I compared how both systems perform on a test data set. Interestingly, both turned out to have about the same prediction probabilities. The forest classified 86% of the data correctly into category 0 or 1, the network achieved 85%. As there is 75% of the data in category 0, I am not too happy with the result and hoped for something better.

I then analyzed, which data sets were classified incorrectly. It turned out that there is a large overlap: 86% percent of the data, which was classified incorrectly by the forest, was also classified wrongly by the network. I then compared the probabilities which were attributed to those test samples by the forest and the network. Also the probabilities were almost always comparable so that when the forest was sure that sample A belongs to category 0, then also the network was sure that 0 is the right category. Also, if the probabilities of the forest were around 50% for both categories, also the network was "unsure" which category was correct.

Is such an outcome somehow obvious? Is there an explanation?


texts: Can I turn any binary classification algorithms into multiclass algorithms using softmax and cross-entropy loss?

texts:
Softmax  + cross-entropy loss for multiclass classification is used in ML algorithms such as softmax regression and (last layer of) neural networks. I wonder if this method could turn any binary classification algorithms into a multiclass one? For instance,

If I am using a polynomial function for binary classification, the decision step being 'predict positive if the output of the polynomial is greater than 0, otherwise predict negative', then I could use k of such polynomials$\dagger$ for k-class classification, each polynomial $f_i(X)$ has its own set of parameters to be learned; the objective is then to minimize the KL divergence between the sample distribution of the one-hot label and $e^{f_i(X)}/\sum_{i=1}^{k} e^{f_i(X)}$ or equivalently, cross-entropy.

Now if $f(X)$ is linear, this is exactly softmax regression algorithm, would  it work if $f(X)$ is a polynomial or factorization machine or any classification algorithms that output a real number?

Some disadvantages of this approach I have in mind:

the parameters scale linearly with the number of classes
the loss function may be non-convex and hard to optimize
the theoretical properties/guarantees of the original binary classifiers may be lost

And how does this compare to classical 1-v-1 or 1-v-all approaches?

$\dagger$ Only $k-1$ is needed since softmax is over-parameterized


texts: Working ofLSTM with multiple Units - NER

texts:
I am trying to understand working of LSTM networks and kind of not clear about how different neurons in a cell interact each other. I had a look at a similar question, but still not clear about few things.

Assume an NER task where each time step get embedding on one word as input and there are 10 time steps. I defined the LSTM cell with 100 units, As per my understanding each units will have 3 separate gates and memory cell as shown in the second figure (total 100 memory cells and 150 gates). From the theory of LSTM, each of these memory states stores different information, like one might store about singularity, other might store information about gender and so on (please correct me if I am wrong). With this set up, I have few doubts about the working of LSTM and the interaction between each units/neurons

1) In the figure, each cell has multiple neuron, lets say 100. Does the first input X1 (first word vector) will be input to the first neuron of the cell along with previous cell's activation function (say h0) and its output will go to the second neuron and so on..? Or same input X1 and h0 will be fed to each 100 neurons parallel (independent) and respective outputs will go to next time step (not sure whether any aggregation happens) ?

2) How different units would be able to store different features (singularity, gender etc..)? is it because they are initialized with random and different weights?

3) If neurons are working parallel, Is there any interaction happens between them to share information? will it be possible that multiple cell state store about same information (say, gender)

4) If LSTM cell 1 has 100 neurons/units and input X1, output h1 will also have 100 values (one for each neuron) ?


texts: What are some recommended visualisations to represent how attributes in a data-set influence each other?

texts:
Looking at this data-set,  I want to illustrate ideally to what degree and in what direction each of the properties (fixed acidity,volatile acidity,citric acid,etc.) influence each other.

Right now I can only think of using a chord diagram or weighted network diagram to illustrate the magnitude of influence properties have on one another.

Are there any reasonable ways to visualise both the magnitude of influence properties have on one another and the direction of influence (positive/negative influence)?

Or any better ways to simply visualise the magnitude of influence?


texts:
I have a pandas dataframe of baseball stats that I have transposed that looks as follows:

Each column represents a separate plate appearance for a player. Each player can (and usually is) represented in many columns. Each row is an outcome for said plate appearance.

I'm trying to split each player's plate appearances into two equal portions (if player_a appears in 50 columns, I'd need two portions of 25).

How would I be able to run Cronbach's alpha (or Kuder-Richardson for binary events) for sets in each sample? My goal is to figure out the split-half reliability of each stat (row) in the data frame.

Is there a simpler way to do this that I'm not immediately seeing? I think I need to run some sort of for loop in Python to achieve my desired result, but it is currently escaping me.


texts:
Attention is all you need is a nice paper that suggests using
positional encodings as an alternative to RNNs in their Transformer architecture.
GPT-2 and GPT-3 are examples of using this architecture which
are trained on input data of a massive scale.
Is there a paper and a model that uses positional encodings
and outcompetes RNN/LSTM based models for small scale datasets (MBs of text data, not terabytes)?
If there are many, which ones are the leading ones in production applications?


texts: Exploratory statistics, how to idenify and remove driver (bias)

texts:
I am looking at customer data, and created frequency tables (+histograms) for customers with different professional statuses and what the best time is to reach them. Status ranges here from employed, retired, self-employed, unemployed, blank.

For each of these statuses, I expected some variation in terms of when the best time is to reach each type of customer. Intuitively and from experience e.g. employed people, on average, should be available early in the morning or early evening, while unemployed are expected to show a more even distribution. However, the distributions look very similar and the peak hours for all statuses are between 8-11, I am pretty sure that the agents drive this peak. There are more agents working early than in the afternoon.

How can I extract this effect to be able to focus on what the best times to reach are for these different group of customers?


texts: Is it appropriate to use polar images as input to CNNs? Or must they be Cartesian transformed first?

texts:
I have built a convolutional neural net which trains on data originally represented in polar space (measurements are a function of angle and distance). My pipeline begins by converting coordinates to a Cartesian grid and re-projecting the images. This however results in a significant increase in the data dimensions (i.e. pixel size of the input images) and requires interpolation steps which I would prefer to avoid.

Is there an appropriate method (implemented in keras/tensorflow) I can use for performing (2D) convolutions on the raw polar data? I have followed the cs231n course, so I have some background but am not yet an expert, especially on the theory side. Thanks!


texts: Find best (most efficient) combination of Ingredients

texts:
Question:
Suggest me an approach to that kind of problem. Is it pure optimization problem?
What might Machine Learning techniques can be helpful here?
Case Study
My reduced case study is as follows:

I have 3 types of ingredients (A, B, C). In each experiment, some
combination from the ingredients is chosen and applied on something.
Result can be 1:poor, 2:middle, 3:high.

Example:
| Input      | Output      |
|:----------:|:-----------:|
| A          |       2     |
| AB         |       1     |
| ABC        |       2     |
| BC         |       3     |
| C          |       2     |
|...         |      ...    |

I need to find out what is the winning combination of ingredients according to the results.
Thanks


texts: Elements of Statistical Learning - question on p. 18

texts:
In the following expression:

What do $E_{X}$ and $E_{Y|X}$ mean, respectively?

In case helpful, I note that the authors derived that expression from the following:

Based on the above, how would one conclude that that minimizing $EPE(f)$ can be accomplished as follows?


texts: Why is my regression predictions values variance different from real values variance

texts:
I am modeling a physical process using a regression(XGBoost).
I'm looking for ways improving my model, have tried different things without success.
Decided to get a better intuition on where my model is wrong.

After training the model the predicting on a hold-out test set, I compare the predicted values distribution to the real values.
The comparison reveals the 'learnt' variance is different from real.

What are the possible explanations for this 'wrong' variance learning?

This is how the variances look


texts: MAE and MSE are Nan for regression with Neural Networks?

texts:
I doing a simple neural network for Reggression, I didn't get any error but the MSE & MAE are Nan.  The code is:

dataset = pd.read_excel('data_Z.xlsx')
#df = pd.DataFrame(stock, columns= ['X1', 'X2', 'X3', 'X4', 'X5', 'Y1',
'Y2'])

#Variables
x=dataset.iloc[:,0:5]
y=dataset.iloc[:,5].values
y=y.reshape (-1,1)
scaler = MinMaxScaler()
print(scaler.fit(x))
print(scaler.fit(y))
xscale=scaler.transform(x)
yscale=scaler.transform(y)
X_train, X_test, y_train, y_test = train_test_split(xscale, yscale)
model = Sequential()
model.add(Dense(12, input_dim=5, kernel_initializer='normal',
activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='linear'))
model.summary()

model.compile(loss='mse', optimizer='adam', metrics=['mse','mae'])
history = model.fit(X_train, y_train, epochs=150, batch_size=50,  verbose=1,
 validation_split=0.2)

I cannot understand why?


texts:
I have a dataset with 11 features, I noticed that manipulating these features (eg dropping one or some of them) doesn't affect the error scores of training and testing data, so I had to check the importance of these features. Here's the following:

As noticed the first feature has a very high contirbution. However, the rest have insignificant importance. Thus I tried to run the model using only the first feature. It was expected that the results scores will not decrease significantly as the rest 10 dropped features have very low feature importance. However, after running the experiment with only the first feature, the abs error percentage of the testing data increased significantly from 14.13010% to 22.96036%. why is this happeneing? I expected that the error will be almost near to the base testing results as I train using the feature which dominates the feature importance?
Also, some of these features are correlated (no more than .62 correlation), is this the reason why the scores can't be so reliable? if so, what mertic can I use to test the feature importance for correlated features


texts: One question guys, someone knows if it should be ok to get one more GPU of type Nvidia Geforce GTX 1070 (gaming version), given that now I have GTX 1070 Titanium? They don't have another Titanium card available here, so I have to get a different one, but closely similar, and I wonder if for using Keras (with TensorFlow backend), will it work fine? They are not exactly the same cards, but similar enough maybe. I want 2 GPUs for Keras.


texts: How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

texts:
I've noticed that people produce such an amazing pandas dataframe which allows people to answer the questions. However, whenever I try to copy my table after doing df.head(), and "paste" it on here so that I am able to ask a question it leaves me with a table that needs to be restructured.

I have searched for similar questions here but have been struggling to find a step-by-step answer on how to ask questions especially when it is related to pandas and that one has to get a sense of the data at hand before being able to provide an answer.


texts:
When applying logistic regression, one is essentially applying the following function $1/(1 + e^{\beta x})$ to provide a decision boundary, where $\beta$ are a set of parameters that are learned by the algorithm, and $x$ is an input feature vector. This appears to be the general framework provided by widely available packages such as Python's sklearn.

This is a very basic question, and can be manually implemented by normalization of the features, but shouldn't a more accurate decision boundary be given by:  $1/(1 + e^{\beta (x - \alpha)})$, where $\alpha$ is an offset? Of course an individual can manually subtract a pre-specified $\alpha$ from the features ahead of time and achieve the same result, but wouldn't it be best for the logistic regression algorithm to simply let $\alpha$ be a free parameter that is trained, like $\beta$? Is there a reason this is not routinely done?


texts: How to deal with a machine learning model which affects future ground truth data?

texts:
I wonder if a binary classification problem which will steer a business process really is binary. As you see steering means affecting the outcome of the business process. As such ground truth data will be different than it would have been without the model.

What strategies are there to deal with this problem (besides a hold out group).
Recently I read http://ieeexplore.ieee.org/document/7280527/ wich additionally incorporates early feedback.

Is it suitable to assume that after initial launch of such a model which predicts 0 and 1 class labels future ground truth data wich is returned with a delay of up  to 30 days will in reality be up to 6 classes:

0 prediction of label 0
1 prediction of label 1
feedback that label was actually

0 for a prediction of 0 (regular outcome)
1 for a prediction of 1 (regular /desired outcome)
0 for a prediction of 1 (model did falsely classify, groundtruth affected)
1 for a prediction of 0 (model did not detect classification, suboptimal and lost some money but ground-truth considered not affected)

How could the feedback be incorporated? Would you suggest to expand the binary classification to a multi-class classification?


texts: Does anyone have memory utilization benchmark for random forest and xgboost?

texts: I want to compare which technique has higher memory utilization while training on the same dataset


texts: depmixS4 - How to fit HMM usgin multiple sequences of observation

texts: Is there any R package that supports fitting an HMM using multiple sequences of observations? to the best of my knowledge depmixS4 does not support this feature


texts: Keras Classifier returns similar output for all Predictions

texts:
I completed training the model with an accuracy of 1.000 and a validation accuracy of 0.9565. Unfortunately whenever i input a image into my model i get the same output regardless. Am i doing something wrong when predicting or during my training. W and A are my class labels.

My folder structure for the image generators are as follows:

images/

a/
   a001.jpg.png..
 w/
   w002.jpg.png..

train_datagen = ImageDataGenerator(
    rescale=1./255,
    shear_range=0.2,
    zoom_range=0.2)

test_datagen = ImageDataGenerator(rescale=1./255)

from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(150, 150,3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(32, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Conv2D(64, (3, 3),padding='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2),padding='same'))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

batch_size = 64

# this is a generator that will read pictures found in
# subfolers of 'data/train', and indefinitely generate
# batches of augmented image data
train_generator = train_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\train',  # this is the target directory
    target_size=(150, 150),  # all images will be resized to 150x150
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')  # since we use binary_crossentropy loss, we need binary labels

# this is a similar generator, for validation data
validation_generator = test_datagen.flow_from_directory(
    'C:\\Users\\Zahid\\Desktop\\Dataset\\val',
    target_size=(150, 150),
    batch_size=batch_size,
    color_mode='rgb',
    class_mode='binary')

model.fit_generator(
    train_generator,
    steps_per_epoch=2000 // batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=800 // batch_size)
model.save_weights('first_try.h5')

img = cv2.imread("C:\\Users\\Zahid\\Desktop\\Data\\TrainingData\\images\\a\\img_0201.jpg.png")
resized_image = cv2.resize(image, (150, 150))
x = img_to_array(resized_image)
x = x.reshape((1,) + x.shape)
x = x/255
print(x.shape)
scores_train = model.predict(x)
print(scores_train)


texts: Mutation on demand in genetic algorithms

texts:
genetic algorithm usually use a "mutation rate" to control the rate of chromosome mutation. Most of the researchers at researchgate recommend to keep this rate low in order to converge quickly, to be able to find local optima and not to make the optimization a random walk. However, I see one major problem with keeping a low mutation rate. If the breeding at one point doesn't result into "new" children/chromosomes, the algorithm will run very inefficiently. Assume that all individuals in the population have the same chromosome, then crossovers will result into the same individuals until a mutation occurs in one of the chromosomes. With a low rate this can take quite long (until then the fitness function with the same chromosomes is repated again and again).

Isn't it much more efficient to mutate on purpose as soon as children/chromosomes would repeat in the next generation in order to assure that we explore new solution space? Wouldn't this make a "mutation rate" obsolete and the entire algorithm more efficient or do I miss a major point here?


texts: Difference between Gensim word2vec and keras Embedding layer

texts:
I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the same thing, they all try to convert a word into a feature vector.

Am I understanding this properly? What exactly is the difference between these two methods?

Thanks!


texts: High Recall but too low Precision result in imbalanced data

texts:
I was training a model using XGBoost Classifier on heavy imbalanced data base with 232:1 of binary class. Because my training data contains 750k rows and 320 features (after doing many feature engineering, feature correlation filtering and low variance filtering), I prefer to use scale_pos_weight to dealing with imbalanced rather than oversampling data. After parameter tuning using Bayesian optimization to optimize PR AUC with 5 fold cross validation, I got the best cross validation score as below:
PR AUC = 4.87%, ROC AUC = 78.5%, Precision = 1.49%, and Recall = 80.4%
and when I tried to implement the result to a testing dataset the result is below:
accuracy: 0.562
roc_auc: 0.776293
pr_auc: 0.032544
log_loss: 0.706263
F1: 0.713779
Confusion Matrix:
[[9946 7804]
 [  18   84]]
          precision    recall  f1-score   support

       0       1.00      0.56      0.72     17750
       1       0.01      0.82      0.02       102

    accuracy                           0.56     17852
   macro avg       0.50      0.69      0.37     17852
weighted avg       0.99      0.56      0.71     17852

My parameter range to be optimize (consume 2-3 days with 100 iteration) is:
{'learning_rate':(0.001,0.2),'min_split_loss':(0,20),'max_depth':(3,10),'min_child_weight':(0,50),'max_delta_step':(0,10),'subsample':(0.5,1),'colsample_bytree':(0.5,1),'colsample_bynode':(0.5,1),'colsample_bylevel':(0.5,1),'reg_lambda':(1e-5,100),'reg_alpha':(0,1), 'objective':'binary:logistic','booster':'gbtree','scale_pos_weight':232,'n_estimators':200}

According to business request, we have more consideration to high recall (to save those in positive class), however I am frustrated by too low precision result (this is impact to the cost to save positive class). Is there any solution to increase the precision at least to be 10% without hurting the Recall?


texts:
I'm reading Hogan et al's Neural Network Design book very closely.  I have run into a couple of questions about its presentation on Hamming networks.

In particular, it says:

  The next network we will consider
  is the Hamming network [Lipp87]. It
  was designed explicitly to solve binary pattern recognition problems
  (where each element of the input vector has only two possible values — in
  our example 1 or
  ).

OK I guess.  That's sort of not what is meant by "binary classification", but they use a slightly different phrase, so I'll give them the benefit of the doubt.

So can a Hamming network do multiclass classification or not?  Certainly, this quote does not exclude that possibility.

If so, what does the weight matrix look like in the recurrent layer?  The example they show is for binary classification (apples and oranges), and give

$$
W^2 =  \begin{bmatrix}
         1         & -\varepsilon \\
         -\varepsilon & 1         \\
       \end{bmatrix}
$$
where there are some constraints on $\varepsilon$.  Can someone confirm that, in the multiclass case, this generalizes to the matrix defined by

$$ W^2_{i,j} = \begin{cases}
                 1, \text{ if } i = j \\
                 -\varepsilon, \text{ otherwise}
               \end{cases}
$$


texts: Why is approximate nearest neighbour with LSH efficient?

texts:
In a brute force nearest neighbour: (consider point in question is 'x', and total number of points is n)

calculate distance between x and every other point O(n)
Compare all these distances to get the minimum O(n)

With LSH:

calculate hash value for each point and construct a hash map O(n)
Calculate hash value of x: h, lookup other points stored against h in the hash map O(1)

Overall, it seems it is still O(n). What are the factors that are really making LSH more efficient? I could think of following two:

the first step in both of the approaches is done using matrix multiplications and because of parallelisation, they aren't really O(n) but more efficient than that. So the bottleneck becomes the second step which is constant time with LSH.
The first step of LSH is not concerned with the point in question x and hence can be precomputed and stored once. Again, with precomputation, the approx. the algorithm becomes only constant time and hence more efficient.

Are there any other factors that make the LSH nearest neighbour efficient?
And also, more importantly, Is my understanding correct? Or is there something very simple that I am missing here?


texts: The Gradient descent different between in Ng coursera and Michael A. Nielsen book

texts:
I am learning the neural networking from NG machine learning course in coursera and the book neural networking and deep learning by Nielson. I have a little confusion about the understanding of the Gradient Descent. I see two different formals to update weight by the Gradient Descent between NG and Nielson.

from Nielson:

the chapter Two, section The backpropagation algorithm

from NG:

both of them are used to update weights after forwarding pass in the backpropagation algorithm. in Neil version, it seems there is a learning rating -η in NG version, it does not have any learning rating, and it more like the learning rating is +1. I am very confused on this. is there anyone can help me to understand it?


texts: Algorithms and techniques for spell checking

texts:
Can anyone suggest any algorithm and technique for spell checking?
After some googling, I found some interesting ones such as this one from Peter Norvig, http://norvig.com/spell-correct.html and few others. However, most of them were written many years ago. Therefore, I am trying to find out whether is there any newer/improved approach to tackle such problem.


texts:
For a particular task, I need a convex activation function with the following properties:

f''(x) > 0
0 <=    f(x) <=  1
f(x) is monotonic
f(x) is not "exploding" i.e. avoiding functions such as f(x) = x²

The only example I have in mind for this is the softplus activation function. Would you have anything else in mind?


texts: best approach to embed random length sequences of words as a fixed size vector without having a maximum length?

texts:
I have a dataset of sentences in a non-English language like:

word1 word2 word3 word62

word5 word1 word2

and the length of each sentence is not fixed.
Now, I want to represent each sentence as a fixed sized vector and give it to my model and i want to keep as much information as possible in the embedding, and i don't want to have a maximum length for sentences because important information might happen in the end.
The only two approaches I can think of so far are:

Convert them to one hot vector and add them

Convert them to a word embedding and then add them

Is there any better way? What is the best approach to represent a variable length sentence without losing information from it (like having a maximum length for each sentence - I want all the words in the sentence to affect the embedding)?


texts: Non-linear decision boundary in logistic regression algorithm with polynomial features

texts:
Lately I have been playing with drawing non-linear decision boundaries using the Logistic Regression Classifier. I used this notebook to learn how to create a proper plot. Author presents a really nice way to create a plot with decision boundary on it. He adds polynomial features to the original dataset to be able to draw non-linear shapes. Then draws few plots for different values of degree param (that polynomial features function works exactly like this one from sklearn).
I followed this notebook on my environment. Instead of using minimize function from scipy.optimize package to find best weights (like in the notebook), I used my own Logistic Regression implementation with gradient descent optimization algorithm. For polynomial features with degree = 1 and degree = 2 my plots look exactly like in the notebook. But for degree = 6 there is a difference. In notebook decision boundary is really curvy, which i would also excpect, because generating more features leads to better fitting to training dataset. In my algorithm there is really small difference between degree = 2 and degree = 6.
Here is a comparsion, where first one is plot from the notebook and second one is mine.

I also ran weights optimization with minimize function and noticed that obtained weights are so different from ones which I obtained using my own implementation.
Weigths obtained from minimize function:

[   35.04544314    44.02433844    69.15673318  -343.69261527
  -197.89262344  -183.96784381  -295.14107156  -620.36490083
  -509.82244752  -327.88923523  1092.78139854  1267.14765298
  1754.90950905   899.49504769   436.09827632   470.00768776
  1233.10546815  1818.64194606  1926.06287183  1129.25103089
   463.40430072 -1140.10610687 -2016.5976485  -3456.09451404
 -3477.54092932 -3247.55983249 -1544.01162994  -510.04081232]

Weights obtained from my own implementation:

[ 3.88423676  2.61252164  4.41490058 -5.49636578 -5.70598669 -6.88741771
  1.17688239 -1.44447375  0.24250327 -1.28316639 -4.71695206  1.22093586
 -3.2154096  -2.74892981 -4.42193179 -0.68483587 -0.28642677  2.34231314
 -2.35831624 -2.0624521   1.26929726 -4.5863475   1.65033913 -0.82767549
  1.64820792 -2.03474736 -2.00457816 -0.50033884]

It can be noticed that weights from minimize function have much much higher values. Why there is so much difference between these two methods? Why weithts obtained from gradient descent algorithm are that different? I thought that adding high power polynomial terms will always make decision boundary more curvy. Is there a mistake in my thinking?
I also attach my Logistic Regression implementation below. For that degree = 6 plot i used alpha=1, epoch=3000.
import numpy as np

class LogisticRegressionModel:

    def __init__(self, alpha=0.05, epoch=100):
        self.__alpha = alpha
        self.__epoch = epoch
        self.__weights = []
        self.__errors = []

    def learn_and_fit(self, X, Y):
        self.__weights = np.random.rand(X.shape[1], )
        m = X.shape[0]

        for _ in range(self.__epoch):
            J = self.cost_function(X, Y)
            activations = self.activation_function(X)
            gradient = 1 / m * np.dot(X.T, activations - Y)
            self.__weights = self.__weights - self.__alpha * gradient
            self.__errors.append(J)

    def cost_function(self, X, Y):
        m = X.shape[0]
        activations = self.activation_function(X)
        J = np.dot(-Y.T, np.log(activations)) - np.dot((1 - Y).T, np.log(1 - activations))

        return 1 / m * J

    def activation_function(self, X):
        Z = X.dot(self.__weights)
        activations = self.sigmoid(Z)

        return activations

    def sigmoid(self, Z):
        return 1 / (1 + np.exp(-Z))

    def predict(self, X):
        activations = self.activation_function(X)

        return np.where(activations < 0.5, 0, 1).flatten()

    def get_errors(self):
        return self.__errors

    def get_weights(self):
        return self.__weights


texts: I want to build an Uplift Model for multiple treatments. To get a good model, I would like to use boosting. How is it possible to use boosting with uplift modeling although we can't really calculate residuals and build our next tree on the last one (Causal inference)? It seems to be impossible to improve our current tree if we don't have a "true" uplift value for each observation, which we can use to calculate residuals.


texts: Artificially expanding the datasets through rotation of images in MNIST

texts:
I came across this question from the 3rd chapter of the book Neural Networks and Deep Learning by Michael Nielsen, this is a question given in his exercise.

  One way of expanding the MNIST training data is to use small rotations
  of training images. What's a problem that might occur if we allow
  arbitrarily large rotations of training images?

I would happy if someone explain why large rotations would be problematic.


texts: How to create a system to detect text structure of a file?

texts:
Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or missing data.

How do I classify it into these class types or classify it is an anomaly if it doesn't belong to anyone of them.

I thought about anomaly detection but couldn't figure how to parse structure information from the text classes like (F1, F2... .etc).

Also what kind of structural information to extract from text files?

These input classes contain 100 - 1000 lines of code per document of each class type.

I looked into Linting or DeepCode ...

A sample log file looks like this:

11-02-11 16:47:35,985 +0000 E Activity class {com.trackingeng/LandingActivity} does not exist.
12-02-11 17:47:35,985 +0000 I Starting: Intent { act=android.intent.action.MAIN
.....

A log file may have stack trace like this also:

Error:
    Error detail 1
    Error detail 2
    ....
Non-Error:
    .....
Warning:
    .....
and similar to this.

Any help in which direction to look for is greatly appreciated.


texts: Which feature to use in feature selection?

texts:
Objective: Multiclass classification with supervised learning, small dataset (25h)

Context: My dataset is composed of mobile network data collected with a smartphone. The labels correspond to the activity of the user (Stationary, Walk, Subway, Train, Car). My features are calculated based on 3 fields: timestamp, ID, and signal strength (SS). All have different overlapping size windows: 15s, 30s, 602, 90s, 120s. So, I have 3 features based on ID and 16 statistical features based on SS for each window size with a total of 95 features.

My Question: Which feature selection should I use? Am I correct saying the features are not independent?

(I'm using python).


texts: What do you call an anomaly/signature detection algorithm in data science

texts:
If I have an algorithm for detecting a set of data points that indicate with a high level of certainty that some event has occured OR that behaviors outside of a set model are occuring.

What would you call the algorithm used to detect these anomalies?

And

Would calling it an "analytic" be incorrect?


texts: Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

texts:
I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.

As I would say my knowledge about more complex analysis is quite limited I've done mostly all basic methods to gather a overall view of the data. This includes simple graphs which already help to identify similar patterns. I am also using SAS Enterprise 9.4.

However I am looking for a more number based approach.

Data Set

The (self) collected data set I am using contain around ~54.000 fares.
All fares were collected within a 60 day time window, on a daily basis (every night at 00:00).

Hence, every fare within that time window occurs $n$ times subject to the availability of the fare as well as the departure date of the flight, when it is passed by the collection date of the fare.
(You can't collect a fare for a flight when the departure date of the flight is in the past)

The unformatted that looks basically like this: (fake data)

+--------------------+-----------+--------------------+--------------------------+---------------+
| requestDate        | price| tripStartDeparture | tripDestinationDeparture | flightCarrier |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 725.32    | 16APR2015:10:50:02 | 23APR2015:21:55:04       | XA            |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 966.32    | 16APR2015:13:20:02 | 23APR2015:19:00:04       | XY            |
+--------------------+-----------+--------------------+--------------------------+---------------+
| 14APR2015:00:00:00 | 915.32    | 16APR2015:13:20:02 | 23APR2015:21:55:04       | XH            |
+--------------------+-----------+--------------------+--------------------------+---------------+

"DaysBeforeDeparture" is calculated via $I=s-c$ where

I & interval (days before departure)
s & date of the fare (flight departure)
c & date of which the fare was collected

Here is a example of grouped data set by I (DaysBeforeDep.) (fake data!):

+-----------------+------------------+------------------+------------------+------------------+
| DaysBefDeparture | AVG_of_sale | MIN_of_sale | MAX_of_sale | operatingCarrier |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 880.68           | 477.99           | 2,245.23         | DL           |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 904.89           | 477.99           | 2,534.55         | DL           |
+-----------------+------------------+------------------+------------------+------------------+
| 0               | 1,044.39         | 920.99           | 2,119.09         | LH               |
+-----------------+------------------+------------------+------------------+------------------+

What I came up with so far

Looking at the line graphs I can already estimate that several lines will have a high correlation factor. Hence, I tried to use correlation analysis first on the grouped data. But is that the correct way? Basically I try now to make correlations on the averages rather then on the individual prices?
Is there an other way?

I am unsure which regression model fits here, as the prices do not move in any linear form and appear non-linear. Would I need to fit a model to each of price developments of an airline

PS: This is a long text-wall. If I need to clarify anything let me know. I am new to this sub.

Anyone a clue? :-)


texts: What does "baseline" mean in the context of machine learning?

texts:
What does "baseline" mean in the context of machine learning and data science?

Someone wrote me:

  Hint: An appropriate baseline will give an RMSE of approximately 200.

I don't get this. Does he mean that if my predictive model on the training data has a RMSE below 500, it's good?

  And what could be a "baseline approach"?


texts:
I'm training a segmentation model, Unet++, on 2d images and I am now trying to find the optimal learning rate.
The backbone of the model is Resnet34, I use Adam optimizer and the loss function is the dice loss function.
Also, I use a few callbacksfunctions:
callbacks = [
            keras.callbacks.EarlyStopping(monitor='val_loss', patience=15, verbose=1, min_delta=epsilon, mode='min'),
            keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=3, verbose=1, mode='min', cooldown=0, min_lr=1e-8),
            keras.callbacks.ModelCheckpoint(model_save_path, save_weights_only=True, save_best_only=True, mode='min'),
            keras.callbacks.ReduceLROnPlateau(),
            keras.callbacks.CSVLogger(logger_save_path)
        ]

I plotted the curves of training loss over epochs for a few learning rates:

The validation loss and training loss seem to decrease slowly. However, the validation loss isn't oscillating (it is almost always decreasing).

The validation and training losses decreased quickly on first 2/3 epochs. After 6 or 7 epochs, the validation loss increases again.

I have a few questions (I hope it is not too much):

*What is normally the best way to find the learning rate i.e.

How many epochs should I wait before considering that the learning rate isn't good?
What are the criteria on the loss function to determine if a learning rate is "good"?

Is there a big difference if I use a small learning (which still converges) instead of the "optimal" learning rate ?
Is it normal that the validation loss function oscillates over the training?
Which learning rate should I use according to my results?

Even a partial response would help me a lot.


texts: Periodically executing a scraping script with Python

texts:
Here is my idea and my early work.

My target

Fetch 1-hour resolution air pollution data from China's goverment continuously.
The website's data which collected from the monitor sites over the country update per hour .

My Code

Now, I can grab the useful information for a single hour. Here is my code:

Input the website links for different pollution(co,no2,pm10, etc)

html_co = urllib.urlopen("http://www.pm25.in/api/querys/co.json?city=beijing&token=5j1znBVAsnSf5xQyNQyq").read().decode('utf-8')
html_no2 = urllib.urlopen("http://www.pm25.in/api/querys/no2.json?city=beijing&token=5j1znBVAsnSf5xQyNQyq").read().decode('utf-8')
html_pm10 = urllib.urlopen("http://www.pm25.in/api/querys/pm10.json?city=beijing&token=5j1znBVAsnSf5xQyNQyq").read().decode('utf-8')

Get the content of the html doc.

soup_co = BeautifulSoup(html_co)
soup_no2 = BeautifulSoup(html_no2)
soup_pm10 = BeautifulSoup(html_pm10)

Extract the useful information from the whole content.

l = soup_co.p.get_text()
co= json.loads(l)
l = soup_no2.p.get_text()
no2= json.loads(l)
l = soup_pm10.p.get_text()
pm10= json.loads(l)

Tight the raw data into neat Pandas.Dataframe.

data = {"time":[],"station":[],"code":[],"co":[],"no2":[],"pm10":[]}
for i in range(0,len(pm10)-1,1):
    ## 'station' is the monitor station's name in Chinese
    data["station"].append(co[i]["position_name"])
    data["time"].append(co[i]["time_point"])
    data["co"].append(co[i]["co"])
    ## 'code' is the monitor station's index
    data["code"].append(co[i]["station_code"])
    data["no2"].append(no2[i]["no2"])
    data["pm10"].append(pm10[i]["pm10"])

My result

Some pre-explanation

Ignore the chinese character in the table.
I only grab one city(Beijing here)'s data, and index from 0-11 notify that Beijing has 12 monitor site.
The columns 'co'/'NO2'/'PM10' represent the concentration of these air pollutants.

My problem

Now, I can grab the web data manually according my code above. But, I want to achieve the working flow below hourly  automatically.

Hour i

Execute the code
(1) Grab the data for Hour i's air pollutants data from website;
(2) Save the data into .csv based on the true date(like 20160101.csv)

After one hour.

Execute the code
(1) Grab the data for Hour i+1's air pollutants data from website;
(2) Save the data into .csv based on the true date.
  if it's the same day liken to hour i --> same .csv(like 2016-01-01.csv)
  if the present day has past --> creat a new .csv(like 2016-01-02.csv)

I havn't done these kind of stuff before. Can somebody offer me some advice?
So, I can get an useful data scraping tool run in the background and I don't have to worry about it.


texts: why does transform from tfidf vectorizer (sklearn) not work

texts:
I'm transforming a text in tf-idf from sklearn. I made the model:

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = words
vectorizer = TfidfVectorizer(min_df = 15)
tf_idf_model = vectorizer.fit_transform(corpus)

And now I'm making vectors for different sets of words (documents), like:

word_set = ['dog', 'cat', 'foo']
v = vectorizer.transform(word_set)

But I want just one vector of these words, to compare to other documents. But when I use transform, the shape of v becomes:

<3x56492 sparse matrix of type '<class 'numpy.float64'>'
    with 3 stored elements in Compressed Sparse Row format>

I want a vector with shape 1x56492, and not 3x56492.. I'm certainly missing something here. Maybe you guys have some tips?

Thank you very much in advance.


texts:
I would like to understand how to compute the accuracy of cluster analysis.
I have hundred of texts. The dataset looks like as follows:

Author  Text Checked
Shakespeare ‘...’ yes
Keats ‘...’ no
...

In text there are some quotes from the author. Checked column has been filled in manually.
What I would like to do is to group texts by authors.
I already did it using kmeans and TfidfVectorizer. However I would like to understand if it is possible to test the accuracy of this approach.
Do you know how I could do it?
In case it would be not possible, could you please tell me what other approach I could take into account?
It would be good to test different approach and see the level of accuracy.
Any example it would be great.


texts: DecisionTreeRegressor under the hood of GradientBoostingClassifier

texts:
I'm inspecting the weak estimators of my GradientBoostingClassifier model. This model was fit on a binary class dataset.
I noticed that all the weak estimators under this ensemble classifier are decision tree regressor objects. This seems strange to me intuitively.
I took the first decision tree in the ensemble and used it to predict independently on my entire dataset. The unique answers from the dataset were the following:
array([-2.74, -1.94, -1.69, ...])

My question is: why and how does the gradient boosting classifier turn the weak estimators into regressor tasks (instead of classification tasks) that are not bound by 0 and 1? Ultimately the GradientBoostingClassifier outputs a pseudo-probability between 0 and 1: why aren't the ensemble of weak estimators doing the same?


texts: Metrics to measure imbalanced multi-class problem

texts:
I have a multiclass imbalanced problem. The dependent variable is shown below.

              Injury
  severe Injury (Class 0)               3394
  Medium Injury (Class 1)              22318
  No Injury (Class 2)                 208753

I have used the random forest algorithm with parameter class_weight='balanced' to handle the imbalance problem. The model yields the below results.

 [[  650    12     9]
  [    3  2938  1670]
  [    7   917 40569]]

  Accuracy Score: 0.9440299305184393
  precision score: 0.9016230160324789
  Recall score: 0.8612021971135553
  AUC Score: 0.8739141097167544
  F1 score: 0.879571098748252

               precision    recall  f1-score   support

    class 0       0.98      0.97      0.98       671
    class 1       0.76      0.64      0.69      4611
    class 2       0.96      0.98      0.97     41493

avg / total       0.94      0.94      0.94     46775

Kappa Score: 0.7391731672532447

Since I am really interested in class 0 and class 1, the precision, recall, and F1 scores are calculated using "Macro" method.

 ex:
 print ('precision score:', precision_score(test_y,ry_pred, average='macro'))

So, my question is Can I take the individual classes (class 0 and class 1) precision, recall, and f1 ? or the overall average(including class 2) score for evaluating the model?

 ex:  F1 score for all three classes = (98+69+97)/3 = 0.8795
      F1 Score for 2 classes = (98+69)/2 = 83.5

If I have to take individual classes scores then, can I take class 0 and class 1 aggregate values to say how well the model is classifying the minor classes?

Also, for class 0 the model is showing high scores around 98. Is this an overfitting problem? I have evaluating model using k fold CV technique and shows below scores. It is showing the overall precision, recall, and F1 score.

     precision score 0.9009562240704383
        recall score 0.8576816035552879
            F1 score 0.8776626709718627

Please suggest.


texts: Backpropagation in a neural network with time series data?

texts:
Using basic Backpropagation and a Sigmoid activation function, solving for $\cfrac{\partial E}{\partial W_i}$ for all $W_i$ is rather straightforward.

However, when using time series data, how do you approach backpropagation?

My intuition tells me that you just do:

$$(1) \sum_{t=1}^n \cfrac{\partial  E_t}{\partial W_i} = \sum_{t=1}^n \cfrac{\partial  E_t}{\partial o_{i_t}} \cfrac{\partial o_{i_t}}{\partial W_i}
$$

where $o_{i_t}$ is the output node at time $t$, and $W_i$ is the weight feeding into that node from the previous node, $x_i$, across all times $t = 1,..., n$.

That would mean, if $\delta_{o_i} = \cfrac{\partial E}{\partial \sigma(o_i)} \cfrac{\partial \sigma (o_i)}{\partial o_i}$, where $\sigma (x) = \cfrac{1}{1+e^{-x}}$

then the appropriate representation of $\delta_{o_i}$ within the time series should be:

$$\sum_{t=1}^n \delta_{o_i} = \sum_{t=1}^n \cfrac{\partial E_t}{\partial \sigma(o_{i,t})} \frac{\partial \sigma (o_{i,t})}{\partial o_{i,t}}$$

or rewritten,

$$\sum_{t=1}^n \delta_{o_i,t} \frac{\partial o_{i,t}}{\partial W_i} = \sum_{t=1}^n (Out_t-Target_t)(Out_t)(1-Out_t)(x_{i,t})$$

This requires me to calculate $n$ backpropagation problems per iteration. Is this the proper way to do it?

It seems that if I use this method and a particular node adds random error across the time series

(say, the mean of $\cfrac{\partial  E_t}{\partial W_i}$ for all $t$ is roughly $0$), then the backpropagation algorithm will not adjust that weight since $$\sum_{t=1}^n \cfrac{\partial  E_t}{\partial W_i}$$ will be close to 0.

Are there any brief summaries of how to approach backpropagation across a time series? Or should I just train the network using each observation in the time series as an independent training point?


texts: Classification of moving pixels with convolutional neural networks

texts:
I have a data set with videos of moving pixels.
Each video contains 32 frames, each frame is 32x32 with two pixels in white and the rest in black.
I have binary labels for 800 of these obtained by human labeling.
I now want to construct a classifier in order to generalize from this set.
I have access to the algorithm generating the pixel movements.
The pixels move 1 pixel in any direction each frame, but they never overlap.
My understanding is that 3DCNNs is a good match for this type of problem, but I have mainly seen this used for "dense" videos, my data is very sparse in comparison.
I have also seen suggestions of using LSTMs or dual stream solutions.
What is the best or more optimal approach given my type of data?
I would very much appreciate any suggestions, thank you.


texts:
Is there any ML framework that readily supports LSTM for multi-dimentionsal data ?

Best would be in Pytorch. But I don't think the official API supports it and I have found an implementation.


texts:
Iterative pruning works like the following:

Evaluate model performance, continue if performance is fine
Apply pruning
Train the model
Go back to step 1

When doing this process using unstructured pruning I set 5% of the weights, which absolute value is closest to zero, to zero.
After that I train the model. The problem is that after the training the 5% of the weights which were set to zero are again very close to zero. So after applying pruning the second time the same weights are set to zero as the first time.
Is there such a thing as iterative unstructured pruning and if yes, how should I do it?


texts: How to improve accuracy of ethnicity prediction using name?

texts:
I've got a Excel file having columns Name, Surname and Community, having around 31 thousand entries. For my college project, the task I have is to write a program to predict "Community" of a person, given their name and surname. For example, the first entry in the file is the name: Yogita Singh. If someone enters a name like Yogta Singha, then approximately it should predict that that person belongs to the Punjabi communtiy.

I did manage to write a program, which does the prediction using the Levenshtein distance. I gave more weightage to the surname (10 times more), than the first name, because surname and community has a greater correlation than first-name and community (naturally, because some first-names are very common and used by people from many different communities).

Any ideas how to improve the accuracy of my model? Has there been any relevant theoretical study regarding name - ethnicity mapping/correlation models? I think Levenshtein  distance is a good start but I'm not sure the exact weights I should give to the surname and first name. Also, perhaps I need to give more weight to the characters towards the beginning of the surname, rather than characters towards the end.

Any suggestions regarding how to improve the accuracy of my model will be greatly appreciated. Thanks.


texts:
I'm trying to predict a score that user gives to a restaurant.

The data I have can be grouped into two dataframes

data about user (taste, personal traits, family, ...)
data about restaurant(open hours, location, cuisine, ...).

First major question is: how do I approach this?

I've already tried basic prediction with the user dataframe (predict one column with few others using RandomForest) and it was pretty straightforward. These dataframes are logically different and I can't merge them into one.

What is the best approach when doing prediction like this? What are the most common methods used for task like this?

My second question is what is the best way to handle categorical data (cuisine f.e.)?

I know I can create a mapping function and convert each value to index, or I can use Categorical from pandas (and probably few other methods). Is there any prefered way to do this?


texts: will k-means clustering converge to the same results given the same data set?

texts:
I did some study on the k-means clustering algorithm. It seems that the only non-deterministic part is the centroid - initialization.

Assume I have 10k data points, and a given k. I then initialize the initial centroids randomly in my each try:

Try_1:
   Initial k-centroids randomly with seed_1. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_2:
   Initial k-centroids randomly with seed_2. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_3:
   Initial k-centroids randomly with seed_3. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_4:
   Initial k-centroids randomly with seed_4. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

Try_5:
   Initial k-centroids randomly with seed_5. Then keep updating the centroids until converge (assuming we can use the 10k data points multiple times)

In these 5 tries, will the final cluster results be the same?


texts: How does SelectKBest() perform feature selection?

texts:
SelectKBest(f_classif, k), where k is the number of features to select, is often used for feature selection, however, I am having trouble finding descriptive documentation on how it works.  A sample of how this works is below:

model = SelectKBest(f_classif, k)
model.fit_transform(X_train, Target_train)

The ANOVA F-value, as I understand it, does not require a categorical response.  (see scipy.stats.f_oneway)  It is computing the value between the features. Why does f_classif require the response?

How does SelectKBest actually achieve a ranking of features based on the F-value when there should only be one F-value for a set of data?


texts: what information can we obtain from t-SNE?

texts:
I see that t-SNE can help us reduce dimensions and visualize the data. But what information are we gaining from this visualization? As we know that the new axis don't have a meaning in our context.
Moreover, if we have a class labeled data, then what information can we gain from the visualization? We already know that there are some 'n' classes and that we have to classify new examples in one of these classes.
Or am I wrong to understand t-SNE?


texts: Data Augmentation techniques for classification of imbalanced time series datasets

texts:
Now I have a task to classify the imbalanced time series datasets using ML classifiers, such as Logistic Regression, Decision Tree, SVM, and KNN. I am not allowed to use the Deep Learning tools, such as CNN and RNN. The time series data is measurements of the Force-Displacement Curve from a production line. The dataset is extremely imbalanced (minority class: majority class= 1:100). So I want to use Data Augmentation techniques to enlarge the size of the minority class, in order to optimize the performance of the classifiers and avoid overfitting.

I have tried many tools in feature-space, such as oversampling, undersampling, SMOTE, ADASYN and so on. But their performance is not so perfect. And I wish to generate synthetic time series data using Data Augmentation techniques, based on the initial data.
Similar to Image Recognition, I have also tried the exiting methods which have been applied to images, such as scaling, rotation, and jittering. But they are also not so useful.

So I want to ask if there are any other DA techniques to generate synthetic time series data. I have only some initial ideas, such as using DTW, Fournier Transform, Markov Chain and so on, but no papers or code about applying them.

Can anyone help me? I really appreciate your help. Thank you!


texts: Is using samples from the same person in both trainset and testset considers being a data leakage?

texts:
Suppose a neural network is built for a binary classification problem such as recognize the face as a smiley face or not, by using a dataset of 1000 persons and each person has ten images of his face.
If the dataset randomly spilt into trainset and testset by a ratio of 70:30, in this case, there is a big chance face image of same persons will be used in both the trainset and testset, so is this considered to be data leakage (train-test contamination)?


texts:
Keras has a problem with the input dimension. My first layer looks like this:

model.add(Dense(128, batch_size=1, input_shape=(150,), kernel_initializer="he_uniform", kernel_regularizer=regularizers.l2(0.01), activation="elu"))

As you can see the input dimension should be (150,) and with the fixed batch_size it is (1, 150)

My data has dimension (150,) and could be for example a numpy array with 150 zeros.

old_qval = model.predict(old_state_m)

Here I call the model to make a prediction. Normally Keras should automatically add the batch size as an extra dimension so I should end up with (1, 150) which would work. But Keras adds the dimension for the batch size at the wrong place and I end up with (150, 1). I tried tensorflow and theano backend.

Do I have a bug in my code or is it a problem with Keras?

How can I fix the problem? I could reshape my input data but it already has the needed shape of (150,) and should be fine. What else could I do?

If I should provide more data or code feel free to ask.


texts: Help in understanding the maths behind Logistic Regression

texts:
I am following the lecture notes available https://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf

I cannot understand how Eqs 12.4 and 12.5 come,

why the Bernoulli probability has $1-p(x)$ in the denominator,
how come $p(x) = \exp(\beta + \beta^Tx)$
and how $log \frac{p(x)}{1-p(x)}$ evaluates to $\beta + \beta^Tx$.

In general $\beta$ is the parameter of the model but I don't quite follow how come the log expression evaluates to it. Is there some mathematical formula which is skipped that is used to evaluate the log expression? This is crucial for me to know as these values are substituted in eq 12.10 where $p(x) = \exp(\beta + \beta^Tx)$


texts: Adding a default date column to a dataframe obtained from a json file

texts:
I have a set of json files that contains a list of weather parameters for every hour in a day for a complete month. The name of the Json files are in this format  " 2016-1" where 2016 denotes the year and 1 denotes the month January. Now considering that the given data in the json file is in order, how can I add a new column to the dataframe that automatically fetches the month and the year from the file name and repeats the date 24 times until the month gets over. The date format should look like this >  DD-MM-YYYY

Thanks in advance :)


texts: What is the job of "RepeatVector" and "TimeDistributed"?

texts:
I read about them in Keras documentation and other websites, but I couldn't exactly understand what exactly they do and how should we use them in designing many-to-many or encoder-decoder LSTM networks?

I saw them used in the solution of this problem here.

model = Sequential()
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))
model.add(RepeatVector(10))
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear'))
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])


texts: Gradient descent with infinite gradient value

texts:
Given a function $f(x)$ and $\frac{\partial f(x)}{\partial x_i}=\frac{f^2(x1,...,x_i+\pi/2,...,x_n)-f^2(x1,...,x_i-\pi/2,...,x_n)}{f(x)}$. When $f(x)\to0$, $\frac{\partial f(x)}{\partial x_i}$ could be infinitely large. ($f^2(x1,...,x_i+\pi/2,...,x_n)-f^2(x1,...,x_i-\pi/2,...,x_n)$ is always non-zero)

I have very little experience in deal with this situation in gradient descent process...In my code, $f(x)$ is in continuous domain but for purpose to simulate some real world process, $f(x)$ is sampled to be discrete and would return values uniformly distributed over $[0,1]$. Assume discrete $f(x)$ has $N$ identity values, at the beginning there is a training set of size $M$ ($M$ is very large), $\{x_i, f(x_i)=\frac{k_i}{N}\}_{i=1..M} (k_i \in 1, 2, ..., N)$.

I found that setting $1/f(x)$ to some value like $0.01$ when $f(x)=0$ would reach the optimizim easily but slightly slower than ideal process, while set to much smaller value like $0.00001$ would let $f(x)=0$ have a great impact on the process and failed to form a descent curve.

Is the method replacing infinitely large values to some large but finite values correct? Or there are any better ways to deal with the infinite gradient problem?

Thanks in advance!


texts:
I am trying to learn relevant features in a 300*299 training matrix by taking a random row from it as my test data and applying sequentialfs on it. I have used the following code:

>> Md1=fitcdiscr(xtrain,ytrain);
>> func = @(xtrain, ytrain, xtest, ytest) sum(ytest ~= predict(Md1,xtest));
>> learnt = sequentialfs(func,xtrain,ytrain)

xtrain and ytrain are 299*299 and 299*1 respectively. Predict will give me the predicted label for xtest(which is some random row from original xtrain).

However, when I run my code, I get the following error:

Error using crossval>evalFun (line 480)
The function '@(xtrain,ytrain,xtest,ytest)sum(ytest~=predict(Md1,xtest))' generated the following error:
X must have 299 columns.

Error in crossval>getFuncVal (line 497)
funResult = evalFun(funorStr,arg(:));

Error in crossval (line 343)
    funResult = getFuncVal(1, nData, cvp, data, funorStr, []);

Error in sequentialfs>callfun (line 485)
    funResult = crossval(fun,x,other_data{:},...

Error in sequentialfs (line 353)
                crit(k) = callfun(fun,x,other_data,cv,mcreps,ParOptions);

Error in new (line 13)
    learnt = sequentialfs(func,xtrain,ytrain)

Where did I go wrong?


texts: Why replay memory store old states and action rather than Q-value (Deep Q-learning)

texts:
Here is the algorithm use in Google's DeepMind Atari paper

The replay memory D store transition (old_state, action performed, reward, new_state)

The old_state and the performed action a are needed to compute the Q-value of this action in this state.
But since we already compute the Q-value of action a in state old_state in order to choose a as the best action, why don't we simply store directly the Q-value ?


texts: For an n-Gram model with n>2, do we need more context at end of each sentence?

texts:
Jurafsky's book says we need to add context to left and right of a sentence:

Does this mean,
for example, if we've a corpus of three sentences: "John read Moby Dick", "Mary read a different book", and "She read a book by Cher"; and after training our tri-gram model on this corpus of three sentences, we need to evaluate the probability of a sentence "John read a book", i.e. to find $P(John\; read\; a\; book)$ as below,

$P(John\; read\; a\; book)$
$=P(<s><s>John\; read\; a\; book<\backslash s><\backslash s>)$
$=P(John|<s><s>) P(read|<s>John) \; P(a|John\; read) P(book|read\; a) P(<\backslash s>|a\; book)\; P(<\backslash s>|book<\backslash s>)$
$=\frac{1}{3}\frac{1}{1}\frac{1}{1}\frac{1}{2}\frac{0}{1}\frac{1}{1}$       (without smoothing)

It would be great, if you let me know if the above understanding is correct?


texts: How to save dynamic files from colab to Google Drive?

texts: I have processed some png images on Colab and saved them in the Colab Directory. Now I want to save these images from Colab to Google Drive. How can I do this?


texts: Choosing best predictors neural networks

texts:
there,

I'm currently working on a project where my database has about 120 patterns with 39 columns and I am trying to build a predictive neural network out of it. It's regression task.

I was trying to get the best predictors ( alone or combinations of'em) in a simple network( 3 neurons only) to then use cross validation to better tune the model.
Problem are
1) the powerset of it is huge and my computer can't even handle generating the whole subset for simple fitting
2) only 3 neurons are already giving pretty poor results (r2<0)

Does somebody know a method or could please recommend a reading on choosing predictors for neural networks ?

Setup: windows 10, using MLPRegressor from scikitleran with hyperparameters
Hidden layer sizes = 3, max_iter= 5000, solver='sgd'


texts: Optimization motion planning by using Bellman equation

texts:
From the Montana article "Kinematics of Contact and Grasp", if I have a ball roll on the plane without sliding, the motion equation is described below:
\begin{equation*}
\begin{bmatrix}
\dot{u}_{2} \\
\psi
\end{bmatrix}
=
\begin{bmatrix}
M^{-1}_{2}R_{\psi}\\
T_{1}+T_{2}R_{\psi}
\end{bmatrix}
M_{1}\dot{u}_{1}
\end{equation*}

${(1)}$ for Ball, ${(2)}$ for Plane.  ${\psi}$ is the angle of contact between ${x}$ axis of coordinates of ball and plane. ${K, T, M}$ are the curvature form, torsion and metric tensor at time $t$ relative to coordinates of ball and plane.

We have \begin{equation*}
R_{\psi} =
\begin{bmatrix}
\cos_{\psi} -\sin_{\psi}\\
-\sin_{\psi} -\cos_{\psi}
\end{bmatrix}
\end{equation*}

How can I use the Bellman equation by Sutton and Barto to discrete path planning of the motion of the ball?

\begin{align}
v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s'} P_{ss'}^a(R_{ss'}^a + \gamma v_{\pi}(s'))
\end{align}

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from one point to another point.

The goal is how to optimize the path planning of the point in the motion.

I do not know how to determine the policy $\pi,$ and how to build up a function from Bellman equation. Please leave comments.


texts: I have a sample dataset containing around 77k of rows and I have clustered them by using k mean clustering and colored the clusters using color manager. Now i would like to view them, However I could not set the No. of rows to display as 77k in Scatter Plot and Is there any way to visualize the clusters in Knime?


texts: Which possible models can I use to predict results from this dataset?

texts:
I'm new at predicting things like this. I have a data set. The head of it is shown here:

I have yet another data set for the upcoming basketball games that are taking place soon with the score columns blank. I have to use the current data that I have that spans from 2010-2020 to predict what will happen soon. It's just that I have little knowledge of this and no experience in machine learning although i would be willing to learn.


texts: Is it necessary to take log transformation on the data values to get the minimum mean squared error?

texts: I have tried the house price prediction problem using simple linear regression by using the square feet as independent variable and price as dependent variable.while trying to check the MSE of the model,I get the output as 61940787124.62475. But after taking the log transformation to both the variables,the model shows the MSE value as 0.14824951630518876. which method is correct?And since the first model gives the larger MSE value,does it means the first model is error prone?


texts: Do Any Frameworks Provide Better Support for End-To-End Integer-Based Feature Engineering, Modeling, and Inference?

texts:
A retail enterprise I work with with wants to switch from its home-grown time series data analysis and prediction system to something more established and with community support.  One unique feature of the existing system is that every bit of input data, from prices to temperature readings, are already converted to integer values in the data warehouse (currently storing over one petabyte of data). The CTO feels strongly that the commodity compute hardware they use will process integer calculations faster (and with greater accuracy in the case of prices, for instance). As an example: A feature with 256 or fewer discrete values can pack much more tightly (8 bits per value) in Streaming SIMD (SSE) registers than the same feature encoded as a 32-bit float. (in such a case, the extended -1.0 to +1.0 value space of the float is not needed and is "wasted" from a certain standpoint)
Certainly, many mathematical operations will produce floating-point results, even if performed on integers (moving averages, etc).  Yet these results themselves may be encoded as fixed-point int values or within a scaled int range before being fed to other functions / layers of the system.
My question is: Are there any established frameworks that have better support for end-to-end integer-based features, modeling, and prediction/inference?
Thank you, in advance, for your consideration of this question.  If you feel strongly that the premise behind the question is flawed, I'd be intersted in learning why, as well.
Caveat: I'm not asking for a list of preferences.  I'm simply not aware of any framework or set of libraries that might fulfill the criteria of my inquiry.  My own research via search engines has not yielded any particular guidance, except for scholarly papers locked behind paywalls such as Training and Inference with Integers in Deep Neural Networks .  It's not even clear to me whether well-used building blocks in data science such as the c libraries beneath numpy and scipy include versions of most of their functions that accept int types rather than float types, for instance.


texts: What kind of research can be done with an email data set?

texts:
I found a data set called Enron Email Dataset. It is possibly the only substantial collection of "real" email that is public. I found some prior analysis of this work:

A paper describing the Enron data was presented at the 2004 CEAS conference.
Some experiments associated with this data are described on Ron Bekkerman's home page
Parakweet has released an open source set of Enron sentence data, labeled for speech acts.
Work at the University of Pennsylvania includes a query dataset for email search as well as a tool for generating spelling errors based on the Enron corpus.

I'm looking for some interesting current trend topics to work with.please give me some suggestions.


texts: Use embeddings to find similarity between documents

texts:
I need to find cosine similarity between two text documents. I need embeddings that reflect order of the word sequence, so I don't plan to use document vectors built with bag of words or TF/IDF. Ideally I would use pre-trained document embeddings such as doc2vec from Gensim.
How to map new documents to pre-trained embeddings ?

Otherwise what would be the easiest way to generate document embeddings in Keras/Tensorflow or Pytorch?


texts:
I'm trying to understand some weights initialisation methods by reading the article http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf .

But I don't understand their notation on variance. Right in equation (5) they refer to a variable $z^i$ and I don't know what they mean by it: if it's a collective index for all examples or not?


texts: What is the difference between multilabel dataset and special dataset with respect to imbalance problem in datasets?

texts: i have a multilabel data set which is imbalenced and noisy.i used BR approach to convert single lablel and perform balanceing.data is fragmented into one class now i want to merge these fragmented class to create multilabel dataset is it possible??


texts: How to convert tf.feature_column into a tensor?

texts:
I'd like to get an average embedding to use as an input.

Without feature_column, it can be done in this way
(from Tensorflow: how to look up and average a different amount of embedding vectors per training instance, with multiple training instances per minibatch?)

with tf.Graph().as_default():
    embedding = tf.placeholder(shape=[10,3], dtype=tf.float32)
    user = tf.placeholder(shape=[None, None], dtype=tf.int32)
    selected = tf.gather(embedding, user)
    non_zero_count =  tf.cast(tf.count_nonzero(user, axis=1), tf.float32)
    embedding_sum = tf.reduce_sum(selected, axis=1)
    average = embedding_sum / tf.expand_dims(non_zero_count, axis=1)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        embedding_ = np.concatenate([np.zeros((1,3)),np.random.randn(9,3)], axis=0)
        user_ = [[3,5,7,0], [1,2,0,0]]
        print(sess.run(average, feed_dict={embedding:embedding_, user:user_}))
        print(np.sum([embedding_[i] for i in user_], axis=1) / np.atleast_2d(np.count_nonzero(user_, axis=1)).T)

If I could convert feature_column into a tensor, I could do similar thing, but don't know how to convert

     user_fc = feature_column.categorical_column_with_vocabulary_list(
         'user', [3,5,7,1,2,0])
     user_embedding_column = feature_column.embedding_column(user, dimension=3)
     embedding = user_embedding.to_tensor() # if I can do this,  I could replace the embedding in the code above with this.


texts: Comparison of different ways of Upsampling in detection models

texts:
There are various ways to increase the resolution of tensor in (width, height) dimensions, frequently used in detection models like FPN, YOLO, e.t.c :

Upsampling (https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html) - which inserts intermediate points between those from previous layers, and the values in this intermediate points are obtained via bilinear intepolation or polynomial interpolation.
MaxUnpool (https://pytorch.org/docs/stable/generated/torch.nn.MaxUnpool2d.html) - which uses the results from the MaxPool layer in the encoding part of the network - location of the max values.
Deconvolution with stride > 1

The first strategy seems to copy the information from
the previous layer and transform it in some known in prior way.
Max Unpool works like in a similar way - uses the information from the previous layers, and transforms them in a certain way.
Deconvolution with stride in case it has learnable weights can do the increase of resolution in some priorly unknown way, with the trained weights, and seems to be a more flexible approach, but it increases total computational cost in a certain way.
Which of the aforementioned strategies perfoms better in most of the detection problems?


texts:
Full code source:
#Download COCO pre-trained weights
!wget --quiet https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5
!ls -lh mask_rcnn_coco.h5

COCO_WEIGHTS_PATH = "mask_rcnn_coco.h5"

model.load_weights(COCO_MODEL_PATH, by_name=True,
                       exclude=["mrcnn_class_logits", "mrcnn_bbox_fc",
                                "mrcnn_bbox", "mrcnn_mask"])
elif init_with == "last":
    # Load the last model you trained and continue training
    model.load_weights(model.find_last()[1], by_name=True)

Can I load my own "*.h5" file?
For example: I interrupted my kernel after 5 epochs. Can I load my last epoch?
Can You explain it for me?
It will be continue a process learning?


texts: Principal Component Analysis, Eigenvectors lying in the span of the observed data points?

texts:
I have been reading several papers and articles related to Principal Component Analysis (PCA) and in some of them, there is one step which is quite unclear to me (in particular (3) in [Schölkopf 1996]).

Let me reproduce their reasoning below.

Consider the centered data set $D = \{\textbf{x}_k\}_{k=1}^M$ with $\textbf{x}_k \in \textbf{R}^N$ and $ \sum_{k=1}^M \textbf{x}_k = 0$. PCA diagonalizes the (sample) covariance matrix

$$
C = \frac{1}{M} \sum_{j=1}^M \textbf{x}_j \textbf{x}_j^T. \tag{1}
$$

To do this we find the solution to the eigenvector equation

$$
\lambda \textbf{v} = C \textbf{v} \tag{2}
$$

for eigenvalues $\lambda \geq 0$ and eigenvectors $\textbf{v} \in \textbf{R}^N\backslash \{{0}\}$. As

$$
\lambda \textbf{v} = C \textbf{v} = \frac{1}{M} \sum_{j=1}^M (\textbf{x}_j^T \textbf{v}) \textbf{x}_j, \tag{3}
$$

all solutions $\textbf{v}$ with $\lambda \neq 0$ must lie in the span of $\textbf{x}_1, \dots, \textbf{x}_M$, hence (2) is equivalent to

$$
\lambda(\textbf{x}_k^T \textbf{v}) = \textbf{x}_k^T C \textbf{v}, \qquad \text{for } k = 1, \dots, M \tag{4}
$$

In (4), doesn't $\lambda(\textbf{x}^T \textbf{v}) = \textbf{x}^T C \textbf{v}$ hold for $\textbf{any}$ value of $\textbf{x}$? Why does (4) only hold when $\textbf{x} \in D$? I do not understand how their end up with (4).

Thanks.


texts: How to decide how many n_neighbors to consider while implementing LocalOutlierFactor?

texts:
I have a data set with rows: 134000 and columns: 200. I am trying to identify the outliers in data set using LocalOutlierFactor from scikit-learn. Although I understand how the algorithm works, I am unable to decide n_neighbors for my data set.

Kindly suggest.


texts: How to represent a document in test data with the Document-Term Matrix created from the training set?

texts: I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how can I represent a new document with the Document-Term Matrix since some terms might not be included in training data?