texts:
I'm trying to teach myself data science, with my particular interest being decision trees. A few steps in, I've come across a term, 'parameter convergence' that I can't find a definition for (because, after all, I'm learning on my own and have no access to teachers or peers):

  However, even in studies with much lower numbers of predictor
  variables, the combination of all main and interaction effects of
  interest – especially in the case of categorical predictor variables –
  may well lead to cell counts too sparse for parameter convergence. (from Strobl et al., 2009)

A web search isn't overly helpful because convergence is such a common term, and I'm not sure which results specifically apply in the context of decision trees. And also, the results don't provide an entry-level definition.

So, while a definition or explanation of parameter convergence (in the context of recursive partitioning) would be great, it would also be handy to be directed to a resource (academic or otherwise) that might have a 'glossary' of this and similar terms...


texts:
How can I plot a mixed graph in R?

I have used this but doesn't work (in bnlearn he calls 'arc' to a directed edge and 'edge' to an undirected edge):

library(bnlearn)
dag=empty.graph(nodes = c("E", "F"))
graphviz.plot(dag)
dag = set.edge(dag, from = "E", to = "F")
dag = set.arc(dag, to = "E", from = "F")
graphviz.plot(dag)

As specified in the documentation https://www.bnlearn.com/documentation/man/arcops.html it will overwrite the arc...
(Note that in bnlearn library he calls 'arc' to a directed edge and 'edge' to an undirected edge)

I want to be able to plot a mixed graph like:

(arc (directed edge) between E and F, also edge (undirected edge) at the same time).

(If R cannot do it, is there any other easy tool I could use?, my goal is just to plot)

edit: I just realized thatBayesian Networks need to be acyclic by definition, so I guess I am hopeless with the bnlearn library...


texts: How can I measure the similarity between 2 IP addresses? Is there any code to re-use?

texts: I need to measure the similarity of 2 IP addresses. I could not find any sample code in scala or other languages to find the distance between 2 IPs.


texts: How to estimate the lambda e to reweighing translation probabilities of entries in a dictionary

texts:
In this paper, Brown et al. proposed the usage of dictionary entries in addition to the machine translation probabilities that we're used to in the Math of Statistical Machine Translation.

He describes:

  We formalize an intuitive account of lexicographic behavior as
  follows. We imagine that a lexicographer, when constructing an entry
  for the English word or phrase e, first chooses a random size s, and
  then selects at random a sample of s instances of the use of e, each
  with its French translation. We imagine, further, that he includes in
  his entry for e a list consisting of all of tile translations that
  occur at least once in his random sample. The probability that he
  will, in this way, obtain tile list fi, ..., f,,, is

That much is understandable, and the Pr(f|e) in the formula is what we get from the normal machine translation probabilities. and then reweighted by the binomial recursion of finding the probability of the lexicographer choosing the sample sentence given the english word/phrase, i.e. Pr(s|e).

Then he went on to simply the formula and estimated a Poisson distribution instead of a binomial one and that will simplify the computation to:

Everything still looks okay but there's an additional variable exp^lambda(e) and that he explains is the mean of the Possion distribution of selecting the sample examples given the English source word.

So the question is how did he estimate that exp^lambda(e) from the corpus?


texts: Reinforcement learning for continuous state and action space

texts:
Problem

My goal is to apply Reinforcement Learning to predict the next state of an object under a known force in a 3D environment (the approach would be reduced to supervised learning, off-line learning).

Details of my approach

The current state is the vector representing the position of the object in the environment (3 dimensions), and the velocity of the object (3 dimensions). The starting position is randomly initialized in the environment, as well as the starting velocity.

The action is the vector representing the movement from state t to state t+1.

The reward is just the Euclidean distance between the predicted next state, and the real next state (I already have the target position).

What have I done so far?

I have been looking for many methods to do this. Deep Deterministic Policy Gradients works for a continuous action space, but in my case I also have a continuous state space. If you are interested in this approach, here's the original paper written at DeepMind:
http://proceedings.mlr.press/v32/silver14.pdf

The Actor-Critic approach should work, but it is usually (or always) applied to discrete and low-dimensional state space.

Q-Learning and Deep-Q Learning cannot handle high dimensional state space, so my configuration would not work even if discretizing the state space.

Inverse Reinforcement Learning (an instance of Imitation learning, with Behavioral Cloning and Direct Policy Learning) approximates a reward function when finding the reward function is more complicated than finding the policy function. Interesting approach, but I haven't seen any implementation, and in my case the reward function is pretty straightforward.
Is there a methodology to deal with my configuration that I haven't explored?


texts: Meaning of equation for CNN probabilities

texts:
So the first equation above refers to a CNN (rather a committee of CNNs) for image classification. I am unable to understand exactly what the author is trying to do in the first equation.

So far, I think they're calculating the index of max likehlihood probabilities for all committees, then adding up the probabilities for all committees for those indices, and finally taking the maximum index.

But this seems overly convoluted and I'm not really sure. Could someone clarify this?


texts: ANN Classifier for extracted discrete image features

texts:
I have a features extraction algorithm that works well to extract features from images. I want to develop an ANN to classify those images based on those features. I have extracted features in a csv file as columns and rows.
I know we can use CNN to extract and classify images but my scerio is only the second half (features classification).
Thanks


texts: Custom thresholds on categorical classification

texts:
When assessing a binary classification task, it is possible to search for particular threshold in order to have better score on some metrics (f1,recall,etc) through numerous methods. Unfortunately, it looks like the method cannot be applied on categorical classification (more than two classes) task. I've thought about training a Simple Classifier (SVC,Log,...,Tree) on top of a already trained model in order to find best thresholds to apply on outputs to maximize similarity of results.
My proposed workflow is to train a model on training set and evaluate it, then use outputs on training set of this model as entry for another model which may learn how to handle threshold to maximize similarity to ground truth on training set (all of those steps being validated using the same strategy).
I've never heard about it and I want to know whether it makes sense ?


texts: When to use linear or logistic regression?

texts:
Still I have no clear concept when I should choose linear OR Logistic regression.
 In addition, when I can say either linear or logistic regression performing better?

In particular, I want to know When should I choose Linear Regression and When to choose Logistic regression?


texts: Text Analysis to analyze qualitative (text based) Survey Results

texts: I ran a survey asking this question: 'What do you like the most about your favorite character in the game?' and I got 23K responses; most of them are phrases/sentences; what's the best way to analyze these results such as to find out what the main responses are? A simple word cloud may not do because it strips out the context by isolating words; but I'm open. Would love help. Didn't know how to attached the dataset (my first time on Stack-Ex) so [here][1] it is. Thanks in advance fellow coders/analysts.


texts: Multidimensional Scaling with Categorical Data

texts:
I have read the following about MDS in a book:

  using MDS requires an understanding of the individual feature's units;
  maybe we are using features that cannot be compared using the
  Euclidean metric. For instance, a categorical variable, even when
  encoded as an integer (0= circle, 1= star, 2= triangle, and so on),
  cannot be compared using Euclidean (is circle closer to star than to
  triangle?).

I accept the statement above, but it raises a few question about the application of MDA:

Given the fact that many conventional data sets contain categorical
features, does it mean that MDA cannot fit to these sets?
Maybe it would be a solution to change the distance measure type (e. g.
"Euclidean") to other, but Sklearn has no other built-in option,
not to mention R, where cmdscale has no option at all to specify
distance type. How to change this feature in general?

An additional question: I have read that PCA is a kind of MDS (or vice versa), apart from the fact that the former focuses on variance, the latter on keeping distance. Am I right that the two "converges" somehow (in case of visualization with the two first component, for instance)?


texts: How can I get the max value and corresponding index of a tensor simultaneously in Tensorflow?

texts: The only way I came out is using tf.argmax with tf.reduce_max. But I'm afraid of that it may be inefficient to find the max value twice.


texts: Data Mining Gear/Goods Websites for Specific Prices

texts:
This is a bit of a stretch topic, but I find myself often browsing gear websites (like, Gear Patrol and Uncrate) for birthday and gift ideas.  I find it cumbersome to click through each and every article looking for items that are either within my price range or within my price range && category of item that I'm looking for.

I know that I can go to Amazon and set specific filters for items that I am looking for, but I enjoy receiving suggestions from websites that are known to give quality product recommendations.

From a data mining / web scraping standpoint, does anyone know of any resources of tutorials online that provide guidance for similar products?

Is R or Python better suited for web scraping applications?

Any insight into my idea would be greatly appreciated :)


texts: Tokenization of data in dataframe in python

texts:
I am performing tokenization to each row in my dataframe but the tokenization is being done for only the first row. Can someone please help me. thank you.
Below are my codes:

import pandas as pd
import json
import nltk

nltk.download('punkt')
nltk.download('wordnet')
from nltk import sent_tokenize, word_tokenize

with open(r"C:\Users\User\Desktop\Coding\results.json" , encoding="utf8") as f:
     data = json.load(f)
df=pd.DataFrame(data['part'][0]['comment'])
split_data = df["comment"].str.split(" ")
data = split_data

print(data)

def tokenization_s(data): # same can be achieved for words tokens
    s_new = []
    for sent in (data[:][0]): #For NumpY = sentences[:]
        s_token = sent_tokenize(sent)
        if s_token != '':
            s_new.append(s_token)
    return s_new

print(tokenization_s(data))

My output is:

[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\User\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\User\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
0                             [enjoy, a, lovely, moment]
1      [I, was, there, for, my, honeymoon., The, hote...
2      [Had, an, amazing, stay, for, 2, nights.\nThe,...
3                 [Had, a, good, time., Food, is, good.]
4      [A, highly, recommendable, hotel., Value, for,...
                             ...
131    [Wonderful, experience,, a, quite, different, ...
132                            [Was, a, paradise, stay.]
133    [It, was, really, a, place, to, be, for, relax...
134    [It, was, just, perfect, with, an, excellent, ...
135                               [It's, was, excellent]
Name: comment, Length: 136, dtype: object
[['enjoy'], ['a'], ['lovely'], ['moment']]

Process finished with exit code 0

What should I do for the system to tokenise each row in the dataframe?


texts: Theoretical limits on correlation between classification algorithm performances

texts:
Are there any known theoretical bounds, or at least heuristic approaches, regarding the relation or correlation between the performances of any two different classification algorithms?
For example, would there exist binary classification datasets for which, say, $k$-nearest-neighbour classifiers would perform with say >90% accuracy, whereas say decision tree classifiers would do no better than 50-60%? (Accuracy here is measured by say $k$-fold cross-validation.)
It seems to me, at first glance, that a dataset which is able to achieve a very high accuracy on some classification algorithm would necessarily have some structure that would make it highly improbable that some other general classification algorithm would be able to perform very poorly. Yet it's also not impossible that there might be some 'exotic' type of dataset that does exhibit such a phenomenon.


texts: Autoencoder ambivalent about order of input data?

texts:
The problem I'm working to solve is this:

Given a musician's prerecorded free-form playing. I want to analyze each of the individual notes to determine how "in-rhythm" it is. See the graph in the screenshot.

The dark shading represents the beat intervals between every note in the vicinity of the one in question (plus or minus say, 10,000 ms) and every other in that vicinity. From this you are able to see peaks where common beat intervals lie and you can begin to deduce the length of a standard quarter note beat, etc.

The light thin lines represent the intervals between the note in question and those in the local vicinity. When I say I want to know how "in-rhythm" this note is, what I mean is, "how close are these lines to their nearest peak?"

It's basically a problem of cluster analysis. So, given a set of numbers (represented by the thin lines in the graph) how close are they as a set to the center of their respective clusters? It strikes me that this could be a good machine learning problem, so I've been attempting to approach it from that direction.

I was thinking a simple autoencoder would be suitable for the task. The input would be lists of numbers representing these beat intervals, but one thing I'm not sure about is this:

There's no telling how many of which peak/cluster any given note's beat intervals might belong to, so is there a way to keep the neural net ambivalent about the order in the list it finds these numbers? I don't want it determining that a certain note was more out of rhythm simply because it was missing an interval from a large cluster, etc.

It seems to me that order would probably always matter, so I'm at a bit of an impasse with my understanding. Am I right about this? Do you see some other solution?


texts:
Intro

I'm quite new to all the subjects in this question. This is my very first shot at Keras, Tensorflow, NNs or Time Series. If you're not a newbie, you'll notice that immediately.

Problem

I have 15 samples from an industrial process that reads temperatures, weights and heatflows every 2 seconds for roughly an hour (1876 rows). Samples look like this:

      temp   weight     heat
t
0  31.9661  9.09606 -0.834733
2  31.9905  9.09503 -0.839906
4  32.0967  9.09374 -0.854042
6  32.3021  9.09315 -0.877126
8  32.5371  9.09390 -0.917037

Each of the 15 samples has been analysed by an expert that has produced QA, rating 5 of them as lousy (0), another 5 as medium (5) and the third lot as perfect (10). My goal is to analyse further samples and produce an equivalent assessment.

Statistical analysis on the data revealed that the only feature showing differences is heatflow, so I'm focusing on that column. The column name is calor (heat in Spanish). I also detected very synchronous activity around the tails but big discrepancies over the middle section, so I'm also focusing on that section (rows 1250 to 3250).

Solution

I'm reading the CSVs into a dictionary and then passing it to a Keras network with the following code:

sequences = [data[k][['calor']].loc[1250:3250].values for k in data.keys()]
final_seq = np.array(sequences)

targets = [5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 10, 10, 10, 10, 10]
groups = [1, 1, 1, 2, 3, 1, 1, 1, 2, 3, 1, 1, 1, 2, 3]

train = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 1])
test = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 2])
validation = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 3])

train_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 1])
test_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 2])
validation_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 3])

# Building the model
model = Sequential()
model.add(LSTM(11, input_shape=train[0].shape))
model.add(Dense(11, input_shape=(11,), activation='softmax'))

adam = Adam(lr=0.001)

chk = ModelCheckpoint('best_model.pkl', monitor='val_accuracy', save_best_only=True, mode='max', verbose=1)

model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

train_binary = to_categorical(train_target)
validation_binary = to_categorical(validation_target)
model.fit(train, train_binary, batch_size=len(train), epochs=100, callbacks=[chk], validation_data=(validation, validation_binary))

model = load_model('best_model.pkl')
test_preds = model.predict_classes(test)
print(f'test_target: {test_target}\ntest_preds: {test_preds}')
acc = accuracy_score(test_target, test_preds)
print(f'Accuracy score: {acc}')

I get different results, but usually around the following:

Model testing...
test_target: [ 5  0 10]
test_preds: [10  0 10]
Accuracy score: 0.6666666666666666

Accuracy of 66% is the best I ever got so far.

Questions

Am I doing this right?

I'm not sure about some hyperparameters. I kind of arbitrated 100 epochs, the batch size as the total sample, the categorical_crossentropy loss function and the groupings as per the above groups list because of the targets (which are fixed).

I'm also not sure about the LSTM thing. My categories are not only numeric but ordinal. I could actually have a regression result here.

Should I even be using NN for this? Basically, any input is welcome.

I'm going to need much more data, correct?

I have 5 samples each. How many more should I have?

Thank you very much, experts!


texts: Machine Learning Algorithm for 'Performance Rating' to Employees

texts: Which Machine Learning Algorithm should i use for Assigning 'Performance Rating' to each Employee based on his LeaveDaysCount and LeaveExtensionPeriod(If Extended).


texts: Can this be a case of multi-class skewness?

texts:
I have been working on an email data set, and trying to predict the owner team for it. But my prediction accuracy is just 58%. I have implemented data cleansing, null value removals, duplicate removal, excluding stopwords, and then calculating tf-idf value to get my final document term matrix.

Tried various classifiers (but with default settings) but max I achieved is 58% accuracy. Might give a try to hyperparameter tuning.

Could you please give me some pointers, based on your previous experience what possibly could be going wrong? I'm simultaneously doing my own research and going through various blogs and articles.

One thought - Could it be a case of data skewness?

My output classes are distributed like this -

Your thoughts please?


texts: While I find a lot of material of SGD (Stochastic gradient descent), I am struggling to find one concrete example with numbers e.g. calculating it by hand for let's say, one iteration would help me a lot.


texts:
I have a CSV file with around 1 Million rows.
Let say its have details like

Name      |   Age   | Salary
name 1      52       10000
name 2      55       10043
name 3      50       100054
name 2      55       10023
name 1      52       100322...

and soon .

but i need to merge the redundant details .
and need a output like

Name      |   Age   | Salary
name 1      52       110322*
name 2      55       20066 *
name 3      50       100054

you might notice that the repeating Name 1 and Name 2 details are merged and the Salary values are added .So i'm looking for a way to apply this change to my original data set. so i need a python script to fix my problem .


texts: Visualizing the Naive Bayes descision boundry

texts:
I am trying to visualize the  Naive Bayes descision boundry where I am trainning a Naive Bayes on Breast Cancer Wisconsin (Original) Data Set . Here is the code  :
from matplotlib.colors import ListedColormap
X_set, y_set = X_train, y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, Model.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('red', 'green')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('Classifier (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

But I am getting this error :
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-35-60b675710c2f> in <module>
      3 X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
      4                      np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
----> 5 plt.contourf(X1, X2, Model.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
      6              alpha = 0.75, cmap = ListedColormap(('red', 'green')))
      7 plt.xlim(X1.min(), X1.max())

~\anaconda3\lib\site-packages\sklearn\naive_bayes.py in predict(self, X)
     76         check_is_fitted(self)
     77         X = self._check_X(X)
---> 78         jll = self._joint_log_likelihood(X)
     79         return self.classes_[np.argmax(jll, axis=1)]
     80

~\anaconda3\lib\site-packages\sklearn\naive_bayes.py in _joint_log_likelihood(self, X)
    454             jointi = np.log(self.class_prior_[i])
    455             n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))
--> 456             n_ij -= 0.5 * np.sum(((X - self.theta_[i, :]) ** 2) /
    457                                  (self.sigma_[i, :]), 1)
    458             joint_log_likelihood.append(jointi + n_ij)

ValueError: operands could not be broadcast together with shapes (257839,2) (9,)


texts: High train and val results. Bad test and predict results

texts:
For my thesis project I've been trying to make a CNN for some challenging data.
There's four classes with the following amount of images respectively
[410, 410, 269, 206] = 1,295 total.

Now I know that this is not perfect, both a small and a unbalanced dataset. I am using this tutorial as an example, resulting in this code:

# dimensions of our images.
img_width, img_height = 200, 200

top_model_weights_path = 'bottleneck_fc_model.h5'
train_data_dir = '/content/output/train'
validation_data_dir = '/content/output/val'
nb_train_samples = 760
nb_validation_samples = 240
epochs = 30
batch_size = 10

def save_bottlebeck_features():

    # build the VGG16 network
    model = applications.VGG16(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))

    train_datagen = ImageDataGenerator(rescale=1. / 255,
                                      horizontal_flip=True,
                                      vertical_flip=True)

    generator = train_datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        classes = folders,
        shuffle=False)

    bottleneck_features_train = model.predict_generator(generator, nb_train_samples // batch_size)
    np.save(open('bottleneck_features_train.npy', 'wb'),
            bottleneck_features_train)

    datagen = ImageDataGenerator(rescale=1. / 255)

    generator = datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        classes = folders,
        shuffle=False)

    bottleneck_features_validation = model.predict_generator(
        generator, nb_validation_samples // batch_size)

    np.save(open('bottleneck_features_validation.npy', 'wb'),
            bottleneck_features_validation)

save_bottlebeck_features()

#def train_top_model():
train_data = np.load(open('bottleneck_features_train.npy', 'rb'))
train_labels = to_categorical(np.array(
    [0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2)), 4)

print(train_data.shape)

validation_data = np.load(open('bottleneck_features_validation.npy', 'rb'))
validation_labels = to_categorical(np.array(
    [0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2)), 4)

print(validation_data.shape)

model = Sequential()
model.add(Flatten(input_shape=train_data.shape[1:]))
model.add(Dense(256, activation='relu',
                kernel_regularizer=regularizers.l2(0.0005),
                activity_regularizer=regularizers.l1(0.0005)))
model.add(Dropout(0.5))
model.add(Dense(4, activation='sigmoid'))

model.compile(optimizer='SGD',
              loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit(train_data, train_labels,
          epochs=epochs,
          batch_size=batch_size,
          validation_data=(validation_data, validation_labels),
          verbose=2)
model.save(top_model_weights_path)

Now my training and validation results are good:

But when I try to evaluate my model with test data which I held back, I get bad results:
Found 261 images belonging to 4 classes.

test acc: 0.458248479333526

test loss: 3.629304162353702

This is just a little better then guessing.

    .......precision    recall  f1-score   support

    0kpa       0.30      0.48      0.37        82
   40kpa       0.16      0.50      0.24        42
   80kpa       0.00      0.00      0.00        82
  160kpa       0.00      0.00      0.00        55

  accuracy                         0.23      261
  macro avg     0.12      0.24     0.15      261
  weighted avg  0.12      0.23     0.15      261

I notice that the last two classes get very bad results, what could cause this? I do shuffle the data before I split it up in train/val/test.
I also noticed that when i fed the model.evaluate_generator(..) with training data instead of test data it performed as bad.. that seems very odd to me..
Can anyone tell me what's going on!? Thanks in advance!


texts: One attribute includes another attribute

texts: I have a telecom dataset that has many attributes, among these attributes, there is "Voice mail plan" attribute that takes yes or no, and another attribute is "voice mail calls" which has many values, but always zero when "Voice mail plan" is no. When removing "Voice mail plan" from the dataset the accuracy of classifiers is lowered, so how can we inform the classifier that No is impeded in zero voice calls


texts: Imputing features with NA values in classification task

texts:
I currently have a dataset where each observation is a person's traffic ticket history over districts.
For each column, which represents a district:

1 represents that a person has received 1+ traffic violations in a district in 2018
0 if they have been to that district but have no traffic violations.
NA otherwise

GOAL:  to (rank districts to) see which district should have more police presence due to increased traffic violation and also use the features to predict whether or not that person has 1+ traffic accidents in 2019.
PROBLEM: not all people have been to every district. I currently just encode the value to 0 if the person has never been to that district. But this should be a valid NA value.
For example, it seems illogical to rank a district if only one person (in the dataset) has been to that district.
QUESTION(S): How exactly should I handle this? I don't think imputing as 0 is the right call here.
Original Data:
PersonId DistA DistB DistC DistD DistE Accident19
1        0     1     1     0      NA     1
2        NA    0     0     0      1      0
3        0     1     1     0      NA     1
4        1     0     0     0      NA     0

Imputed Data:
PersonId DistA DistB DistC DistD DistE Accident19
1        0     1     1     0      0     1
2        0     0     0     0      1     0
3        0     1     1     0      0     1
4        1     0     0     0      0     0

Many thanks in advance!


texts: Padding before or after truncation for stateful LSTM in Keras

texts:
I am training an LSTM on a dataset with variable timesteps (between 10 and 6000). Using the truncated backpropagation through time (TBPTT) technique, I am truncating the sequences to windows of 128 timesteps. Hence, sequences will have between 1 and 47 windows.
There are two possible ways I think I can do this.
One way is I can first pad all sequences with zeros (post-padding) to the 6000th timestep, and then truncate each sequence to 47 windows. Then, I can train a stateful LSTM on each window, resetting the LSTM state after all 47 windows:
def pad_and_truncate(X_batch, num_windows=47, windowlen=128, padvalue=0):
    X_padded = tf.keras.preprocessing.sequence.pad_sequences(
                                        X_batch,
                                        maxlen= num_windows * windowlen,
                                        dtype='int32',
                                        padding='post',
                                        value=padvalue)
    X_truncated = [X_padded[:, i*windowlen : (i+1)*windowlen] \
                   for i in range(num_windows)]

    return X_truncated

for batch in range(batches):
    X_batch, y_batch = getBatch(data, batch, batch_size) # Split entire dataset into batches
    X_truncated = pad_and_truncate(X_batch, num_windows=47,
                                   windowlen=128, padvalue=0)

    assert len(X_truncated) == 47

    for window in range(len(X_truncated)):
        model.fit(X_truncated[window], y_batch, batch_size=batch_size)
    model.reset_states()

Alternatively, I can first truncate each sequence to windows of size 128, and then only pad the last window with zeros. For example a sequence with 200 timesteps will be truncated to 2 windows (128 + 72 timesteps). Then, only the second window will be padded with 56 zeros (128 + 128 timesteps). Using this method, sequences will have varying window sizes (between 1 and 47), rather than all sequences having 47 windows as in the first method. The shortest sequence (10 timesteps) will only have 118 zeros padded to it rather than 6006 zeros.
for batch in range(batches):
    X_batch, y_batch = getBatch(data, batch, batch_size) # Split entire dataset into batches
    num_windows = np.ceil(X_batch.shape[0] / 128).astype(int)
    X_truncated = pad_and_truncate(X_batch, num_windows=num_windows,
                                   windowlen=128, padvalue=0)

    assert (1 <= len(X_truncated) <= 47)

    for window in range(len(X_truncated)):
        model.fit(X_truncated[window], y_batch, batch_size=batch_size)
    model.reset_states()

The first method (padding before truncation) seems very inefficient, since there will be a lot of zeros added to shorter sequences, and the second method (padding after truncation) seems more practical.
However, in a simple test with Keras LSTM models, the first method converged to a solution, though it was very slow (due to the larger number of windows). However, the second method did not converge at all despite the speed.
What would you suggest is the best way to apply TBPTT in dealing with very long sequences with keras LSTMs? Is there a more efficient way than padding all sequences to 6000 timesteps?


texts: Are there any objections to using the same (unlabelled) data for pre-training of a BERT-Based model and the downstream task?

texts: I'm looking to train an Electra model using unlabelled data in a specific field. Are there any objections to using the same data for unsupervised learning and then using the same data downstream for the supervised learning task?


texts: validation/training accuracy and overfitting

texts:
If we randomly split the data into training data and validation data, and assume the training data and validation data have similar "distributions", i.e. they are both good representations of the whole data set.

In this case, should the validation accuracy always be roughly the same as the training accuracy if there is no overfitting? Or is it possible that, for some cases, there could exist an "intrinsic" gap between the training and validation accuracy that is not due to overfitting or bad representation of the validation data?

If such "intrinsic" gap exists, how to tell if the gap between the training and validation accuracy is "intrinsic" or caused by overfitting?


texts: How to measure the performance of a deep learning model in parasite detection?

texts:
I’m working with segmented cells from thin blood smear images and using deep learning models to classify parasitic and uninfected cells.

I obtained the following values: Accuracy: 0.986, AUC: 0.99, Sensitivity: 0.981, Specificity: 0.992, F1-score: 0.987, MCC: 0.972, TP: 1329 , FN: 26, FP: 11, TN: 1364.

In one of my submissions, I have got a question from a reviewer like this:

Given a system (with p sensitivity, r specificity), how many RBCs need to be analyzed to confidently decide that a person is infected by malaria (given he has q% parasitemia).

With the above reasoning/study, report how many RBCs you need to analyze to confidently decide (statistically significant) a person (with q% parasitemia) is infected or not by the best system that you have developed. Also report by the same system, what is the lower level of parasitemia you can diagnose.

Can anyone suggest me an answer (assuming the degree of parasitemia) since we don't have that measure.


texts: What is the difference between all the different types of learning within machine learning?

texts:
This is a question that is really hard to google, and the differences are confusing. Does anyone have good examples of the differences between them all?

Supervised Learning
Semi-Supervised Learning
Distant Supervision
Active Learning
Lightly Supervised Learning
Weakly Supervised Learning
Competitive Learning
Reinforcement Learning
Unsupervised Learning


texts: Implementing sklearn.linear_model.SGDClassifier using python

texts:
I have an excel file that contains details related to determining the quality of a wine and I want to implement the linear model concept using the function sklearn.linear_model.SGDClassifier(SVM => Hinge loss) and (Logarithmic regression =>log loss) using python. I learned the basics about these function through the scikit learn website and I am not able to implement the model using excel file. I am very new to python and machine learning and I finding it hard to implement the model. I opened the excel file in python and tried to take two columns [randomly] from the file and use that as an input to call the fit function available in the model. But, I got an error stating Unknown label type: array. I tried a couple of other methods too, but, nothing worked. Can someone guide me with the implementation process?

from xlrd import open_workbook
from sklearn import linear_model
i = 0
fa = []
ph = []

book = open_workbook('F:/BIG DATA/winequality.xlsx')
sheet = book.sheet_by_name('Sheet1')
num_rows = sheet.nrows - 1
num_cols = sheet.ncols - 1
curr_row = 0
while curr_row <num_rows:
    curr_row += 1
    cell_val = sheet.cell_value(curr_row,0)
    cell_val1 = sheet.cell_value(curr_row,10)

    fa.append([float(cell_val),float(cell_val1)])
    cell_val2 = sheet.cell_value(curr_row,8)
    ph.append(float(cell_val2))

model = linear_model.SGDClassifier()
print(model.fit(fa,ph))

The error message screenshot:


texts:
This is a major confusion for me - I've always thought filters (those small sliding windows size 3x3 or 5x5) are strictly 2D, this among other is specified in Caffe API:

  kernel_size / kernel_h / kernel_w. The filter dimensions, given by
  kernel_size for square filters or kernel_h and kernel_w for
  rectangular filters

Now, I've never been 100% clear on what these convolutions mean (taken from Matt Zeiler's article (Visualizing and Understanding Convolutional Networks) and how they were obtained:

After searching it turned out that filters can be 3D?? How does it work then? I found a bit of information on CS 231n course, but it's not explained in detail anywhere.


texts: Is there a term for "this month last year" in a report?

texts: I'm building a report that has month over month data, but also "this month last year". Is there a better/standard way of describing this?


texts: Trying to find the correlation between inputs and output

texts:
I have tried the pandas code for trying to find out the correlation between the output and the inputs I am feeding. Here is the code:

dataframe.corrwith(dataframe['output']).plot(kind='barh',figsize=[20,10], legend=True,grid=True)

I got the following image:

I was trying to understand which column will affect the result more in the positive or negative or neutral way.

The above image I got, I am not able to conclude what exactly it means.
Can someone please tell me whether I am on the right direction of what I am trying to achieve? What is the meaning of the above image?
Let me know. Here is the link to the sample data set:  Training.csv


texts: Estimating the progress through a workflow with an arbitrary number of steps

texts:
I have a dataset where a record (a customer order, to choose a made-up example) goes through a set of steps until eventual completion. The set of steps is generally very similar, with small variations (and potential loops). For example:

Draft order created
Order paid for
Order queued for fulfillment
Order processed by team A
Order QAed by team B
Order ready for customer approval
Order updated by customer (eg. an extra product added to the cart)
Order re-processed by team C
Order QAed by team B
Order approved by QA team
Order approved by customer
Order delivered
Order closed

Certain properties of an order may trigger the inclusion of extra steps. Eg. if the user orders fresh food, there might be an extra 'refrigeration' step. If they order a car, there might be extra 'inflate tyres' and 'safety check' steps.
The problem: if an order has completed some (but not all) of the steps, I would like to be able to show a list of the probable remaining steps that it might go through. It's fine  to ignore any loops, in favor of displaying a simple linear list of steps.
I understand that this is a prediction problem, but most of the similar problems I've seen involve predicting a single outcome, and I'm not sure of the best way to approach this. Are there any standard techniques for predicting a sequence of events?


texts: I'm trying to quantify phasing... (unabashedly sports related)

texts:
I'm working to visualize NBA minutes wherein one player may have their playing minutes reduced, another player's minutes are increased.  I've got the data laid out in a matrix of actual minutes per game per player, with a little color for specific thresholds, but I feel like their might be a model to detect dips and peaks as they are relative to one another?

Any input is greatly appreciated :)


texts: Clustering with multiple distance measures

texts:
I'm trying to use clustering to automate a group-finding process with the aim of being able to automatically detect groups in unseen data. The data are html elements within any given webpage, this includes numerous features including spatial and visual information rendered from a web browser. The detected groups are usually semantically meaningful, for example a group of navigation links or a list of items in the main content area.

I've implemented a subtle distance measure that captures how similar or different these elements are and I'm passing these values to DBSCAN for clustering. So far it works quite well but I'm trying to think of ways to improve it.

The distance function is actually a weighted sum of 10 separate distance measures that capture different kinds of similarities and differences. I've weighted the more reliable and decisive measures higher than others. In theory, the measures that are weighted lower should only have an effect when they are congruent and are otherwise overpowered by the higher-weighted measures. Additionally each measurement function can itself amplify or attenuate it's own weight for a given item-pair, for example set its weight to zero if it is unsure and wants to abstain, or set it higher if it has found a similarity/difference that is particularly notable or reliable.

Although it's broadly working quite well, I've noticed that often one or two of the largest clusters should be further split. I've considered two methods for this: (a) attenuating the weights of measures that are mostly giving a low distance value within the cluster (thereby amplifying other measures with higher variance) and then rerunning DBSCAN or (b) searching clusters for subsets with large numbers of co-occurring features (e.g. some css class in the parent element and a particular font-size) and either splitting the cluster based on these, or adjusting the distance measures to reflect this apparent grouping and then rerunning the global DBSCAN with the adjusted values.

Currently it all works unsupervised but I'm open to manually creating a set of ground-truth data for improve it. Maybe then this could be used to optimize the weights in such a way that minimizes within-cluster distances and maximizes between-cluster distances? I also like the idea of running clustering multiple times, repeatedly adjusting the weights based on the variances of each measure within the cluster.

I'm looking for more ideas on how to optimize this, I would appreciate any input. I have seen some comments elsewhere that unsupervised clustering is only suitable for exploring data and not automating decisions, but I haven't been able to find a better way of finding these groups with my limited ML/data-science experience. Given all the time I've spent building this, the ideal thing would be to figure out a clever way of adaptively adjusting these weights in a more intelligent way.


texts: State-of-the-art methods for out-of-sample-extension

texts:
I'm using a kernel based dimensionality reduction algorithms, and interested in extending out-of-sample data points for further analysis.
I've been using the Nystrom method for this task, and some variants of it (see 1 and 2). I know also that there are works concerning the procedure of sampling for the Nystrom method. My question is - are there other methods, outside this scope, that perform better?


texts: Bandwidth of the gaussian kernels in t-sne

texts:
I'm trying to understand t-SNE better and I was hoping someone could elaborate on how the $\sigma _i$'s are chosen.

I was also wondering why they aren't just calculated in the normal way standard deviation is calculated from the data with $x_{i} $ as the expected value for each $\sigma _i$?


texts: Alternative regression model algorithms for machine learning

texts:
I am looking for not so known regression models and if possible a python library that implements it.
In my quiver I have:

Generalized Linear Models, Linear, Lasso, Ridge...
Decision Tree based model: CART, Random Forest, Gradient Boosting Decision trees(xgb,lightgbm,catboost)
Support Vector Machines
Knn Regressor
Neural Networks

Are there regression algorithm less known? I am looking for alternatives models to add to my quiver.


texts: Ensemble learning for multiple hypothesis classes

texts: Just to confirm if the following description falls in the category of ensemble learning. Suppose given a training set $D=\{(X,Y)\}$ we are asked to train a regressor. But now the way we do it is to train regressors on multiple hypothesis classes, say, one find the "closest" polynomial function of maximum degree 3, and another find the "closet" 2-layer neural network. And output the final result as, say, a weighted sum of these two. Can this also be called ensemble learning or there is some other terminologies used for it? The reason I feel confused is most of the ensemble learning algorithms I have seen assume one hypothesis space, like the final output is a weighted sum of decision tree, and what they do is just to simply adjust the weights?


texts: Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?

texts:
I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution and get its parameters. Can anyone please help me , how can I generate synthetic data from this approach ?
Here is my code snippet , which I tried.
   import scipy.stats as st
    def get_best_distribution(data):
        dist_names = ["norm", "exponweib", "weibull_max", "weibull_min", "pareto", "genextreme",\
                      "beta","invgauss","uniform","gamma","expon","lognorm","pearson3","triang"]
        dist_results = []
        params = {}
        for dist_name in dist_names:
            dist = getattr(st, dist_name)
            param = dist.fit(data)

            params[dist_name] = param
            # Applying the Kolmogorov-Smirnov test
            D, p = st.kstest(data, dist_name, args=param)
            #print("p value for "+dist_name+" = "+str(p))
            #print("D value for {} = {}".format(dist_name,str(D)))
            #print()
            dist_results.append((dist_name, p))

        # select the best fitted distribution
        best_dist, best_p = (max(dist_results, key=lambda item: item[1]))
        # store the name of the best fit and its p value

        print("Best fitting distribution: "+str(best_dist))
        print("Best p value: "+ str(best_p))
        print("Parameters for the best fit: "+ str(params[best_dist]))
        print()

        return best_dist, best_p, params[best_dist]

columns_list = list(synthetic_data.columns)

distribution_dict={}

for tmp_column in columns_list:

    sample_data=synthetic_data[tmp_column].astype(float)

    best_dist_sample, best_p_sample , params_sample = get_best_distribution(data=sample_data)
    distribution_dict[tmp_column] = tuple((best_dist_sample,best_p_sample,params_sample))

    pass


texts: Isn't the depth of a convolutional layer, the number of colors (or colorspace size)?

texts: I have been going through a CNN tutorial and noticed that depth of a convolutional layer is equal to the number of filters. But, shouldn't the depth be the number of colors in the image? I mean, if it's RGB then, depth is 3 right? Am I missing something here?


texts: Can't interpret the text information and ratings matrix imported to NN

texts:
I have a Recommender system which uses a Collaborative bayesian approach using pSDAE for recommending scientific articles from the Citeulike Dataset

The text information (as input to pSDAE) is in the file mult.dat and the rating matrix (as input for the MF part) is in the file cf-train-1-users.dat and is loaded using the following code:

def get_mult():
    X = read_mult('mult.dat',8000).astype(np.float32)
    return X

def read_user(f_in='cf-train-1-users.dat',num_u=5551,num_v=16980):
    fp = open(f_in)
    R = np.mat(np.zeros((num_u,num_v)))
    for i,line in enumerate(fp):
        segs = line.strip().split(' ')[1:]
        for seg in segs:
            R[i,int(seg)] = 1
    return R

The raw data is in proper Excel format with citations as doc-id, title, citeulike-id, raw-title, raw-abstract.

The mult.dat file containing hte text information looks like:

63 1:2 1666:1 132:1 901:1 1537:2 8:1 9:1 912:1

The trainusers.dat file looks like:

10 1631 3591 10272 14851 4662 13172 12684 5324 3595 3404

Here is the link to th ipynb for the whole Recommender system:

https://github.com/js05212/MXNet-for-CDL/blob/master/collaborative-dl.ipynb


texts: Bayes Optimal Decision Boundaries for Gaussian Data with Equal Covariance

texts:
I am drawing samples from two classes in the two-dimensional Cartesian space, each of which has the same covariance matrix $[2, 0; 0, 2]$.  One class has a mean of $[1.5, 1]$ and the other has a mean of $[1, 1.5]$.  If the priors are $4/7$ for the former and $3/7$ for the later, how would I derive the equation for the ideal decision boundary?

If it turns out that misclassifying the second class is twice as expensive as the first class, and the objective is to minimize the expected cost, what equation would I use for the best decision boundary?


texts: May I use the same data in several time series intervals?

texts:
I am playing with RNNs / LTSMs for a classification task in predicting financial data.

I have a time-series going many years back, and are planning to divide it into a number of shorter time-intervals that will be used to obtain a classification.

For example divide the data into random length intervals from 30 minutes to 24 hours, with a binary response 'higher' and 'lower' that answers if the value goes higher or lower for the next hours after the interval.

My question is if I can "reuse" the data, as in for example for a certain 24 hour period create the following intervals: 24 hours, 12 hours, 6 hours, 3 hours and 1 hour and train a model on all of these examples. If I do so, obviously the same data will be reused with different responses.

Or is it better to create a number of models; for example a separate model for 24 hour intervals than the 3 hour intervals?


texts:
I'm new to data science , while i'm implementing decision tree. I'm facing the following error. Where i went wrong;

Sample data in csv is:

x=dataset.iloc[:,:-1].values
y=dataset.iloc[:,:2].values
from sklearn.preprocessing import LabelEncoder,OneHotEncoder
from sklearn.compose import ColumnTransformer
labelencoder_x=LabelEncoder()
x[:,0]=labelencoder_x.fit_transform(x[:,0])
onehotencoder=OneHotEncoder()
columntransformer=ColumnTransformer([('dummy cols',onehotencoder,[0])],remainder='passthrough')
x=columntransformer.fit_transform(x)

from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.2,random_state=1)

from sklearn.tree import DecisionTreeRegressor
regressor=DecisionTreeRegressor(random_state=0)
regressor.fit(x,y)  #Error occurance

Error:
'''
ValueError: could not convert string to float: 'Business Analyst'
'''


texts:
I would like to implement custom regularizer in Keras/TF.
Regularizer is suppose to prevent rapid change between weights of the neighbouring layers.
Thus, it should have the following form:
$$
R(K) = \alpha \sum \|K_j - K_{j-1} \|_2^2,
$$
where $K_j$ and $K_{j-1}$ are weights of neighbouring layers.
What is correct way to implement this?
I had a look at regularizer class, but it seems that it has access only to weights of one layer.


texts: Naive Bayes for Categorical Features (Non Binary)

texts:
How do i use Naive Bayes Classifier (Using sklearn) for a Dataset
considering that my feature set is categorical, ie more than 2 categories per feature are present.

I've looked everywhere, some people have used GaussianNB even though
the data is categorical (1,2...8)

https://www.google.com/url?q=https://blog.sicara.com/naive-bayes-classifier-sklearn-python-example-tips-42d100429e44&sa=D&source=hangouts&ust=1567686518476000&usg=AFQjCNGORHakHbk2ZntzNZ1wsTYG6QUhig

Some people recommend converting it to One-Hot encoding and then using BernoulliNB which doesn't make sense to me because then the newly created features would have a high correlation which is against one of the core assumptions of Naive Bayes. (For example color feature has 3 values - Blue, Green, Red and we create 3 features out of it; then if Blue is 1 then its obvious that Red and Green will be 0. Hence the dependency)
Some people recommend using MultinomialNB which according to me doesn't make sense because it considers feature values to be frequency counts.

Can someone point me in the right direction?


texts: Data Interpretation with a double line graph

texts:
The atmosphere is made up of 4 distinct layers: the troposphere, stratosphere, mesosphere, and thermosphere. Different types of clouds form in the different layers depending on the pressure in the atmosphere and the ambi- ent temperature. The cloud types include nimbus, stratus, cumulus, and cirrus. Figure 1 shows the location of the
barriers of the atmosphere when the temperature and pressure are at an ideal condition for cloud formation. It also shows the different types of clouds formed at the different levels.

->How do you interpret this kind of graph? I sort of get it that as we go higher, pressure increases, while temperature decreases. But the graph looks weird to me and I can't seem to understand the right side of the graph, which sort of "shows that temperature also increases". Can anyone help me explain the concept behind this type of graphical representation? Thanks in advance.


texts: Keras Neural Network training is stuck (gets stuck around epoch 6)

texts:
I have a training dataset with 8000 rows, and I am trying to train a Keras Neural Network on it, using 100 epochs. However, the training process gets stuck around epoch 6 every time, as shown below. I'm not sure if it's because of my computer (8GB RAM Macbook Pro) or because of some inappropriately set parameters for my model. Thanks!

    import keras
    from keras.models import Sequential
    from keras.layers import Dense

    classifier = Sequential()

    classifier.add(Dense(activation="relu", input_dim=11, units=6, kernel_initializer="uniform"))

    classifier.add(Dense(activation="relu", units=6, kernel_initializer="uniform"))

    classifier.add(Dense(activation="sigmoid", units=1, kernel_initializer="uniform"))

    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

    classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 100)

8000/8000 [==============================] - 3s 348us/step - loss: 0.3890 - acc: 0.8367
Epoch 2/100
8000/8000 [==============================] - 3s 329us/step - loss: 0.3894 - acc: 0.8376
Epoch 3/100
8000/8000 [==============================] - 3s 314us/step - loss: 0.3887 - acc: 0.8370
Epoch 4/100
8000/8000 [==============================] - 3s 346us/step - loss: 0.3895 - acc: 0.8379
Epoch 5/100
8000/8000 [==============================] - 3s 320us/step - loss: 0.3887 - acc: 0.8370 - ETA: 2s - loss: 0.4087 - acc: 0.8077
Epoch 6/100
7650/8000 [===========================>..] - ETA: 0s - loss: 0.3881 - acc: 0.8363
//STUCK HERE

I do get the following message on my terminal window though (even though I'm using jupyter notebook):

2017-12-24 20:45:30.464660: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.


texts: Which are valid covariates in CausalImpact?

texts:
I am working lately with CausalImpact developed by Google. The paper described it is this one Inferring Causal Impact Using Bayesian Structural Time-Series Models

In short, what you can do with CausalImpact is study the effect of a specific event in your time series. In the above paper, they mention the importance of covariates or control groups where you use together with your time series.

  The first is the time-series behaviour of the response itself, prior
  to the intervention. The second is the behaviour of other time series
  that were predictive of the target series prior to the intervention.
  Such control series can be based, for example, on the same product in
  a different region that did not receive the intervention or on a
  metric that reflects activity in the industry as a whole.

However, I have seen a couple of blog posts around the web using totally unrelated time series as covariates. One of those is a study on the Daily Active Users in World Of Warcraft using page views of Wikipedia articles. This is the link www.datascience.com

Comparing to the original CausalImpact paper, it looks totally wrong, but taking into account the resource of the post (datascience.com from Oracle) and also other related examples confused me on what is exactly the right and wrong.


texts: What does the notation mAP@[.5:.95] mean?

texts:
For detection, a common way to determine if one object proposal was right is Intersection over Union (IoU, IU). This takes the set $A$ of proposed object pixels and the set of true object pixels $B$ and calculates:

$$IoU(A, B) = \frac{A \cap B}{A \cup B}$$

Commonly, IoU > 0.5 means that it was a hit, otherwise it was a fail. For each class, one can calculate the

True Positive ($TP(c)$): a proposal was made for class $c$ and there actually was an object of class $c$
False Positive ($FP(c)$): a proposal was made for class $c$, but there is no object of class $c$
Average Precision for class $c$: $\frac{\#TP(c)}{\#TP(c) + \#FP(c)}$

The mAP (mean average precision) = $\frac{1}{|classes|}\sum_{c \in classes} \frac{\#TP(c)}{\#TP(c) + \#FP(c)}$

If one wants better proposals, one does increase the IoU from 0.5 to a higher value (up to 1.0 which would be perfect). One can denote this with mAP@p, where $p \in (0, 1)$ is the IoU.

But what does mAP@[.5:.95] (as found in this paper) mean?


texts: Factors in choosing a continuous encoder?

texts:
Apart from heuristically treating an encoder like a hyperparameter, how can one decide which encoder to use for continuous features?

Rule out encoders that do/ don't accept negative values.
Can we look at stdev and skewness to determine which encoder would provide the best transformation?


texts: Timeseries LSTM: does test data need to come after training data?

texts:
I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).
Because the time series is very long, I plan to extract shorter, partially-overlapping subsequences and use each of them as one training input for the LSTM.
In my train/validation/test split, do I have to use older subsequences for training and newer for validation and test? Or can I treat them as if they were independent samples and just randomly shuffle them, given that anyway the LSTM will start each subsequence with empty memory?
The reason I ask is because I noticed that, due to how the timeseries was collected, the first half contains mostly behaviour A while the second half mostly behaviour B. This would cause training to be mostly on A and testing mostly on B, which does not reflect the fact that, in production, the system will see both periods of predominant A and periods of predominant B.


texts:
I'm having difficulties to wrap my head around how I can prepare my dataset to train an LSTM.
Below is a screenshot of a subset representation of my dataset.

There are several other feature not included in this screenshot. The last column is inhospital_mortality which is 0 or 1 for each row.
Each feature was taken at a certain time x. features with the same feature_1,2,3 were taken at the same time.
My idea is that I will need to break every row (sample) as the example below: So in this case, each row would become 6 new rows.
| tc_tb1 | spo2_tb1 | g1_tb1| inhospital_mortality (label 0 | 1) |
| tc_tb2 | spo2_tb2 | g1_tb2| inhospital_mortality (label 0 | 1) |
| tc_tb3 | spo2_tb3 | g1_tb3| inhospital_mortality (label 0 | 1) |
...
| tc_tb6 | spo2_tb6 | g1_tb6| inhospital_mortality (label 0 | 1) |

Am I correct here? If so, how could I accomplish this dataframe manipulation in a more straight forward way? Perhaps there's a better way to reshape turn this dataset into the format I want. I wasn't able to accomplish it.
Thank you.


texts: Deeplearning without an objective function?

texts:
In this article, the author talks about how deeplearning models no longer are trained for an objective function that humans specify, but find their own objective function. Specifically, he is talking about GANs.

Is there a good resource explaining this idea that GANs find their own objective function? Based on what I've read about GANs, I don't think of them this way.


texts: Python - Todoist API - Store data into Dictionary

texts:
I've this Python code that allows me to collect data from Todoist API. Now I want to store this data into a Dictionary in order to make some data processing and some advanced analytics. Using this code how can I store this values into the structure ?

from pytodoist import todoist
user = todoist.login('####','""""')
projects = user.get_projects()
for project in projects:
    print(project.name)

tasks = user.get_tasks()
for task in tasks:
    print("Project: ", task.project.name, ", Task: ", task.content, ", Date: ", task.date_string, ", Priority: ", task.priority, ", Added in: ", task.date_added)

My key_pais: The strings that I've in For loop "Project", "Task", etc.
Values: the values from taks

Many thanks!


texts:
I'm training deep learning network using images (to be exact - I'm solving semantic segmentation problem).

What's the proper order of resizing (I need to resize images to fixed width X height) and normalization (dividing by 255 value) of images in preprocessing?

Does it make more sense to do first resizing and then normalization? Or first normalization and then resizing?


texts: if for same input values we have different class labels, Should we have to remove those training examples before building any model on it?

texts:
Ex:

 A1  A2  A3  Class attribute
---------------------------
 X   Y   Z    A

 X   Y   Z    B

 P   Q   R    C

Should we remove first and second training examples.

If yes, why? If we remove those we are loosing information that if inputs are (X,Y,Z) then class may be A or B but not C

If no, why not?


texts: What is the difference between active learning and reinforcement learning?

texts:
From Wikipedia:

Active learning is a special case of machine learning in which a learning algorithm can interactively query a user (or some other information source) to label new data points with the desired outputs.

Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment in order to maximize the notion of cumulative reward.

How to distinguish them? What are the exact differences?


texts: What is the difference between lightgbm.LGBMModel and lightgbm.LGBMClassifier?

texts:
https://lightgbm.readthedocs.io/en/latest/Python-API.html

I have compared both of them in the lightgbm documentation and could not figure out which one I would choose and why I would choose one over the other?


texts: What is the difference between nDCG and rank correlation methods?

texts:
When do we use one or the other?

My use case:

I want to evaluate a linear space to see how good retrieval results are. I have a set of data X (m x n) and some weights W (m x 1). I want to measure the nearest neighbour retrieval performance on W'X with a ground truth value Y. This is a continuous value, so I can't use simple precision/recall.

If I use rank correlation, I will find the correlation between retrieved Ys and retrieval rank. If I use nDCG, I will use sorted Y to compute IDCG.

I would like to compare this to the correlation value I get when I change Y also. (For example, Y could be head pose angle in one case and age in another)


texts: Research in random forest algorithms able to switch data sets

texts: I'm curious as to whether research been done into random forests that combine unsupervised with supervised learning in a way allowing a single algorithm to find patterns in, and work with, multiple different data sets. I have googled every possible way to find research on this, and have come up empty. Can anyone point me in the right direction?


texts: Tfidfvectorizer in R for character n-grams

texts:
Is there a similar tool as the tfidf vectorizer which converts a collection of raw documents to a matrix of TF-IDF features in a R package?

See this for reference.


texts: Keras LSTM using timesteps = 1 and train_on_batch

texts:
I'm using an LSTM to achieve a classification problem. I have a dataset composed by sentences, each sentence is composed by a variable number of words and I have to predict a label for each word of each sentence.

For example, I have a dataset of shape (300000, 800), so 300000 words and each word is made of 800 features (word embeddings, etc...).
Moreover, each sentence is divided in subtrees, which each subtree is a syntactic dependency tree between words. My task is to learn the dependencies between words of a subtree.

I thought to train the network using timesteps=1 and  the train_on_batch function, where the batch size is variable and equal to the subtree cardinality.

Is it a reasonable process?


texts:
This is perhaps not an ideal question by StackExchange rules, but I don't know where to find a more informed community, so I'll attempt to make it as specific as possible.

I'm trying to wrap my brain around 'data analysis', and I'm looking for attempts to analyze the field, to perhaps create a taxonomy of the analytic process, or in some other way to impose an order on 'analysis', where analysis might include both descriptive and predictive modeling.

In the field of data graphics, there are several well known examples: One could look at Visualizing Data by Cleveland, in which he organizes graphical techniques by dimensionality, or more broadly, at Bertin's Semiology of Graphics, or Wilkinson's The Grammar of Graphics.

In a related domain (a sub-domain of analysis, actually), Codd's relational algebra attempts to organize the filtering and summarization of tabular data. (I'm very much focused on tabular data.)

What I'm asking for are pointers to a proposed "grammar" of analytics.


texts: What makes columnar databases suitable for data science?

texts: What are some of the advantages of columnar data-stores which make them more suitable for data science and analytics?


texts: Correlation between products based on purchases placed around the same date

texts:
Association rule learning has a fair bit of material based around the correlation of products purchased on the same order/at the same time. However I'd like to discover if there is a method for identifying such a relationship between products that are ordered near each other, but not together.

Say for example a customer purchases a pencil in week one, but later purchases an eraser in week two. Then a year later they do the same thing. But then also apply this across all customers who show similar behaviour. I would then aim to combine this with the correlation coefficient of products purchased in an order together.

I considered giving each of the dates before or after a purchase a diminishing weight (1/n days) and then comparing the aggregate date value of the customer purchase date history with a product A against product B. But this seems like a fairly drawn out method.

Just looking for a process to achieve this, but python hints would also help.

Thanks in advance.


texts: What type of neural network should I use to detect meteors in images?

texts:
I am currently building a project that takes fisheye images from cameras and detects whether the picture contains a meteor, and if it does it tries to identify where the meteor is.

The images look like this:

I am currently building a project that takes fisheye images from cameras and detects whether the picture contains a meteor.

The images look like this:

The 'circle' in the middle is the actual sky, everything outside it is just black pixels

The meteor is the long flash of light in the middle, the small flashes of light on the 'sides' are lights that come from the cities nearby.

Using opencv's HoughLines I was able to create a program that detects the meteors, however it also detects the lights on the 'sides', so my idea was to train a neural network to only detect the meteors.

How would I go about doing this? I was thinking of using a U-Net neural network, using the images and masks of the images where only the meteor is 'detected' for training, sort of like this example.


texts: How to overcome training example's different lengths when working with Word Embeddings (word2vec)

texts:
I'm working on Sentiment Analysis over tweets using word2vec as word representation.

I have trained my word2vec model. But when I'm going to train my classifier, I'm facing the issue that every tweet has different length and the classifier (RandomForest) needs all examples to be of the same size.

Currently, for every tweet I'm averaging the vectors of all of its words, to end up with a vector representation of the tweet.

For example:
My word2vec model represents each word as vectors of size 300.

I have Tweet1 formed by 10 words, and Tweet2 formed by 5 words.

So what I do is, for Tweet1

(v1_Tweet1 + v2_Tweet1 + ... +v10_Tweet1)/10 = v_Tweet1 #avg vector of 300 elements.

For Tweet2:

(v1_Tweet2 + v2_Tweet2 + ... +v5_Tweet1)/5 = v_Tweet2 #avg vector of 300 elements.

*Being v1_TweetX the vector of the first word of the TweetX and so on.

This works 'fine' but I would like to know what other approaches do you take to overcome the different sizes on the train and text examples for the classifier.

Thanks.


texts: Implementing "full convolution" to find gradient w.r.t the convolution layer inputs

texts:
I've been trying to implement "full convolution" w.r.t to convolution layer inputs. According to this article, it looks like this:

So, I wrote this function:

def full_convolve(filters, gradient):
    filters = np.ones((5,5))
    gradient = np.ones((8,8))
    result = list()
    output_shape = 12
    filter_r = filters.shape[0] - 1
    filter_c = filters.shape[1] - 1
    gradient_r = gradient.shape[0] - 1
    gradient_c = gradient.shape[1] - 1

    for i in range(0,output_shape):
        if (i <= filter_r):
            row_slice = (0, i + 1)
            filter_row_slice = ( 0 , i + 1)
        elif ( i > filter_r and i <= gradient_r):
            row_slice = (i - filter_r, i + 1)
            filter_row_slice = (0, i + 1)
        else:
            rest = ((output_shape - 1) -  i )
            row_slice = (gradient_r  - rest, i + 1 )
            filter_row_slice = (0 ,rest + 1)
        for b in range(0,output_shape):
            if (b <= filter_c):
                col_slice = (0, b + 1)
                filter_col_slice = (0, b+1)
            elif (b > filter_c and b <= gradient_c):
                col_slice = (b - filter_c, b + 1)
                filter_col_slice = (0,b+1)
            else:
                rest = (output_shape - 1 ) - b
                col_slice = (gradient_r - rest , b + 1)
                filter_col_slice = (0, rest + 1)
            r = np.sum(gradient[row_slice[0] : row_slice[1], col_slice[0] : col_slice[1]] * filters[filter_row_slice[0]: filter_row_slice[1], filter_col_slice[0]: filter_col_slice[1]])
            result.append(r)
    result = np.asarray(result).reshape(12,12)

I tested this with ones and the output seems correct (if I get "full convolution" right):

[[ 1.  2.  3.  4.  5.  5.  5.  5.  4.  3.  2.  1.]
 [ 2.  4.  6.  8. 10. 10. 10. 10.  8.  6.  4.  2.]
 [ 3.  6.  9. 12. 15. 15. 15. 15. 12.  9.  6.  3.]
 [ 4.  8. 12. 16. 20. 20. 20. 20. 16. 12.  8.  4.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 5. 10. 15. 20. 25. 25. 25. 25. 20. 15. 10.  5.]
 [ 4.  8. 12. 16. 20. 20. 20. 20. 16. 12.  8.  4.]
 [ 3.  6.  9. 12. 15. 15. 15. 15. 12.  9.  6.  3.]
 [ 2.  4.  6.  8. 10. 10. 10. 10.  8.  6.  4.  2.]
 [ 1.  2.  3.  4.  5.  5.  5.  5.  4.  3.  2.  1.]]

However, I don't like all these manual checks and if/else statements. I feel there is a better way to implement this in NumPy (perhaps, using some zero paddings or something like this). Can anyone suggest a better approach? Thanks


texts: A/B testing with non-Gaussian distributions

texts:
I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the distributions are not normal and I'm interested in the maximum change, not the mean values! So if all you know is how CLT is gonna make everything Gaussian, this is a good point to stop and move on to the next question.
The data are distances, so there's a minimum of 0, but there's no max and no guarantee what the distribution is going to look like. As an example, the histograms look like this:

My gut feeling tells me that the maximum of orange sample is just randomly higher than the blue one, but gut feelings are usually wrong. So I want to have some method of testing. I would appreciate any input.

PS: Welch's t-test tells me that with 100.000% confidence, these two distributions are different, but are they?


texts: Unsupervised Clustering for n-length word arrays

texts:
I have a series of arrays

[Apple,Banana,Cherry,Date]
[Apple,Fig,Grape]
[Banana,Cherry,Date,Elderberry]
[Fig,Grape]

and I would like to build some clusters that associate the arrays into groups based on overlap

Group1: Array1 and Array3 as they have 3 common words
Group2: Array2 and Array4 as they have 2 common words
etc..

I was thinking kmeans but there is really not a distance calculation - more like an overlap one.

Does anyone have a suggestions?

Thanks!


texts: When is there enough data for generalization?

texts:
Are there any general rules that one can use to infer what can be learned/generalized from a particular data set?  Suppose the dataset was taken from a sample of people.  Can these rules be stated as functions of the sample or total population?

I understand the above may be vague, so a case scenario: Users participate in a search task, where the data are their queries, clicked results, and the HTML content (text only) of those results.  Each of these are tagged with their user and timestamp.  A user may generate a few pages - for a simple fact-finding task - or hundreds of pages - for a longer-term search task, like for class report.

Edit:  In addition to generalizing about a population, given a sample, I'm interested in generalizing about an individual's overall search behavior, given a time slice.  Theory and paper references are a plus!


texts: TF-IDF Augmented Frequency vs Cosine Normalization

texts:
I am using TF-IDF for text classification and have been curious about the following two concepts.

The augmented term frequency which is basically used for weighting in order to eliminate the bias towards longer documents.

On the other hand, there is a cosine normalization which appears to be done for the same purpose.

Are they both similar? Can we use any one of them to eliminate the bias towards larger documents?


texts: Why is the testing set standardized using fit values from the training set

texts:
I was wondering, in the feature scaling part of data preprocessing, why is the data in the testing set standardized using the fit values from the training set? Why aren't the fit values recalculated separately for the testing set and then used for transforming the testing set? Here is the code I am using as reference:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
x_train[:, 3:] = sc.fit_transform(x_train[:, 1:3])
x_test[:, 3:] = sc.transform(x_test[:, 1:3])

In this example, x_train is the training set and x_test is the testing set.

Thanks!


texts: How come same cluster category be separated?

texts:
I have these 200 vectors which were clustered using K-means clustering based on keywords weight similarity that was given by TF-IDF (Term Frequency - Inverse Document Frequency). The vectors were clustered with respect to the vectors in four cities which are Amsterdam, Rotterdam, The Hague and, Utrecht. I have chosen k-cluster centroid = 6, which means I have cluster 0 to cluster 5. On each cluster, I also calculated the average number of keyword's numerical weight so that then I got the most relevant and least relevant set of keywords just like the picture below:

Both relevant keywords and least relevant could help the interpretation of what is the cluster about. For example, cluster 0 is related to rail transportation because in the most relevant keywords include tram, line, tramway, station, and rail. And the least relevant keywords emphasized the interpretation of cluster 0 where the keywords include photography, bicycle, iphoneography, green, nature, and flower.
And I have the cluster map of all six clusters in the Amsterdam city shows in this picture:

The problem is that in Amsterdam city, there is no cluster 0, which related to rail transportation. In my analysis opinion, it is because all the vectors that relate to rail transportation were clustered to cluster 3, which is also related to rail transportation (based on my interpretation on most relevant and least relevant keywords on both clusters). Cluster 3 also related to rail transportation because in the most relevant keywords include tram, line, tramway, station, and rail. And the least relevant keywords emphasized the interpretation of cluster 0 where the keywords include photography, bicycle, iphoneography, green, nature, and flower.

There is also the evidence that cluster 3 could not be found in Rotterdam and The Hague city because all vectors related to rail transportation in both cities were all clustered to cluster 0. Below you could find the pictures of the cluster map in both cities:

My question is whether my analysis could be justified? But how come two clusters with the same theme could be separated? why they do not cluster together?


texts: How does multi-head attention on "multiple attention axes" works?

texts:
I would like to apply an self-attention mechanism on a multichannel audio spectrogram, so a 3D tensor. In the original Transformer paper, self-attention is applied on vector (embedded words) within a kind of temporal sequence. On my multichannel spectrogram, I would like to apply self-attention both on the temporal and frequency axes, so that the analyzed vectors are "through" the channel axes.
On tensorflow.keras MultiHeadAttention layer, there is a attention_axes parameter which seems to be interested for my problem, because I could set it up to something like (2,3) and hope attention will be applied on the wanted dimensions. However I don't understand how it works since it's different from the original Transformer paper, and I don't find any relevant paper addressing self-attention in several dimensions in the same manner.
Also the source code doesn't help me, the algorithm is split into several sub-modules which are not self-explanatory to me.
Any insights would be precious!
Thanks a lot


texts: What has to be considered when creating a radar chart?

texts:
I'm currently analyzing a WhatsApp chat history. One thing I'm interested in is the time when the two people communicated. I thought this is a perfect use case for a radar chart (aka spider chart, star chart). So here is one example:

I find this super hard to read. In contrast, have a look at the following bar chart with exactly the same information:

Here it is way easier for me to see patterns:

0 - 5: No activity (sleeping)
6 - 9: A morning peak (messages after waking up)
10 - 16: Little activity (work times)
17 - 19: Increased activity (work is done)
20 - 21: High activity
22 - 23: Sinking activity (going to bed)

Question

What are typical use-cases for radar charts? Are there specific requirements on the data? When are they better than bar charts? How should I set the ticks for radar charts?

Before people starting to close-vote: I could imagine very well that there are similar studies like for color maps - how many errors are done in analysis? How fast can people pick up main insights?

Example code

In case you want to improve the existing visualizations:

# core modules
from math import pi

# 3rd party modules
import matplotlib.pyplot as plt
import pandas as pd

def main():
    df = pd.DataFrame({'date': [209, 13, 1, 2, 1, 25, 809, 3571, 1952, 1448, 942, 1007, 1531, 1132, 981, 864, 975, 2502, 2786, 2717, 3985, 4991, 2872, 761]},
                      index=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23])
    create_bar_chart(df)
    create_radar_chart(df)

def create_bar_chart(df, output_path='bar_chart.png'):
    df.plot(kind='bar')
    plt.savefig(output_path)

def create_radar_chart(df, output_path='radar_chart.png'):
    """
    Create a radar chart.

    Parameters
    ----------
    df : pandas.DataFrame
        Has a column 'date'
    """
    values = df['date'].tolist()

    df = df.T.reset_index(drop=True)
    df.insert(0, 'group', 'A')

    # number of variable
    categories = list(df)[1:]
    N = len(categories)

    # What will be the angle of each axis in the plot?
    # (we divide the plot / number of variable)
    angles = [n / float(N) * 2 * pi for n in range(N)]
    angles += angles[:1]

    # Initialise the spider plot
    ax = plt.subplot(111, polar=True)

    # Draw one axe per variable + add labels labels yet
    plt.xticks(angles[:-1], categories, color='grey', size=8)

    # We are going to plot the first line of the data frame.
    # But we need to repeat the first value to close the circular graph:
    values = df.loc[0].drop('group').values.flatten().tolist()
    values += values[:1]
    values

    # Plot data
    ax.plot(angles, values, linewidth=1, linestyle='solid')

    # Fill area
    ax.fill(angles, values, 'b', alpha=0.1)
    plt.savefig(output_path)

if __name__ == '__main__':
    main()


texts:
I have a merchants dataset with 800,000 samples and 18,000 labels. Each sample is associated with a single label and the labels are independent.

An example sample looks like

description, label
int'l 0028240525 amazon uk retail amazon.co.uk, Amazon

In addition to the existing samples there will be new retailers added to the dataset. In this case there may well only be a single sample for that new retailer.

To summarise, I need a classifier that

handles a large number of labels (~18,000, independent, single label per sample)
is able to classify undersampled labels (i.e. a single retailer)

Is there an approach that will handle both? Perhaps two separate classifiers makes more sense?


texts: Dealing with diverse groups in regression

texts:
What happens if a certain dataset contains different "groups" that follow different linear models?

For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we can see that some points follow a linear relationship with a coefficient $\beta_A<0$ while other points clearly have $\beta_B>0$. We can infer that these points belong to two different populations, population $A$ responds negatively to high values of feature $x_i$ while population $B$ responds positively. We then create a categorical feature (or one hot encoding) to show which population each row belongs to.

Is splitting the dataset required or are commonly used algorithms able to recognize the different relations between features from different categorical variables?


texts: How to further improve the kaggle titanic submission accuracy?

texts:
I am working on the Titanic dataset. So far my submission has 0.78 score using soft majority voting with logistic regression and random forest. As for the features, I used Pclass, Age, SibSp, Parch, Fare, Sex, Embarked.

My question is how to further boost the score for this classification problem?

One thing I tried is to add more classifiers for the majority voting, but it does not help, it even worthens the result. How do I understand this worthening effect?

Thanks for your insight.


texts: Fastest way for 1 vs all lookup on embeddings

texts:
I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array.

I wish to compare a new unseen text to all the 1 000 000 pre-computed embeddings and perform cosine similarity to retrieve the most semantic similar document in the corpus.

What is the most efficient to perform this 1-vs-all comparison?

I would thankful for any pointers and feedback!


texts:
I would like to try and create my own audio dataset which I can then use to train machine learning models for classification. The data that I've gathered consists of multiple long audio files of around 1 hour each, containing audio of different categories. Since this is my first time working with audio files instead of data in a tabular format, I am a bit lost on how the do the labeling/preparation.

Most information I find on the internet is mainly related to applying machine learning models to existing pre-labeled datasets. I am hoping to find some more information specifically about what would be the best way to approach the labeling and, if possible, also some information regarding the division of the long audio files into much shorter audio snippets.


texts: How would gather requirements for a data science project? How do you convince stakeholders that whatever you develop is going to benefit them? How to build trust to make use of data science for the business operations?


texts:
I understand Random Forest models can be used both for classification and regression situations.

Is there a more specific criteria to determine where a random forest model would perform better than common regressions (Linear, Lasso, etc) to estimate values or Logistic Regression for classification?


texts: How to reduce the error in linear regression

texts: In linear regression, how can we minimize the error term?


texts: Standardization on training and split data

texts:
I am confused on which of the following should be used for standardization:

method 1: fit transforming training data and only transforming test data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

method 2: fit transforming both training and test data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
# scaler_train=sc.fit(X_train)
#X_train_sd=scaler_train.transform(X_train)
X_test = sc.fit_transform (X_test)
#scaler_test=sc.fit(X_test)
#X_test_sd=scaler_train.transform(X_test)

this is a follow up question to:
StandardScaler before and after splitting data


texts: How do attention mechanisms in RNNs learn weights for a variable length input

texts:
Attention mechanisms in RNNs are reasonably common to sequence to sequence models.

I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output vectors from the encoder network. This is used to produce a new input vector.

What I don't understand is that the learned weight vectors $\alpha$ must be a fixed size vector because it's treated as learned weights, but it's applied to a variable length sequence.

If someone could help me understand this particular mechanism I'd appreciate it.


texts: SNIP (single-shot network pruning) definition in SynFlow paper

texts:
I am trying to understand how the SNIP saliency metric definition in the original paper is equivalent to the definition in the SynFlow paper on neural network pruning.
The former defines saliency $\mathbf{s} = s_{j \in \left[1, params \right]}$ as follows:
$$
\Delta L_{j}(\mathbf{w} ; \mathcal{D}) \approx g_{j}(\mathbf{w} ; \mathcal{D})=\left.\frac{\partial L(\mathbf{c} \odot \mathbf{w} ; \mathcal{D})}{\partial c_{j}}\right|_{\mathbf{c}=1}=\left.\lim _{\delta \rightarrow 0} \frac{L(\mathbf{c} \odot \mathbf{w} ; \mathcal{D})-L\left(\left(\mathbf{c}-\delta \mathbf{e}_{j}\right) \odot \mathbf{w} ; \mathcal{D}\right)}{\delta}\right|_{\mathbf{c}=\mathbf{1}}
$$
$$
s_{j}=\frac{\left|g_{j}(\mathbf{w} ; \mathcal{D})\right|}{\sum_{k=1}^{m}\left|g_{k}(\mathbf{w} ; \mathcal{D})\right|}
$$
while the latter defines it as:
$$
\mathbf{s} = \left|\frac{\partial \mathcal{L}}{\partial \theta} \odot \theta\right|
$$
In the first, the gradients are over 1's multiplied with the weights, not over the weights themselves.


texts: Set seed for a Class that calls Keras Models

texts:
I have class that I use to optimize parameters of a Keras LSTM model. It is known that to set seed for keras, one must input the follow on its code. But what I'm not understanding is where to put it in the case of a class that will build and modify the models.
from numpy.random import seed
seed(1)
from tensorflow.random import set_seed
set_seed(2)

Should it be in the __init__ like bellow?
from numpy.random import seed
from tensorflow.random import set_seed

class OptimizeLSTM:
    def __init__(self, X_train, y_train, X_test, y_test, verbose=False):
        self._X_train = X_train
        self._y_train = y_train
        self._X_test = X_test
        self._y_test = y_test
        self._verbose = verbose
        seed(1)
        set_seed(2)


texts: Dummy Variables of Weights in RNN Backpropagation Through Time

texts:
In the deep learning book RNN chapter (https://www.deeplearningbook.org/contents/rnn.html), it is mentioned that -

To resolve this ambiguity, we introduce dummy variables $W^{(t)}$ that are deﬁned to be copies of W but with each $W^{(t)}$ used only at time step t. We may then use $∇_{W^{(t)}}$ to denote the contribution of the weights at time step t to the gradient

What is the purpose of defining such a dummy variable?
Moreover, it seems like in the equation (10.26) in the book,
$∇_{W^{(t)}}h(t)$ = $diag(1-(h^{(t)})^2)h^{(t−1)^{T}}$.
Shouldn't it be $∇_{W^{(t)}}h^{(t)}$ = $diag(1-(h^{(t)})^2)(h^{(t−1)^{T}}+ W ∇_{W^{(t)}}h^{(t−1)})$ instead? Some backpropagation derivations, infact, apply chain rule in this manner (e.g. https://arxiv.org/pdf/1610.02583.pdf). What is the difference between such derivations with the derivation depicted in the deep learning book?


texts: Which Policy Gradient Method was used by Google's Deep Mind to teach AI to walk

texts:
I just saw this video on Youtube.
Which Policy Gradient method was used to train the AI to walk?
Was it DDPG or D4PG or what?


texts: Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

texts:
The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function.

It seems to be very simple, you just have to act greedily, by maximizing at each step the value function.

In the case of a small grid world, once you have applied the Policy Evaluation algorithm, you get for example the following matrix for the value function :

You start from the up-left corner and the unique actions are the four classic displacements.

If we respect a greedy evolution, by maximizing at each step the value function, we gonna go right and then down, right, up ... never reaching the state with the best value (20).

So the question is simple : why don't we have to guess what will be the future rewards to get the optimal policy from the optimal value function ? In other words, why is one-step look ahead enough to get the optimal policy ?

A similar problem already appear with the rewards, and we solved this issue by weightening a sum of future expectations... Why don't we need to do the same ?

Could you help me understand what's wrong with my interpretation ?

Thanks a lot for your help !


texts: Recurring events - finding in a time series

texts:
I have an event dataset from which I would like to detect recurring events (i.e: weekly, bi-weekly, monthly). The dataset contains:

Timestamp (date)
Event type which can get any value (e.g: purchase, withdrawal, inquiry, bid, etc...).
Amount in USD (can be bid or price for example)

My goal is to find and label recurring events and their frequency. Most of the time there's one event per day, but sometimes there are two or more events for the same day (can be with the same event type, but doesn't have to be). I should also be able to account for a change in recurrence (from bi-weekly to weekly or vice-versa).

How would you approach this?

Any help would be appreciated!


texts: Different significant variables but same Adjusted R-squared value

texts:
I performed a multiple linear regression on 64 variables with 3 different models:

Performed Multiple Linear Regression on all 64 variables
Perform Feature Selection with Random Forest and then perform multiple linear regression on selected features
Performed Stepwise Linear Regression

I achieved the same adjusted R squared value for all 3 models but different significant variables. How should I make sense of this? Which model should I go with?

Will appreciate any advice! Thank you!


texts: Is Minimax Linkage a Lance-Williams hierarchical clustering?

texts:
I found the following article on "Hierarchical Clustering With Prototypes
via Minimax Linkage".

It is stated in Property 6 that

  Minimax linkage cannot be written using Lance–Williams updates.

A succinct proof using a counter-example is given:

  Proof.
  Figure 9 shows a simple one-dimensional example that could not
  arise if minimax linkage followed Lance– Williams updates. The upper
  and lower panels show two configurations of points for which the
  right side of (4) is identical but the left side differs; in
  particular, $d(G_1 \cup G_2,H) = 9$ for the upper panel, whereas
  $d(G_1 \cup G_2,H) = 8$ for the lower panel.

But I do not understand their proof. For both cases (upper and lower panels), $d(G_1,H) = 16$, $d(G_2,H) = 7$, $d(G_1,G_2) = 5$.

I cannot see any reason that $\alpha(G_2)$ in the first case must equal $\alpha(G_2)$ in the second case. For instance, $G_2$ has not the same cardinal.


texts: What is the shape of conv3d and conv3d_transpose?

texts:
I want to do a GAN with coloured pictures. This means I need a three dimensional input and therefore I like to use conv3d and conv3d_transpose.

Unfortunately in the TensorFlow documentation, I can't find any formula for the output shape. Can anyone give me a hint on how to find the shape of the function's results?


texts: What makes a problem good for an evolutionary strategy vs a genetic algorithm vs particle swarm optimization?

texts:
I understand that evolutionary strategies (ES), genetic algorithms (GA), and particle swarm optimization (PSO) are all algorithms used to solve optimization types of problems, but what might make an optimization problem better to choose one of those techniques over the other?

For example, I read in an article that genetic algorithms are typically preferred for combinatorial problems, but other than that one sentence or so, I haven't been able to find any source that explains when you might choose one of those techniques over the other.

Is there a general approach one might use when trying to decide which technique to use, or is it more of a guess and check type of situation? It would be great if someone could help me understand when each of the three cases might be preferred.


texts: Career transition into bioinformatics and data science? - Postdoc or no postdoc

texts:
I am close to completing my PhD in Nutritional Sciences. Pre-PhD I did research in developmental biology, tissue engineering and molecular biology in general. To say the least, my background is very broad.

Since the beginning of my PhD program, I realized I wanted to pursue a career in data science. Throughout my PhD I have invested innumerable hours to acquire skills in programming and data analysis (understanding HPC environments, coding in R, Python, specific bioinformatic tools for my area of work and some others outside my area of work that I learned but have not applied in my own work). I have a big passion for my research topic but no interest in developing an academic career.

I took a long time to complete my Phd in part because of the extra skills I was acquiring and because of personal reasons. My original plan was to obtain a postdoctoral position at THE leading academic research laboratory in my field of study, the interview process has been working out for me but if I got it I know it would be an amazing opportunity that could come at a very high economic and personal cost since it involves a move cross country and postponing the opportunity to earn more money for my family in the short term.

I am starting my job search in industry and looking at what prospects I actually have for a bioinformatics career in industry since my PhD is not in a Bioinformatics or Computational Biology program.

For those of you who have careers in Bioinformatics, my questions are:

How can I overcome the lack of a Bioinformatics PhD in my current applications for industry jobs?
I am a little insecure since not all my bioinformatics skills are sharp since I have not had the chance to apply all of them to my current research, but I am confident I would be able to tackle any challenge I was given. It may take longer for me than for others, but I would get it done and I am willing to invest the time doing the work to get to proficiency.

If given the opportunity of a postdoc at the lab with the biggest name in the field, would it be "worth" the sacrifices and investment required? or should I just jump into industry if given the chance and start building a career there.

Any advice is welcome!
Thanks for reading :)