texts: How to deal with missing data for Bernoulli Naive Bayes?

texts:
I am dealing with a dataset of categorical data that looks like this:

    content_1   content_2   content_4   content_5   content_6
0         NaN         0.0         0.0         0.0         NaN
1         NaN         0.0         0.0         0.0         NaN
2         NaN         NaN         NaN         NaN         NaN
3         0.0         NaN         0.0         NaN         0.0

These represent user downloads from an intranet, where a user is shown the opportunity to download a particular piece of content. 1 indicates a user seeing content and downloading it, 0 indicates a user seeing content and not downloading it, and NaN means the user did not see/was not shown that piece of content.

I am trying to use the scikit-learn Bernoulli Naive Bayes model to predict the probability of a user downloading content_1, given if they have seen downloaded / not downloaded content_2-7.

I have removed all data where content_1 is equal to NaN as I'm obviously only interested in data points where a decision was actively made by the user. This gives data as:

    content_1   content_2   content_3   content_4   content_5   content_6
0         1.0         NaN         1.0         NaN         NaN         1.0
1         0.0         NaN         NaN         0.0         1.0         0.0
2         1.0         0.0         NaN         NaN         NaN         1.0

In the above framework, NaN, is a missing value. For data points where a Nan is present, I want the algorithm to ignore that category, and use only those categories present in the calculation.

I know from these questions: 1, that there are essentially 3 options when dealing with missing values:

ignore the data point if any categories contain a NaN (I.e. remove the row)
Impute some other placeholder value (e.g. -1 etc.) or
Impute some average value corresponding to the overall dataset
distribution.

However, these are not the best option for the following reason:

Every single row contains at least 1 NaN. This means, under this
arrangement I would discard the entire dataset. Obviously a no go.
I do not want the missing value to add to the probability
calculation, which will happen if I replace Nan with say -1. I'm also using a Bernoulli Naive Bayes, so as I understand, this requires singly 0 or 1 values.
As this is categorical data, it does not make sense for me to do this,
in this way (it was either seen or not, and if not, it is not needed).

The answer here indicated that the best way to do this, is, when calculating probabilities, to ignore that category if it is a missing value (essentially you are saying: only compute a probability based on the specific categories I have provided with non missing values).

I do not know how to encode this when using the scikit-learn Naive Bayes model, whether to do this as a missing value.

Here's what I have so far:

df=pd.read_clipboard()
from sklearn import datasets
from sklearn.naive_bayes import BernoulliNB
# Create train input / output data
y_train = df['content_1'].values
X_train = df.drop('content_1', axis=1).values
# Loud Bernoulli Naive Bayes model
clf = BernoulliNB()
clf.fit(X_train, y_train)

Obviously, this returns an error because of the present NaNs. So how can I adjust the scikit-learn Bernoulli model to automatically ignore the columns with NaNs, and instead take only those with 0 or 1?

I am aware this may not be possible with the stock model, and reviewing the documentation seems to suggest this. As such, this may require significant coding, so I'll say this: I am not asking for someone to go and code this (nor do I expect it); I'm looking to be pointed in the right direction, for instance if someone has faced this problem / how they approach it / relevant blog or tutorial posts (my searches have turned up nothing).

Thanks in advance - appreciate you reading.


texts: Extracting data, sorting and removing duplicates from an array using python 2.7

texts:
I have a text file data set in the form of
[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]

I want to extract the complete data from it, sort it and remove duplicates of the same (i.e., the same row should not appear again).
The output should look like (in rows):
11090702
11090404
07060202
09090606
09090604

I want this data for machine learning algorithm implementation.


texts: Can linear classifiers be used at each node of a decision tree instead of the lines parallel to any one of the axes?

texts: I am relatively new to AI/ML. I came across this question while reading some content on ML. Would be of great help if anyone can answer this


texts: Feature extraction from relational database

texts:
Inorder to build a classifier, I need to extract a few features from the data stored on a MySQL database. I need to join multiple tables and it is taking a lot of time. I have joined 2 tables at one time and has got results in multiple csvs. I need to combine them. Writing a script will be the best option?
How do people extract features from large relational databases? Am I missing something?

Thanks


texts: Differences between applying KMeans over PCA and applying PCA over KMeans

texts:
Short question:

As stated in the title, I'm interested in the differences between applying KMeans over PCA-ed vectors and applying PCA over KMean-ed vectors.

Long question:

Let's suppose we have a word embeddings dataset. Each word in the dataset is embeded in R300.

We want to perform an exploratory analysis of the dataset and for that we decide to apply KMeans, in order to group the words in 10 clusters (number of clusters arbitrarily chosen).

After doing the process, we want to visualize the results in R3. We could tackle this problem with two strategies;

Strategy 1 - Perform KMeans over R300 vectors and PCA until R3:

Apply KMeans to the R300 embeddings.
Perform PCA to the R300 embeddings and get R3 vectors.
Plot the R3 vectors according to the clusters obtained via KMeans

Result: http://kmeanspca.000webhostapp.com/KMeans_PCA_R3.html

Strategy 2 - Perform PCA over R300 until R3 and then KMeans:

Perform PCA to the R300 embeddings and get R3 vectors.
Apply KMeans to the R3 embeddings.
Plot the R3 vectors according to the clusters obtained via KMeans

Result: http://kmeanspca.000webhostapp.com/PCA_KMeans_R3.html

Are there any differences in the obtained results? Any interpretation?

In case both strategies are in fact the same. Why is that?


texts:
I am currently working on a project in which I have to develop a model to predict how much money other companies will make by using the services provided by my company. The money made is a type of tax return. The model should predict a range (e.g 100k$-200k$).
Basically, if I give the model the data of the target company, it will give me how much they can potentially make with us.
The data I have is the financial statements of the companies we worked with. This includes their general and financial data such as capital, number of employees, city, type of industry, and of course the amount of money they made with us.
After researching online, I found that most solutions are forecasting and time series based. While the data I have has the year in which we worked with the clients, I do not think this is the right approach for me. I experimented with decision trees and a regression model but I do not seem to get a good result.
Any tips on where to look to solve this problem?


texts: Deep Q-Learning with large number of actions

texts:
I'm using DQN with large number of actions in [0, 10000, step = 1000]. This means I have an action space of size 11 (including 0 and 10000). Action space is still discrete. My problem is that, instead of choosing sequence of best actions, DQN is selecting zero as best action at every training epoch, which is not desired. At the starting of training exploration rate is one, as training goes on it decreases by 0.99. Any help in this is appreciated.

import random
from collections import deque
from keras.models import Sequential, load_model, model_from_json
from keras.layers import Dense
from keras.optimizers import sgd
import os
class DQNAgent:
    def __init__(self, state_size, action_size):
        self.state_size = state_size
        self.action_size = action_size
        self.memory = deque(maxlen=2000) # memory size
        self.gamma = 0.95    # discount rate
        self.epsilon = 1.0  # exploration rate
        self.epsilon_min = 0.01 # minmun exploration rate
        self.epsilon_decay = 0.99 # exploration decay
        self.learning_rate = 0.001 # learning rate
        self.model = self._build_model()

    def _build_model(self):
        # Neural Net for Deep-Q learning Model
        model = Sequential()
        model.add(Dense(256, input_dim=self.state_size, activation='relu'))
        model.add(Dense(256, activation='relu'))
        model.add(Dense(self.action_size, activation='linear'))
        model.compile(loss='mse',
                      optimizer=sgd(lr=self.learning_rate))
        return model

    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    def act(self, state):
        if np.random.rand() <= self.epsilon:
            return random.randrange(self.action_size)
        act_values = self.model.predict(state)
        return np.argmax(act_values[0])  # returns action

    def replay(self, batch_size):
        minibatch = random.sample(self.memory, batch_size)
        for state, action, reward, next_state, done in minibatch:
            target = reward
            if not done:
                target = (reward + self.gamma *
                          np.amax(self.model.predict(next_state)[0]))
            target_f = self.model.predict(state)
            target_f[0][action] = target
            self.model.fit(state, target_f, epochs=1, verbose=0)
        if self.epsilon > self.epsilon_min:
            self.epsilon *= self.epsilon_decay


texts:
I am trying to compute Percentage Contributions on a pandas dataframe. I have a data frame which has three columns (Buyer, Product, CountofProductBought).

I create a new dataframe using groupby at Buyer level and using sum on the Count of Products Bought. I use the as_index=False.

Finally, I merge this data frame with the original data frame to compute the % [count of Product Bought/ SUM of count of Product Bought].

I get the following error when I call pd.merge:

  The column label 'Buyer' is not unique. For a multi-index, the label
  must be a tuple with elements corresponding to each level.

Please help as there is no concrete help on this.


texts:
I am new in Keras and would want to apply a neural network on this dataset:
https://www.drivendata.org/competitions/57/nepal-earthquake/

I have proprocessed the dataset transforming categorical variables into numerical with pd.get_dummies pandas method. Also, target (that is 1, 2 or 3 - depending on the damage grade) is tranformed into three columns indicating the probability of being one of those values.

The NN I wrote is simple, but I don't understand what are the values that predict method returns.

# first neural network with keras tutorial
from numpy import loadtxt
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf
import keras.backend as K

# define the keras model
model = Sequential()
# 12 8 3 Accuracy: 57.08
# 25 12 3 Accuracy: 56.89
model.add(Dense(25, input_dim=68, activation='relu'))
model.add(Dense(12, activation='relu'))
model.add(Dense(3, activation='softmax'))
# compile the keras model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[get_f1])
# fit the keras model on the dataset
model.fit(X, y, epochs=5, batch_size=10)

predicted = model.predict(test)

This is the output for the first individual:

array([0.09522187, 0.57914054, 0.32563758], dtype=float32)

Any idea?

PD: If you request some additional information or code, please let me know :)

Thanks a lot!!!


texts: How to normalized term vector for document clustering?

texts:
I have over a million text documents that I would like to cluster. I used tf-idf modeling and term vector cosine for identifying similar documents in the corpus, which appeared to work well.

Some documents are annotated with issues labels (e.g., a,b,c,d – twenty in all).  Each issue has as least 100 documents.

I would like to compute the average document for each cluster. What would be the best approach to normalization in this context? Right now, I am normalizing using $0.5 + \big{(}0.5 * \frac{term}{\max{term}}\big{)}$. Would it be better to compute the average of the normalized vectors instead? Alternatively, I could compute the sum of all raw term frequencies and then normalize that. What is the best approach to create a normalized term vector for each issue?


texts: Train and predict on different data structures

texts:
Is it possible to train a Machine Learning model with a set of the same data structure, but use another data structure as input in prediction/classification ? If so, how ?

Let me elaborate and explain my motivation :
I am a summer intern in a research lab (and will be starting my last year in Mathematical Engineering, so I am not too deep into ML yet, I will be specializing this year). I have followed the work of a previous intern on Chronicle Mining.
A Sequence is a list of timestamped events, e.g.
<({A}, 1), ({A, C}, 2), ({A, B}, 4), ({B, C}, 7)>
is a sequence in which the event A happens at time 1, the events A and C happen at time 2, the events A and B happen at time 4 and the events B and C happen at time 7.
A Chronicle is a data structure based on graphs. It translates time constraints between events, or in my case, sequences of events.

This tells us that a first A event must come between 1 and 3 units of time before the sequence B, C, but B, C must come between 1 and 3 units of time before a second A event and between -2 and 2 units of time before A, C (so A, C can actually come before B, C). A first A event must come between 1 and 4 units of time before A, C, and you get the rest.
So as you can see, it is a directed graph, there can be mutliple nodes with the same label (which are sequences of events), multiple vertices with the same label (which are time intervals) and there can be negative values within the intervals. In my case, events are partially ordered (in this example it is lexicographical order) to make some things simpler.
I have accomplished efficient extraction of these weird things from sequential databases, so let's say I have a lot of them. In my case, there is always a final state (like the middle A in the example), and it is always the same (all sequences end with the same event/sequence).
Now, my tutor would like me to "perform Machine Learning" on these graphs. But there is a catch : the prediction/classification input has not the same shape as the training input. He would like for me to feed the algorithm a simple sequence (not a chronicle) and output some results. Thus, I have come up with two problems :

Given a large set of chronicles and a partial sequence (without the common final state), predict the sequence (and intervals between each event) between the last event on the sequence and the common final state.
Given a large set of chronicles and a partial sequence (without the common final state), tell whether the next k-th step will be the common final state.

Both are similar, but one is a prediction problem and the second one is a classification problem. However, in both cases, I don't know where to start !
I have read papers on k-th steps sequence prediction with LSTM NN and other techniques, but what gets me is the fact that I apparently have to train my model with chronicles, but predict using sequences. Is that even possible ? Do I have to transform my chronicles in some kind of way ?


texts: How to user Keras's Embedding Layer properly?

texts:
I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the TensorFlow se2seq machine translation tutorial in Keras). My questions are the following:

I understand that Embedding layers turn word values in a sentence into fixed-dimension-long representation. But I observe two distinct usage of Embedding layers: one on one hand (like this tutorial on Keras Blog) utilizes external pre-trained word2vec vectors via the weights parameter:

from keras.layers import Embedding

embedding_layer = Embedding(len(word_index) + 1,
  EMBEDDING_DIM,
  weights=[embedding_matrix],
  input_length=MAX_SEQUENCE_LENGTH,
  trainable=False)`

while in other cases there is no such an external output but users just leave to the Embedding layer to decide the representation vectors. I don't understand what is the real difference between these approaches regarding the desired outcome? Maybe the internal-only solution is not a semantic representation? What is the point of applying embedding layer to an external matrix of which the rows already have fix length?

Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer? Am I correct guessing that this set to True let the Embedding layer fine-tune the imported word2vec weights to take the actual training examples into consideration?

Further, how to instruct Embedding layer to properly encode "metacharacters"? Setting the mask_zero parameter True it can incorporate padding zeroes but what about UNK (unknown), EOS (End of Sentence)? (By the way, I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input...)

And finally: how could a model predict the translation of a word which is not represented in the training set? Is it tries to approximate it with the "closest" one in the vocabulary?


texts: Why use deep neural networks over methods like linear regression or SVM?

texts: This is a very broad question, but I was wondering why researchers would choose a deep neural network over linear regression or SVM? As in, what are the advantages and disadvantages of both?


texts: running model.evaluate many times results different accuracy and loss value tensorflow 2

texts:
I have trained a CNN network using dataset = tf.data.Dataset.from_tensor_slices((data, label)) to create the dataset. training went well but evaluating the model on test dataset results different values every time without changing anything in test dataset or the network and I am not using any Dropout or Batchnormalization:

If necessary my code:

 model = tf.keras.Sequential([
    Input((1,30,30)),
    Conv2D(filters = 8, kernel_size=(3,3), padding="same", activation="relu", name="c1", data_format="channels_first"),
    Conv2D(filters = 16, kernel_size=(3,3), padding="same", activation="relu", name="c2", data_format="channels_first"),
    MaxPool2D(pool_size=(2,2), strides=(1,1),padding="same", name="m1", data_format="channels_first"),

    Conv2D(filters = 16, kernel_size=(3,3), padding="same", activation="relu", name="c3", data_format="channels_first"),
    MaxPool2D(pool_size=(2,2), strides=(1,1),padding="same", name="m2",data_format="channels_first"),

    Flatten(),
    Dense(256, activation="relu", use_bias=True),
    Dense(5,  use_bias=True)])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=["accuracy"])
model.fit(train_data, verbose=1, validation_data=valid_data, epochs=20)

model.evaluate(test_data)

How I made the Dataset:

def split_dataset(dataset: tf.data.Dataset, validation_data_fraction: float):

    validation_data_percent = round(validation_data_fraction * 100)
    if not (0 <= validation_data_percent <= 100):
        raise ValueError("validation data fraction must be ∈ [0,1]")

    dataset = dataset.enumerate()
    train_dataset = dataset.filter(lambda f, data: f % 100 >= validation_data_percent)
    validation_dataset = dataset.filter(lambda f, data: f % 100 < validation_data_percent)

    # remove enumeration
    train_dataset = train_dataset.map(lambda f, data: data)
    validation_dataset = validation_dataset.map(lambda f, data: data)

    return train_dataset, validation_dataset

def load_data(path):
    data, label = data_prep(path)
    dataset = tf.data.Dataset.from_tensor_slices((data, label))
    dataset = dataset.shuffle(100000)
    train_dataset, rest = split_dataset(dataset, 0.3)
    test_dataset, valid_dataset = split_dataset(rest, 0.5)
    train_data = train_dataset.shuffle(1000).batch(10)
    valid_data = valid_dataset.batch(10)
    test_data = test_dataset.batch(10)
    return train_data, valid_data, test_data

for example running model.evaluate(test_data) gives:

885/Unknown - 2s 2ms/step - loss: 0.1039 - accuracy: 0.9663
885/Unknown - 2s 2ms/step - loss: 0.0959 - accuracy: 0.9675
885/Unknown - 2s 2ms/step - loss: 0.0999 - accuracy: 0.9661
885/Unknown - 2s 2ms/step - loss: 0.0888 - accuracy: 0.9688
885/Unknown - 2s 2ms/step - loss: 0.0799 - accuracy: 0.9715


texts: What is meant by outliers in text data set. How to detect them?

texts: I know that outliers are present in data but their behaviour varies a lot from remaining data points. But today while learning about naive-Bayes  they mentioned that naive-Bayes can affected by the outliers. But which points in data set are termed as outliers and how do we identify them?


texts: Best practice - memory management when data wrangling

texts:
I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:

df.read_csv() #one instance of a df
df1 = df.drop_duplicates #another instance of a df, total 2
df1_melt = df1.melt() #another instance of a df
df1_aggregated = df1_melt.groupby()... #another instance, total 4
df_mutual = pd.merge(df1_aggregated, df1_melted) #created another instance of a df, total 5

In the example above, we created 5 dataframes and stored them in memory, but we're really only interested in one. I have read that you can reduce the reference count to these variables to 0, and then when you delete the variable, it may be garbage collected. Is there a better way to program in a manner that I avoid creating multiple copies of the data in memory?


texts: Why does this model print Incompatible dilated conv1d layers?

texts:
I was trying to see the layers used in a Wavenet model for speech generation and I can't seem to make sense of the output layers printed by the TF model. Model is this: https://github.com/Rayhane-mamah/Tacotron-2/tree/master/wavenet_vocoder. Paper that introduced this is https://arxiv.org/pdf/1609.03499.pdf, and it's supposed to operate on a sequence of output-freq spectrum frames (80 values per frame).

Print gives me this (only shown first_conv layer and 2 out of ~30 dilated conv layers):

(Pdb) p model
WaveNet(
  (first_conv): Conv1d(1, 512, kernel_size=(1,), stride=(1,))
  (conv_layers): ModuleList(
    (0): ResidualConv1dGLU(
      (conv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(2,))
      (conv1x1c): Conv1d(80, 512, kernel_size=(1,), stride=(1,))
      (conv1x1_out): Conv1d(256, 512, kernel_size=(1,), stride=(1,))
      (conv1x1_skip): Conv1d(256, 256, kernel_size=(1,), stride=(1,))
    )
    (1): ResidualConv1dGLU(
      (conv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(4,), dilation=(2,))
      (conv1x1c): Conv1d(80, 512, kernel_size=(1,), stride=(1,))
      (conv1x1_out): Conv1d(256, 512, kernel_size=(1,), stride=(1,))
      (conv1x1_skip): Conv1d(256, 256, kernel_size=(1,), stride=(1,))
    )
....

The first_conv layer is supposed to operate on a freq spectrum each frame of which has 80 values. How can then Conv1d(1, 512) seem to be operating on just 1 input-channel?

Also, The ResidualConv1dGLU that are dilated convs, each have 4 different types of conv1d operating on different input channels. How could these layers make sense as consecutive convs due to their mismatching input/output channels. How can the ResidualConv1dGLU block make sense with respect to input/outputs?

     (conv): Conv1d(512, 512, kernel_size=(3,), stride=(1,), padding=(2,))
      (conv1x1c): Conv1d(80, 512, kernel_size=(1,), stride=(1,))
      (conv1x1_out): Conv1d(256, 512, kernel_size=(1,), stride=(1,))
      (conv1x1_skip): Conv1d(256, 256, kernel_size=(1,), stride=(1,))


texts: Switching Keras backend Tensorflow to GPU

texts: I use Keras-Tensorflow combo installed with CPU option (it was said to be more robust), but now I'd like to try it with GPU-version. Is there a convenient way to switch? Or shall I re-install fully Tensorflow? Is the GPU version reliable?


texts: Why does my model only converge when I add a MaxPooling with stride of 1 layer at the beginning?

texts:
I have a model that takes an input of inertial sensor data collected at 60 hertz and outputs one of two classes. I have broken the data into 1 min snapshots which seems to be appropriate. I have split the data into training and validation sets. As a baseline, 64% of my data is class 1 versus class 2. I am using keras + tensorflow.

I have tested out two first layers. One is a Conv1D with a kernel_size of (6,) [model performs at baseline] and the other is a MaxPooling1D with a pool_size of (1,) [performs near ideal at close to 92%]. My working assumption is that a MaxPooling layer of kernel size 1 should not do anything of significance but somehow the model does not converge unless I add this layer. Can someone help me understand how adding a MaxPooling1D layer with a pool_size of 1 would help my model converge?

if MAX_POOL_FIRST:
    model.add(MaxPooling1D(pool_size=(1, ), input_shape=input_shape))
    model.add(Conv1D(16, kernel_size=(6, ), activation='relu'))
else:
    model.add(Conv1D(16, kernel_size=(6, ), activation='relu', input_shape=input_shape))


texts: what does random seed value mean in weka?

texts:
I use weka to make a dataset classification but there is an option in classifier evaluation ( random seed for XVAL/% split). What does this option mean? ,and what is the seed value ? ,and What is the effect of changing the value of this option from one to two or three or other values?.

I read that the value of the seed is the starting point, but what is the difference if it is the starting point(seed value) 1 or 2 or 10 , for example.


texts: Using bias rather than weights to train RBM (Contrastive Divergence)

texts:
When training an RBM, we used the above to figure out how to update the weights (contrastive divergence).
We are now being asked to update the bias and not update the weights to train.  The hint was to rework this partial derivative but with respect to the bias rather than the weights.  Could anyone provide some help on this?  Thanks.


texts:
I've used svm classifier. Now I need to construct the confusion matrix. Here is the code that I have used.

from sklearn.svm import SVC

svm = SVC(kernel='rbf')

svm.fit(feat_train,np.argmax(Y_train,axis=1))
svm.score(feat_train,np.argmax(Y_train,axis=1))
svm.score(feat_val,np.argmax(Y_val,axis=1))
y_pred = svm.predict(feat_val)
y_pred.shape
(950,)
y_class = np.argmax(y_pred, axis = 1)
y_check = np.argmax(Y_val, axis = 1)

cmatrix = confusion_matrix(y_check, y_class)
print(cmatrix)
print('accuracy Score :'),accuracy_score(y_check, y_class)

print('Report : ')
print(classification_report(y_check, y_class))

I'm getting the following error

---------------------------------------------------------------------------
AxisError                                 Traceback (most recent call last)
<ipython-input-102-67006c959068> in <module>
----> 1 y_class = np.argmax(y_pred, axis = 1)
      2 y_check = np.argmax(Y_val, axis = 1)
      3
      4 cmatrix = confusion_matrix(y_check, y_class)
      5 print(cmatrix)

~\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in argmax(a, axis, out)
   1101
   1102     """
-> 1103     return _wrapfunc(a, 'argmax', axis=axis, out=out)
   1104
   1105

~\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     54 def _wrapfunc(obj, method, *args, **kwds):
     55     try:
---> 56         return getattr(obj, method)(*args, **kwds)
     57
     58     # An AttributeError occurs if the object does not have

AxisError: axis 1 is out of bounds for array of dimension 1

I think the problem is with y_pred as the dimension is given as (950,). Please help me solve this


texts: Should I scale data before or after balancing dataset?

texts:
I have 3 datasets which I each split into 3 separate classes [Buy/hold/sell]. I randomly up-sample each class's frequency in each dataset to 10,000 data points each.

My question is, should I scale the training set before I do random up-sampling, or afterwards? Does it somehow skew the final training set in some way?

I have provided a function which balances each dataset for me, note that at this point, the data has already been scaled.

def balance_dataset(df):
    training_set = df[:round(len(df.values) * TRAINING_LENGTH)]

    label_frequencies = training_set['Label'].value_counts(sort = True, ascending = True)

    highest_occurence = resample(training_set[training_set['Label'] == label_frequencies.index[2]], n_samples = 10000, random_state = 0, replace = True)
    middle_occurence = resample(training_set[training_set['Label'] == label_frequencies.index[1]], n_samples = 10000, random_state = 0, replace = True)
    lowest_occurrence = resample(training_set[training_set['Label'] == label_frequencies.index[0]], n_samples = 10000, random_state = 0, replace = True)

    balanced_training_set = pd.concat([highest_occurence, middle_occurence, lowest_occurrence])

    return balanced_training_set[['MACD', 'MFI', 'ROC', 'RSI', 'Ultimate Oscillator', 'Williams %R', 'Awesome Oscillator', 'KAMA',
            'Stochastic Oscillator', 'TSI', 'Volume Accumulator', 'ADI', 'CMF', 'EoM', 'FI', 'VPT','ADX', 'ADX Negative', 'ADX Positive',
            'EMA', 'CRA']],  balanced_training_set['Label']


texts: Why should re-sampling change the value of model's coefficients?

texts:
I have the code below in python to create LinearRegression model. When I train the model with re-sampled data, I get different values for its coefficients. I can't understand why that happens. Can you help me in this please?

[Update]

I assume that resampling is the same as shuffling. And that means the order of the data is changed but not the data itself.
In the use case presented, the number of rows are are the same as I inspected it and as I understand the order of the data is changed.

Thanks!

from sklearn.linear_model import LinearRegression
from sklearn.utils import resample

model = LinearRegression(fit_intercept=False)

model.fit(X, y)
print('model.coef_',model.coef_)

model.fit(*resample(X, y))
print('model.coef_',model.coef_)

model.fit(*resample(X, y))
print('model.coef_',model.coef_)


texts: Handling categorical variables in large df

texts:
I have a df with nearly 40 million rows and ~20 columns (total size is 2.2+GB). 15 of my features are categorical. I figured that the most reasonable way to go about this problem without making the df any bigger would be fit/transform each with LabelEncoder and then convert each feature to category data type.

The only thing is that none of the categorical features are ordinal. Should I fit/transform them with StandardScaler or is that unnecessary?


texts: Odd Neural Net performance on 2 different machines

texts:
So I have a neural net using pytorch, which I ran on my macbook pro (dual core 3.5GHz 16GB RAM), and my mac pro (8 core 3GHz, 64GB RAM). ON the macbook pro I was getting over 50 it/s whereas the mac pro was only getting about 20 it/s. Both were on 100% CPU usage, so the mac pro being slower doesn't make sense? Or am I missing something?

Thanks!


texts:
I have a pandas dataframe df, and I want to show the histogram.

df.hist(bins=100, label="myhist")

Now, for some reason I have lots of zeros in this df, so I only want to show the values between 1 and 100. I tried

df.hist(bins=(1,100), label="myhist")

but that gives a flat line which has nothing to do with the data.

How do I do it?


texts: Is (manual) feature extraction outdated?

texts:
I recently attended a PhD thesis defence in which one committee members claimed that "manual feature extraction is outdated. Nowadays, we have [deep] machine learning models doing that job for us automatically."

Is this statement true? If yes, please provide a reference substantiating this claim.

Edit: Apparently, there seem to be different answers depending on the data type. Thus, please let me know about any references substantiating your claims for images, time series, etc... separately.


texts:
What are the common techniques for encoding discrete features with tree-path value (e.g. file/folder path) without expanding tree into multiple features corresponding to each node?

Update.

Overall, I am looking for encoding which can be used with different algorithms (e.g. linear regression, not only clustering).
Something like Huffman coding but usable with machine learning.

For example, consider this tree described with lisp-like notation:
(a (b b1 b2) (c c1 c2 c3))

It could be encoded by creating feature for each node
but this will be very expensive for any sufficiently large tree.

a  b  b1 b2 c  c1 c2 c3 Path
1  1  0  0  0  0  0  0  /a/b
1  0  0  0  1  0  1  0  /a/c/c2

Instead of creating feature for each node,
there could probably be a feature for each depth level.

L1 L2 L3 Path
1  1  0  /a/b
1  2  6  /a/c/c2

where
L2 can have value of 0,1,2 for none,b,c correspondingly;
L3 can have value of 0,1,2,5,6,7 for none,b1,b2,c1,c2,c3 correspondingly
(with gap between b2, and c1 so that there is numerical distance between nodes with different parents).

There are some obvious problems with it though,
e.g. c1 and c3 are further away from each other than c1 and c2.

I understand that encoding really depends on how you want to interpret the tree and relation between nodes. I was hoping there might be some article/paper which describes different approaches and trade-offs.


texts: why does Scikit Learn's Power Transform always transform the data to zero standard deviation?

texts:
all of my input features are positive. Whenever I tried to apply PowerTransformer with box-cox method, the lambdas are s.t. the transformed values have zero variance. i.e. the features become constants

I even tried with randomly generated log normal data and it still transform the data into zero variance.

I do understand that mathematically, finding the lambda s.t. the standard deviation is the smallest, would mean the distribution would be the most normal-like.

But when the standard deviation is zero, then what's the point of using it?

p.s. so one of the values of lambda I get by using PowerTranformer is -4.78

If you apply it into the box-cox equation for lambda != 0.0, then for any input feature y values, you technically get the same values. i.e. (100^(-4.78)-1.0)/(-4.78) is technically equals to (500^(-4.78)-1.0)/(-4.78)


texts: Cause of periodic jumps in loss function

texts:
I might be missing something obvious as I am new to machine learning. I am training an SSD Inception V2 for detecting buildings from satellite images. I use the Tensorflow Object Detection API. I am having troubles interpreting why the value of loss seems to change periodically:

Please let me know if I need to add more information


texts: Can closer points be considered more similar in T-SNE visualization?

texts:
I understand from Hinton's paper that T-SNE does a good job in keeping local similarities and a decent job in preserving global structure (clusterization).

However I'm not clear if points appearing closer in a 2D t-sne visualization can be assumed as "more-similar" data-points. I'm using data with 25 features.

As an example, observing the image below, can I assume that blue datapoints are more similar to green ones, specifically to the biggest green-points cluster?. Or, asking differently, is it ok to assume that blue points are more similar to green one in the closest cluster, than to red ones in the other cluster? (disregarding green points in the red-ish cluster)

When observing other examples, such as the ones presented at sci-kit learn Manifold learning it seems right to assume this, but I'm not sure if is correct statistically speaking.

EDIT

I have calculated the distances from the original dataset manually (the mean pairwise euclidean distance) and the visualization actually represents a proportional spatial distance regarding the dataset. However, I would like to know if this is fairly acceptable to be expected from the original mathematical formulation of t-sne and not mere coincidence.


texts: Quantile regression with inhomogeneous density of points

texts:
I am working on a dataset that can be represented this way:

We can see that the lower values on the y-axis are increasing linearly along the x-axis. I want to estimate the coefficients of this linear function, and quantile regression is what seems most appropriate for this. However, as we can see the density of points is much higher on the left-hand side, which gives more weight to this part of the dataset:

Is there a way to avoid this ?


texts: Anomaly detection using clustering of highly correlated Categorical data

texts:
My data has two columns and both are highly correlated e.g. if column1 has value ABC, column2 should be XYZ i.e. ABC-->XYZ. If column2 has anything else its Anomaly. Likewise there are thousands of combination. I already tried KModes clustering where number of clusters = unique values in column1. However each cluster do not have equal density hence some bad data with high density is classified as normal and good data with low density is marked anomalous.

I want to have unsupervised algo where I can force it to use column1 as primary criteria for clustering. One with the highest frequency of column2 data for each unique value of column1 is good data. Rest is anomalous. Kindly suggest what would be the best algo and how to approach this problem.


texts:
I am working on an image classification tasks in which I have 4 classes (For example A, B, C, D). I used CNN model (transfer learning) to train the model and predict the video frames.

Ideally, there would not be any sudden transition from one class to another. Ideal Transition would be as given below:
A A A A A ..... A A A A B B B B B B ..... B B B B B B B D D D D D ..... D D D C C C C C C C ............. C C C

However when predicting using the trained model, I could see some frames being mis-classified. For example I given an video input of Class A which has around 30 frames. In that 30 frames 5 would be predicted as Class C or D.

How would I make the smooth transition from one class to another. There should be sufficient number of evidences so that I can make transition from one class to another.

As of now I found moving average technique which does similar kind of smoothening. I would like to know is there any other method which is more related to probability.

Kindly let me know if you need more details.

Thank you,
KK


texts: "residual error" of LSTM during backprop vs usual "error"

texts:
What does the residual error mean when we are talking about LSTM?

Taken from the middle of section 3 of this paper, where it says:

"...of the residual error $\epsilon$"

Where $s_0$ is the initial state of the RNN network.

Question: how is a residual error different to a usual error? Why to use such a term?


texts: A way to Identify tuning parameters and their possible range

texts: I am a novice in Machine Learning. But when I started learning, I figure out that all the methods have some tuning parameters and those parameters take a range of possible values. By grid searching, we identify a set of these parameters that optimize some function. But is there any way to find the possible domain of the tuning parameters? This would definitely save my time and the computer's job. In addition, some methods such as xgboost have loads of tuning parameters. Is there any way to know which one to tune and which one to leave as it is. I have been using sklearn python library.


texts:
Sorry for such a weird question - I do not even know if this makes sense, however I thought of this during my intro to Python course at uni and have wondered about it since.

So I have some experience with Python for data science, but have never taken a structured course. I am in an intro class, and on the first day we were asked to count how many times a certain name appeared in a list. Obviously the answer I came up with was to initialize a variable to zero then iterate through the list and add one to itself whenever the i'th name in the list was equal to the target name.

However, in Andrew Ng's deep learning courses, there is a heavy emphasis on vectorizing our calculations whenever possible. This got me wondering how I could vectorize this task.

What I came up with, in an ideal scenario, is to create a vector where each the target name is replaced with 1, and the other names are replaced with 0. Then I could just take the sum of the vector and I would have my answer. The problem with this is the only way I know how to create said vector is to iterate through the original list, defeating the purpose this way of solving the problem.

Anyways, while I understand that creating this vector is not exactly one hot encoding, is there any way to vectorize this process? Is one hot encoding done iteratively. If not, are there other examples where basic iterative tasks can be turned into tasks able to be processed in parallel?

Sorry if this is a dumb question or does not make any sense - just something I was curious about as a noob to base Python.


texts: Prevent model from predicting the same examples for different inputs

texts:
I have a ANN model, that predicts a fixed length curve. The problem is, those curves are really similar to each other, for example these two:

To compare those curves, I use RMSD between their points. The problem is, that the model for most of the parameters, learns a mean-like curve representation, that has low RMSD for most of the curves, so the model predicts the same curve for different inputs.
Is there any loss function/metric to penalize for predicting the same outputs for different inputs ?


texts: Dealing with null values in categorical string data

texts:
Which of the following is a better approach in dealing with null values and why?

Fill the null values with the mode (most occurring value).
Consider the null itself as a category by converting it to a string.
Remove the records with the null value.
Predict it using another non-null columns.

Edit
P.S: I am work with the following dataset
https://data.ontario.ca/dataset/confirmed-positive-cases-of-covid-19-in-ontario
It is about the covid-19 cases in Ontario. In this dataset a string categorical column named "Case_AcquisitionInfo". It has some values with string "MISSING INFORMATION" so should I consider this null or consider this as a category.


texts: Will crypto currency mining hardware work fine for machine learning?

texts:
I've found that because of recession in crypto mining business a lot of mining rigs are available on sale in pretty reasonable prices. For around $1000 we can buy used machine. Will rig like this work fine for machine learning:

6x NVIDIA GTX 1070 8GB
Intel Celeron G3900
120GB SSD
DDR4 8GB
Asus Z 170 Pro Gaming

We are a small development company and looking for good priced rig efficient enough to teach some models on it. Such hardware seems to have a lot of CUDA cores, but what might be a bottleneck?


texts: Is it possible to save specificity and sensitivity during training?

texts:
I'm using Repeated Stratified k-fold cross-validation and Oversampling to deal with imbalanced classes in a binary classification problem, using the ROC AUC metric. My question is: Is it possible to estimate and save specificity and sensitivity during training?
# define pipeline
steps = [('over', RandomOverSampler()), ('model', LogisticRegression())]
pipeline = Pipeline(steps=steps)
# evaluate pipeline
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1)
scores = cross_val_score(pipeline, X, Y, scoring='roc_auc', cv=cv)
score = mean(scores)
print('ROC Oversampling: %.4f' % score)


texts: How to test/train a model for realtime data with new data points and classes in a ML pipeline

texts:
First, For a text classification problem, if I have trained the model on 2 classes and it gives good accuracy. Now, when I use the model in real-time, there is a completely new class from a totally different business team which I need to use the trained model and test this data on. Here, obviously the model will have worse results, and also the pre-processing or any other data cleaning could be way different. So, the question is - Is the only possible way is to re-train the model again?

Second, In a segmentation problem, the approach is to use K-Means clustering and group customers. On a live environment, how do we handle new data points? How do we know which group does the data point belong to? Coz the model has been standardized/normalized properly and then fitted. The same cannot be expected for a new data point. How do we handle this?


texts: TS-SS and Cosine similarity among text documents using TF-IDF in Python

texts:
A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix.

TF-IDF matrix is calculated using TfidfVectorizer().

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(stop_words='english')
tfidf_matrix_content = tfidf.fit_transform(article_master['stemmed_content'])

Here article_master is a dataframe containing the text content of all the documents.
As explained by  Chris Clark here, TfidfVectorizer produces normalised vectors; hence the linear_kernel results can be used as cosine similarity.

cosine_sim_content = linear_kernel(tfidf_matrix_content, tfidf_matrix_content)

This is where my confusion lies.

Effectively the cosine similarity between 2 vectors is:

InnerProduct(vec1,vec2) / (VectorSize(vec1) * VectorSize(vec2))

Linear kernel calculates the InnerProduct as stated here

So the questions are:

Why am I not divding the inner product with the product of the magnitude of the vectors ?
Why does the normalisation exempt me of this requirement ?
Now if I wanted to calculate ts-ss similarity, could I still use
the normalised tf-idf matrix and the cosine values (calculated by
linear kernel only) ?


texts: I am trying to understand using DQL for playing board games and how we can do function approximation of the q-learning Bellman equation in order to detect the best next move , if anyone can give the general overview I will be thankful .Also What this graph tell us about the algorithm ,I can't understand why they plot the max over all actions , what that tell us ?


texts: Generating synthetic data based off existing real data (in Python)

texts:
I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future anomalies).
I would like to mimic the statistical properties of the real data, such as mean, mode, standard deviation, etc., to create the synthetic data, then inject anomalies based off reasonable extreme values (if we know the statistical properties of each column in the real data then we can deduce what an extreme value might look like for that column).
Are there any Python packages that generate synthetic data based off known statistical properties in real data. I imagine this is similar to differential privacy, but we are not doing this to protect privacy and don't need an overkill method.
scikit-learn can generate synthetic data but it doesn't seem to have a method to base it off of existing real data statistical properties.
I can do something simple like this:
res = {}
for column in df:
    nrows = len(df[column].index)
    mean = df[column].mean()
    std = df[column].std()
    mu, sigma = mean, std # mean and standard deviation
    synthetic_data = np.random.normal(mu, sigma, nrows)
    res[column] = synthetic_data

...which just detects the means and standard deviation of each column then recreates it using a numpy draw from a normal distribution (big assumption), but obviously this doesn't mock the data well:
Real Data

Synthetic Data


texts: How do you overcome results oriented thinking from a client?

texts:
I am working with a client on a special project that I am going to obfuscate in this question. Basically, I'm trying to overcome some short-term, results-oriented thinking from my client.

Let's say you have a model to forecast the performance of a racehorse. Your model tells your client to sell racehorse X because the probability of its performance is low (<10%). The horse is sold and said racehorse goes on to win 3 races. Your client says, "See, we should have never let go of that racehorse! The model is wrong!"

As data scientist we can understand that anomalies happen and that this horse may have also lost some races - we were just on the wrong side of probability this time. But how do you overcome those objections with the client? How do you turn short-term thinking into a long-term outlook of predictive modeling?


texts:
I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some questions about how to choose values for the learning rate.

Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent?
If so, how do you use this information to inform your decision about a value?
If it's not what sort of values should I choose, and how should I choose them?
It seems like you would want small values to avoid overshooting, but how do you choose one such that you don't get stuck in local minima or take to long to descend?
Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?

In short: How do I choose the learning rate for SGD?


texts: Find the consecutive zeros in a DataFrame and do a conditional replacement

texts:
I have a dataset like this:

Sample Dataframe

import pandas as pd

df = pd.DataFrame({
    'names': ['A','B','C','D','E','F','G','H','I','J','K','L'],
    'col1': [0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0],
    'col2': [0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0]})

I'd like to replace some of the 0's in col1 and col2 with 1's, but not replace the 0's if three or more 0's are consecutive in the same column.  How can this be done with pandas?

Original Dataset:

names   col1    col2
A   0   0
B   1   0
C   0   0
D   1   0
E   1   1
F   1   0
G   0   1
H   0   0
I   0   1
J   1   0
K   0   0
L   0   0

Desired Dataset:

names   col1    col2
A   1   0
B   1   0
C   1   0
D   1   0
E   1   1
F   1   1
G   0   1
H   0   1
I   0   1
J   1   0
K   1   0
L   1   0


texts: Reinforcement learning: decreasing loss without increasing reward

texts:
I'm trying to solve OpenAI Gym's LunarLander-v2.

I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score.

Generally the loss decreases over many episodes but the reward doesn't improve much.

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.

Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?


texts: How do I identify specific parts of a PDF document?

texts: I have a bunch of medical records that I have to input manually. I would like to automate this but all of the records are in different formats. What is the best strategy to build a deep learning model to extract this information automatically? Does it make sense to use something like YOLO to recognize specific parts of the image and then Tesseract to actually perform OCR? A bit of guidance would be much appreciated. Also, I have seen things like Amazon Textract but am looking for something I can build and run locally without depending on another service. Thanks!


texts: How to choose the best parameter values for TfidfVectorizer in sklearn library?

texts: Recently, I used  TfidfVectorizer in scikit-learn library to calculate a matrix of TF-IDF features. However, I do not know how to set some parameters such as max_features, min_df, max_df, etc.


texts: CNN computing time on good CPU vs cheap GPU

texts:
I am a researcher working on my first deep learning project, which consists of using a CNN (pre-trained VGG16+2 densely connected layers) to classify drone imagery of vegetation.

In trying to hack computing times for both training and prediction of new images, I am considering asking my employer for money to buy a (cheap) NVIDIA GPU.

Being a biologist and not a computer scientist, I do not have any sense of the upgrade this would get me. Searching online I found contradicting opinions.

I am currently working with Keras+Tensorflow on a desktop PC with i7, 3.6 ghz, 32 Gb RAM.

Question: how good a GPU would I need to get a sensible performance increase?

Thanks a lot!


texts: Using numpy.ndarray in machine learning sklearn.preprocessing model


texts: Grid search model isn't recognized as fitted for Graphviz

texts:
I find this really weird, and the code is really straight forward.
What am I doing wrong ?

from sklearn.model_selection import GridSearchCV
scoring_type="accuracy"
preprocess_data(X,y,0)
p_grid = {'min_samples_split':np.arange(2,10),'min_samples_leaf': np.arange(1,10)}
Tree_opt = GridSearchCV(estimator=DecisionTreeClassifier(random_state=42), param_grid=p_grid, scoring=scoring_type, cv=10)
Tree_opt.fit(X_train,y_train)

print("Best training params: {}".format(Tree_opt.best_params_))
print("Best training Score: {}".format(Tree_opt.best_score_))
dot_data = tree.export_graphviz(Tree_opt, out_file=None,feature_names=labels,class_names=class_names,filled=True, rounded=True,special_characters=True)
graph = graphviz.Source(dot_data)

graph
print(Tree_opt.score(X_test, y_test))

And I'm getting the following error:

NotFittedError                            Traceback (most recent call last)
<ipython-input-14-1c7fa906f99b> in <module>
      8 print("Best training params: {}".format(Tree_opt.best_params_))
      9 print("Best training Score: {}".format(Tree_opt.best_score_))
---> 10 dot_data = tree.export_graphviz(Tree_opt, out_file=None,feature_names=labels,class_names=class_names,filled=True, rounded=True,special_characters=True)
     11 graph = graphviz.Source(dot_data)
     12

~\Anaconda3\lib\site-packages\sklearn\tree\export.py in export_graphviz(decision_tree, out_file, max_depth, feature_names, class_names, label, filled, leaves_parallel, impurity, node_ids, proportion, rotate, rounded, special_characters, precision)
    394                 out_file.write('%d -> %d ;\n' % (parent, node_id))
    395
--> 396     check_is_fitted(decision_tree, 'tree_')
    397     own_file = False
    398     return_string = False

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_is_fitted(estimator, attributes, msg, all_or_any)
    949
    950     if not all_or_any([hasattr(estimator, attr) for attr in attributes]):
--> 951         raise NotFittedError(msg % {'name': type(estimator).__name__})
    952
    953

NotFittedError: This GridSearchCV instance is not fitted yet. Call 'fit' with appropriate arguments before using this method.

I also have this warning I don't think it's important:

C:\Users\Flow\Anaconda3\lib\site-packages\sklearn\model_selection\_search.py:841: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.
  DeprecationWarning)

I think the fit works because I can get the first prints as follows:

Best training params: {'min_samples_leaf': 3, 'min_samples_split': 2}
Best training Score: 0.8809523809523809


texts:
Given the anonymized dataset of features below, where:

"code" is a categorical variable.
"x1" and "x2" are continuous variables.
"x3" and "x4" are extracted features. They are the mean values of "x1" and "x2" respectively for each individual code.

       code  x1  x2  x3  x4
    0   100   1   2   2   4
    1   100   2   4   2   4
    2   100   3   6   2   4
    3   200   4   8   5  10
    4   200   5  10   5  10
    5   200   6  12   5  10
    6   300   7  14   8  16
    7   300   8  16   8  16
    8   300   9  18   8  16

Looking at the columns, for each code, x3 and x4 features have similar values - when x3 is 2 or 5 or 8, the code would be 100 or 200 or 300 respectively, and when x4 is 4 or 10 or 16, the code would be 100 or 200 or 300 respectively.

Intuitively, leaving these columns as they are without dropping any would lead to redundant features while training a model. My question is how true is this my hypothesis? I'm not so confident about it. Does it really matter when training a model? Does it depend on the model type (tree based or otherwise)?


texts: True positives and true negatives, F1 score: multi class classification

texts:
I have 4 classes for an application of classification of animal kingdom: 1 --> invertibrates; 2 --> vertibrates; 3--> mammal; 4 ---> ambhibian. Given a mixture of images the objective is to identify mammals correctly. In the confusion matrix for this example will the TP denote the class 3 (mammal)?

Q1: Therefore, in general TP is the class category which is of utmost significance? What if all the classes are equally important, then how to denote TP and TN?

Q2: How to calculate the F1 score? should it be done separately for each class? If then there will be multiple TN's w.r.t each class !!

Can somebody please help clear these confusions? Thank you.


texts: Merging common Columns values in two DataFrame Pandas

texts:
I have a dataset of patients from which I want to predict whether patient suffering from diabetes or not.
In that I have a DataFrame  res_total_Df with columns suppose [patient_id, urine output, Blood pressure] and

another DataFrame key_df with columns suppose [patient_id , haemoglobin, Blood pressure].

I want to merge into single dataFrame in which common columns values should be added as list(for which later I would take mean). Resultant dataFrame would be [patient_id, urine output, haemoglobin, Blood pressure].

How can I achieve it using pandas. the number of columns in second dataFrame can vary because I am extracting them from the text. I cannot paste the dataFrame snapshot because it is health data.

Here is my code

def find_common_col(res_total_df, key_df):
    # patient_id is always common
    common_list = ['patient_id']
    for i in res_total_df.columns:
        if i in key_df.columns:
            if i not in common_list:
                common_list.append(i)

common_col = find_common_col(res_total_Df, key_df)
res_total_Df = pd.merge(res_total_Df, key_df, how='outer', on=common_col)

The problem with above code it has duplicates rows. Suppose Blood pressure value for patient_id 1993 in res_total_Df is 180 and in key_df is 200. Then it adds two rows one with value 180 and other with value 200 for patient_id 1993.

It want Blood pressure for patient_id 1993 as [180, 200].


texts: California housing prices dataset on Kaggle

texts:
This is the dataset I'm referring to: https://www.kaggle.com/camnugent/california-housing-prices
I am a bit confused regarding the dataset. Specifically, why is the number of houses not provided? We are given the number of households, but that's not equivalent to the number of houses. There's probably a near monotonic relation between the two, but I feel like the dataset should include the number of houses per block.


texts: Using extracted feature vector to perform zero shot detection

texts:
I've developed a deep learning model trained from scratch on fruits and vegetables. However, as the data is limited, I can only cover a few different types of fruits and vegetables with the model alone. However, the accuracy on those categories is very high, 90% training and validation accuracy with around 80% testing accuracy (from a different set of data).
Now I tried to use this trained model on unseen data. So I've removed the softmax layer and only kept the other layers of the model. The idea is to use the feature vector and cosine similarity to compare vegetables.
Let's say bananas were in my original dataset but apples and oranges were not. I use the model to extract features of apples and oranges off of google, then compare the vectors of apples with origins. I would expect the cosine similarity to be very low as they are very different vegetables. However, I get a cosine similarity of 0.9, which is very confusing. To test my sanity, I used a pretrained Mobilenet v2 model and also removed the last layer. Then used the same method to extract the features of apples and oranges. Now they only have a similarity of around 0.5, which is much more reasonable.
I have a few ideas as why this could be the case, perhaps the model could not generalize past the categories given to it during training, or it did not learn similar features that could be translated to apples and oranges so it could not differentiate between the two. However I have no certain explanation. Does anyone have any idea why this is happening, and/or how I can prevent this from happening?


texts: Reinforcement learning: understanding this derivation of n-step Tree Backup algorithm

texts:
I think I get the main idea, and I almost understand the derivation except for this one line, see picture below:

I understand what we're doing by using the policy probability to weight the rewards from time t + 2 (because getting here depends on the prob of taking an action that gets here). But I don't understand why we similarly subtract the value function from the return...

It also doesn't seem to match the example target return (G) implied for 2 step backup on slide 15 of this lecture's slides:

https://www.dropbox.com/sh/3xowt7qvyadvejn/AABpWQMKWX3KVbeqVlBcxNYra/slides%20(pdf%20and%20keynote)?dl=0&preview=13-multistep.pdf

Thanks for any insight. I could be missing something simple/obvious as I dive into these details.

EDIT - for more context, see pg. 160 of this pdf which is where the picture comes from: http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf


texts: Ideal score of a model on training and cross validation data

texts:
The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.

I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:

(1) Train set: 0.99 | Cross-validation set: 0.72
(2) Train set: 0.75 | Cross-validation set: 0.70
(3) Train set: 0.69 | Cross-validation set: 0.69

These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.

My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?


texts:
What is the best way to model compositional data problems?

Compositional data is when each example or sample is a vector that sums to 1 (or 100%). In my case, I am interested in the composition of minerals in a rock and I have sensors that tell me the sum of the minerals but not the components that make up the sum.

For example, lets say I have two minerals, $m_1$ and $m_2$, that are made up of 3 elements (like copper and other elements from the periodic table) which form a vector of length 3:

m1 = [0.1, 0.3, 0.6]
m2 = [0.6, 0.2, 0.2]

If a rock has 25% of $m_1$ and 75% of $m_2$, the sensor reading produces the sum of the two minerals (shown in bottom-left subplot below):

$$
\begin{align}
&0.25*m_1 + 0.75*m_2 \\
=&0.25*[0.1, 0.3, 0.6] + 0.75*[0.6, 0.2, 0.2] \\
=&[0.475, 0.225, 0.3]
\end{align}
$$

I would like to know how to model and solve the problem of unmixing a composition into its underlying components, where the sum of the elements is normalized to 100% (e.g. $0.25m_1 + 0.75m_2$ has the same composition as $0.50m_1 + 1.50m_2$).

Furthermore, my example is simplistic; in reality a composition can have more than just 2 minerals (up to 3000) and each mineral is made up of 118 elements, not just 3 (all the elements of periodic table - though many elements will be zero). The elemental composition of a mineral is assumed to be known (definition of $m_1$ and $m_2$ in the example). Also, the sensor reading is noisy - each element of the observed composition is assumed to have Gaussian noise.


texts: Proportion of positive/negative label in Supervised Learning

texts:
I'm working on a Supervised Machine Learning problem and I have a question about the proportion of positive/negative label.

I would like to categorize some batch as OK or NOK. But actually my batchs are most often OK.
To give you an idea, at the moment I have approximately 400 batchs OK and 10 NOK.

My questions are :
What will be the impact on the results of this proportion?
Is it still possible to have conclusive results? If so, with what type of algorithm?


texts: Natural language Generator using Data from table

texts:
I am working on some natural language generator part.

eg.1
Input to it will be

Col1 Col2 Col3
A      B    13
X      Y    14

Output should be two sentence, one for each row

Number of columns and rows can change every time.
Please can any one suggest any approach/free source lib for it


texts: How to impute Missing values not the usual way?

texts:
I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has around 250-350 missing records.

How do I know whether this is missing at random, missing completely at random or missing not at random. For ex: 4400 patients have the readings and 330 patients don't have the readings. But we expect these 330 to have the readings because it is a very usual measurement. So what is this called?

In addition, for my dataset it doesn't make sense to use mean or median straight away to fill missing values. I have been reading about algorithms like Multiple Imputation and Maximum Likelihood etc.

Is there any other algorithms that is good in filling the missing values in a robust way?

Is there any python packages for this?

Can someone help me with this?


texts: Causes of inconsistent results with neural network?

texts:
I've created my own neural network in python. I am tracking the performance of each run. What are some things I can investigate when my performance results are inconsistent? When I compare my network results against my expect results (during cross-validation) I can get everything from 0.46 to 0.51 all the way up to 0.56 and various numbers in between.

Is this a sign of bad data? Not enough training epochs? Need for a different learning rate?


texts: How can I handle a column with list data?

texts:
I have a dataset which I processed and created six features:

['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList',
   'totalProducts']

And the target variable is a binary class (gender).

The feature 'productList' is a list:

    df['ProductList'].head()
Out[169]:
0    [13, 25, 113, 13793, 2, 25, 113, 1946, 2, 25, ...
1    [12, 31, 138, 14221, 1, 31, 138, 1979, 1, 31, ...
2                               [13, 23, 127, 8754, 0]
3    [13, 26, 125, 5726, 2, 26, 125, 5727, 2, 26, 1...
4           [12, 23, 119, 14805, 1, 23, 119, 14806, 0]
Name: ProductList, dtype: object

Now, it is obvious that I can't use this feature as it is. How do I handle this feature? I can explode the list and create a row for each list item, but will it serve my purpose?

Update: I applied OHE after exploding the list, and it results in 10k+ columns, which my GCP instance and my computer can't handle; when applying PCA.

PS: There are over 17,000 unique products.


texts: What will cause high accuracy but a big loss?

texts:
In the question of What is the relationship between the accuracy and the loss in deep learning?, @Jérémy Blain gave a fantastic interpretation of 'relationship' between accuracy and loss:

1 - low accuracy and big loss means you made huge errors on a lot of data
2 - low accuracy but small loss means you made little errors on a lot of data
3 - high accuracy with small loss means you made low errors on a few data (best case)
4 - high accuracy but a big loss, means you made huge errors on a few data.

As I understand,

1 - implies bad algorithm 'in most of time'
2 - implies overfitting
3 - is what we want

Questions are:

 1   Any other reasons caused 1 and 2?
 2   what will lead to case 4 : high accuracy but a big loss?


texts: I have a question about the training example for machine learning. I know most people make training examples to a column vector. However, most of the time when we get a training example, like from a panda data frame object and transfer it to a NumPy array, each example will be a row vector. I know I can use transpose to change it to a column vector. But I want to ask why do we do that? What makes a column vector better suited for training examples?


texts: Minimum number of samples to train XGBoost without overfitting

texts:
When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron.
Is there a similar rule of thumb for classifiers such as XGBoost, presumably taking into account the number of features and estimators?
And, considering the 'curse of dimensionality' shouldn't the rule of thumb be that n_training is geometric in n_dimensions, and not linear?


texts:
Traditionally, a state for RNN is computed as
$$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$

For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single matrix times a concatenated vector:

$$W_m[x, h_{t-1}]$$

where $[...]$ is concatenation.

In other words, we would end up with a long vector like $\{x_1, x_2, x_3, h_{1,t-1}, h_{2,t-1}, h_{3,t-1} \}$ multiplied by $W_m$.

It seems like the second approach would have a significantly larger matrix, which has more elements than $W$ and $U$ combined.

Does that mean $W$ and $U$ are a simplification, what do we lose by using them, and adding up the results?


texts: Finding the most phonetically similar word from WordNet

texts: Besides soundex and other libraries, that take two words and determine whether they are similar, is there any way to find the most similar sounding word from WordNet, for a given word? I tried to use autospell as a well to correct the spelling, but unable to get the desired results.


texts: very large difference between cross_val and (multiple) r2 model evaluation

texts:
I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and run an r2 score (accuracy) on the predicted train & test set:

X_train, X_test, y_train, y_test = train_test_split(
    df2, y, test_size=0.33, random_state=np.random.randint(0,100))
gradModel = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)
gradModel.fit(X_train,y_train)
y_pred=gradModel.predict(X_test)
(r2_score(y_train, gradModel.predict(X_train)),r2_score(y_test, y_pred))

which gives (0.9325975114210778, 0.8791805732349958) Repeatedly over multiple runs, I believe this is a reasonable score?

When I run a cross validation on the same model (same parameters) I do get a much lower score.

gradModel = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)
from sklearn.model_selection import cross_val_score
scores = cross_val_score(gradModel, df2, y, cv=15)
print ("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.45 +/- 0.21

Which I believe is a more realistic score, but I can't explain the large differences. Also because the original r2 score is more or less equal over multiple reruns


texts: Transfer learning within Tensorflow's inception model

texts:
I want to freeze the layers, except the first three layers, in the Inception v3 model with TensorFlow in Python 3, and modify the weights of these three layers to be able to re-initialize and re-train only the three first layers of the network.

If this can't be done within the inception model, is there any other network (in TensorFlow) with which this could be done?


texts: Cross validation for convolutional neural network

texts:
I am using Keras to create a CNN model, and I would to use K-fold cross-validation to train the dataset.

The dataset contains images and I am using flow_from_directory function.

Do you have any idea how to use K-fold cross-validation in Keras to create a CNN model??


texts:
I wonder why most people now don't use matlab. I guess the reason is matlab is not free, so companies don't want to use it, then interviewees don't use it, then schools don't encourage it, then nobody ends up using it.
But I like it, compared to C++, Python, for it's convenient to plot figures without needing to import any libraries or header files, let alone those powerful ODE solvers.
I might be wrong on any of the above points, as I'm not most familiar with other languages. Any comments?


texts: How to load a csv file into [Pandas] dataframe if computer runs out of RAM?

texts:
I have been trying to train a neural network, but my computer is always running out of RAM memory when I'm loading the dataframe with Pandas. Its a .csv file that is like 7+ GB.

I wanted to try some primitive batching but in order to one hot encode I need to find number of all unique values, which i can't do without loading data into a dataframe first.

Are there any other tools that I can use to attempt loading the file into a dataframe? Does Pyspark also have a limit of when it starts crashing? I know that its capable of breaking down operations into stages, does that help with RAM management or just execution?


texts:
I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a classification model.

Now, I am experimenting the gensim word2vec approach, each word is now a vector from word2vec. That is, if my sentence have 10 words, it would become a 10x30 array (assume word2vec embedding dimension is 30). It's not clear to me how do I send such outputs to a logistic regression or neural network model like before ...

What type of model should I use after the gensim word2vec outputs to solve a classification problem? Thanks!


texts: What is the standard model for CNN to compare against other classifier technique?

texts:
I have created a new method to do binary image classification. I think it would be interesting to compare it to the convolutional neural network that would do the same binary classification given the same data sets. The question I have is what would be a standard architecture of such CNN model? How much should I tune it to increase the accuracy of the CNN model so it is fair to compare it to my methodology? I want this to be as fair as possible with the chance of maybe having my new methodology published. I tried googling my question but I could not stumble upon any blog about it.

I would appreciate some guidance or links.


texts:
I am currently working into a problem that requires to redefine the same model multiple times, just having "logical" variations that do not change the model structure. In doing that, I would like to not restart the training process each time I redefine the model. So, I have opted to use the save_weights() TensorFlow function to save weights, but I was wondering how to properly save the optimizer state. Currently, I am doing opt_old = model.optimizer, then assigning opt_old to the new model when new_model.compile(opt=opt_old,...) is called. Even though no error is returned and the execution runs smoothly, I am not really sure this is the correct approach to obtain what I am really looking for. So, how to save and restore the optimizer state correctly?
Concerning the fit methods, can I just use the epochs parameter specifying the number of epochs I want to train the model for the current round, or have I to use also the starting epoch one, working in an incremental fashion?
Thank you.


texts: How to optimize cohort sizes to reduce pair-wise comparisons?

texts:
I am making all pairwise comparisons in a dataset. The use-case is collapsing records into a unique ID based on fuzzy names and dates of birth. The size of the database is around 57,000 individuals. So this is a total of 57,000 choose 2 pairwise combinations. (This is a tiny example I know, but I have other databases with the same problem that are much larger.)

Based on other analysis, I concluded that I do not want to consider people with birthdates of more than three years apart. So I can subset the database into overlapping cohorts, and then only do all pair-wise comparisons within each cohort.

It is easy to show examples of where this cohort approach will reduce the number of pairs I need to compare. Here is one example with my data just based on the quintiles of the year of birth (and those with missing birthdays go into all cohorts).

(Min,1968]  : 13,453
[1962,1980] : 17,335
[1974,1988] : 21,188
[1982,1993] : 21,993
[1987,Max)  : 17,449

Which saves me around 0.7 billion comparisons.

So this brings up two questions:

are the choosing the bins based on quantiles a good strategy, or is there anther strategy that works better?
how many bins should I make?


texts: K-means program in python for large excel database

texts:
I have an excel database of 50 rows and some 28,000 columns. The database provides information of water parameters such as pressure, roughness, pipe length, diameter, valve, pump, flow and so on. I want to apply k-means code on this data to find outliers.

Code

from pandas import read_excel
import numpy as np
from numpy import arange
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode

dataset = read_excel('D:\M.tech\Dataset\Testing database1 New.xlsx')
#print(dataset.tail())
dataset.fillna(0)
cols=dataset.columns[1:]
#print(cols)
np.isnan(dataset).any()
dataset = dataset.fillna(method='ffill')

clusters=KMeans(n_clusters=4)
dataset["clusters"]=clusters.fit_predict(dataset[cols])
dataset.head(20)

#Principal component separation to create 2 dim picture
pca=PCA(n_components=2)
dataset['x']=pca.fit_transform(dataset[cols])[:,0]
dataset['y']=pca.fit_transform(dataset[cols])[:,1]
dataset=dataset.reset_index()
print(dataset.tail())

trace0= go.Scatter(x=dataset[dataset.clusters == 0]['x'],
                   y=dataset[dataset.clusters == 0]['y'],
                   name="Cluster1",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(15,152,152,0.5)",line=dict(width=1,color="rgb(0,0,0)")))

trace1= go.Scatter(x=dataset[dataset.clusters == 1]['x'],
                   y=dataset[dataset.clusters == 1]['y'],
                   name="Cluster2",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(180,18,180,0.5)",line=dict(width=1,color="rgb(0,0,0)")))

trace2= go.Scatter(x=dataset[dataset.clusters == 2]['x'],
                   y=dataset[dataset.clusters == 2]['y'],
                   name="Cluster3",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(132,132,132,0.8)",line=dict(width=1,color="rgb(0,0,0)")))

trace3= go.Scatter(x=dataset[dataset.clusters == 3]['x'],
                   y=dataset[dataset.clusters == 3]['y'],
                   name="Cluster4",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(230,20,30,0.5)",line=dict(width=1,color="rgb(0,0,0)")))

data =[trace0,trace1,trace2,trace3]

iplot(data)

Above code displays no error but when I run the code in ipython console, it does not display any figure.

How can I do this?


texts: Multiple testing of unbalanced data by R

texts:
I have a general question of unbalanced data. I'm performing a T test on Group A and Group B. Group A has 20 data while Group B has 500. I set unequal variance (Welch) for the adjustment and the P-value is 0.01. Can I conclude that there is significance?

I have another idea that I'm not sure about. I randomly select about 20 samples from Group B multiple times and test against the Group A. Should I do some adjustment for the result? I am not sure if this is works or not. If it works, what kind of adjustment I should use for multi-testing?

Thank you!

I appreciate any replies!


texts: sklearn - How to create a sequential pipeline

texts:
Update: The examples in this post were updated
I am reposting this question here after not getting a clear answer in a previous SO post
I am looking for a help building a data preprocessing pipleline using sklearn's ColumnTransformer functions where the some features are preprocesses sequentially. I am well aware of how to build separate pipelines for different subsets of features. For example, my pipleline may look something like this:
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import Normalizer

ColumnTransformer(remainder='passthrough',
                  transformers=[
                              ('num_impute', SimpleImputer(strategy='median'), ['feat_1', 'feat_2'])
                                ('Std', StandardScaler(), ['feat_3', 'feat_4']),
                                ('Norm', Normalizer(), ['feat_5', 'feat_6']),
                                ])

Notice that each transformer is provided a unique set of features.
The issue I am encountering is how to apply sequential processing for the same features (different combinations of transformations and features). For example,
ColumnTransformer(remainder='passthrough',
                  transformers=[
                              ('num_impute', SimpleImputer(strategy='median'), ['feat_1', 'feat_2', , 'feat_5'])
                              ('Std', StandardScaler(), ['feat_1', 'feat_2','feat_3', 'feat_4', 'feat_6']),
                              ('Norm', Normalizer(), ['feat_1', 'feat_6'])

                                ])

Notice that feat_1 was provided to three transformations, feat_2 was provided to two transformers (impute and Std), and feat_6 was provided to two transformers (Std and Norm)
A pipeline like this will two duplicate columns for feat_2 and feat_3, and three duplicate columns for feat_1. Building a separate pipeline for each transformation/feature combination is not scalable.


texts: Decision Tree gives 100% accuracy - what am I doing wrong?

texts:
My assumption is that my training set includes the test set, but I don't know how to change this.

from sklearn.model_selection import train_test_split
import sklearn.metrics as metrics

# dataframe to store model performances
scores=pd.DataFrame([],columns=['model', 'recall', 'f1', 'accuracy'])
from sklearn.tree import DecisionTreeClassifier

cv_scores_ac=[]
cv_scores_f1=[]
cv_scores_re=[]
for cv in range(1, 6):
    print ("Decision Tree - Iteration %i" % cv)
    X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=cv)

    tree=DecisionTreeClassifier(min_samples_split=20)
    tree.fit(X_train, y_train)
    y_pred=tree.predict(X_test)
    cv_scores_ac.append(metrics.accuracy_score(y_test, y_pred))
    cv_scores_f1.append(metrics.f1_score(y_test, y_pred))
    cv_scores_re.append(metrics.recall_score(y_test, y_pred))


texts: How to use trained Random forest model in cpp application?

texts: I have a trained Random Forest model which was trained using sklearn and saved as model.sav. I want to use this model in my cpp application where I will perform inference on it. What is the best way to do it? How do I read and load this model in my cpp application?


texts:
I'm trying to find an algorithm that would fit this use case:

My data: a bunch of fixed-size integer arrays, e.g.

[0,2,3,4,5]
[1,2,3,1,5]
[4,1,2,4,5]
...

Input: an array of integers, and output: prediction for the rest of the array, e.g.

[0,1] -> [2,4,2]
[3] -> [1,3,2,5,5]
[3,2,4,1] -> [4]


texts: is validation and train set should be all different files?

texts:
Let say I have train set and validation set
if 'A' included in the train set. 'A' should not be include in the validation set? or some is ok?


texts: Data prepration for logistic regression : Value either "not available" or a "year"

texts:
I have some data of houses that have been renovated.

In my data there is one column (among others) that captures this information.

It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".

I would like to apply logistic regression.

However, I do not know how to treat this value.

IMHO it looks like a missing value although it is not a missing information.

So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?

On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.

Any suggestions are highly appreciated.


texts: How to calculate $\phi_{i,j}$ in VGG19 network?

texts:
In the paper Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network by Christian Ledig et al., the distance between images (used in the loss function) is calculated from feature maps $\phi_{i,j}$ extracted from the VGG19 network,
where $\phi_{i,j}$ is defined as "feature map obtained by the j-th convolution (after activation) before the i-th maxpooling layer".

Can you elaborate on how to calculate this feature map, may be for VGG54 mentioned in the paper?

$\phi_{5,4}$  means 4th convolutional layer before 5th max-pooling layer right? But 4th layer has so 512 filters. So we would have 512 feature spaces. Which one to choose from this? Also what does "after activation" mean?

I found this answer related to the same issue, but the answer didn't explain much.


texts: How to add numbers to the axes of a graph?

texts:
I am trying to make a graph of k (as in k-means) vs error, and I can't get it to show the actual number of k on the x-axis (1-15, not just the even numbers), nor any any numbers at all if I add a title to X.

I have looked at the data frame documentation and the matplotlib axes documentation, and I couldn't find anything telling me how to get it to show the numbers? Even if I have to manually specify them?

Please see images below:

rdf.plot.line(title='Number of clusters Vs mean accuracy')

rdf.plot.line(title='Number of clusters Vs mean accuracy',
                x='parameters',
                y='mean_validation_score')

I don't understand how to get the x axis to show all 15 numbers (1-15).


texts: Clustering customer data stored in ElasticSearch

texts:
I have a bunch of customer profiles stored in a elasticsearch cluster. These profiles are now used for creation of target groups for our email subscriptions.

Target groups are now formed manually using elasticsearch faceted search capabilities (like  get all male customers of age 23 with one car and 3 children).

How could I search for interesting groups automatically - using data science, machine learning, clustering or something else?

r programming language seems to be a good tool for this task, but I can't form a methodology of such group search. One solution is to somehow find the largest clusters of customers and use them as target groups, so the question is:

How can I automatically choose largest clusters of similar customers (similar by parameters that I don't know at this moment)?

For example: my program will connect to elasticsearch, offload customer data to CSV and using R language script will find that large portion of customers are male with no children and another large portion of customers have a car and their eye color is brown.


texts: Are stopwords helpful when using tf-idf features for document classification?

texts:
I have documents of pure natural language text. Those documents are rather short; e.g. 20 - 200 words. I want to classify them.

A typical representation is a bag of words (BoW). The drawback of BoW features is that some features might always be present / have a high value, simply because they are an important part of the language. Stopwords like the following are examples: is, are, with, the, a, an, ...

One way to deal with that is to simply define this list and remove them, e.g. by looking at the most common words and just deciding which of them don't carry meaning for the given task. Basically by gut feeling.

Another way is TF-IDF features. They weight the words by how often they occur in the training set overall vs. how often they occur in the specific document. This way, even words which might not directly carry meaningful information might be valuable.

The last part is my question: Should I remove stopwords when I use TF-IDF features? Are there any publications on this topic? (I'm pretty sure I'm not the first one to wonder about this question)


texts:
I want to convert timedelta64[ns] to int

timedelta64[ns] example :
01:00:00 , 02:00:00 , 03:00:00 , 04:00:00

I want to convert as : 1, 2 , 3 , 4

Help me please T^T


texts:
Does Keras include functionality to perform something similar to following operation?
Or would it be possible to implement following 2D pooling?
As shown in the picture, I want to move a floating window of size 3x3 (or 5x5, 7x7 etc.) over a matrix to find a maximum value in given window, than divide a cell in the middle of window by found maxima and write result to the output (to the same coordinates). So in the result, local maxima will be marked by "1"s.


texts: Chi square distribution for feature selection

texts:
In one paper on ML I read that chi square distribution is used to reduce the number of features. In that paper, features are words. That paper is related to Sentiment Analysis, so we have "positive", "negative" and "neutral" category.

How to calculate chi square distribution in that case?
In Python there is scipy.stats.chisquare which gives chi_square value and a p_value. What do we do then with these two pieces of information?
What to do for example with word "good" as a feature?
How to calculate chi square distribution, and what to do with that?
What does it mean to exclude some feature from the set of features, because in that paper it is mentioned that we take n of them with top chi square.

I really don't know how to do it. If there is any paper or book or link to learn that, please tell me.


texts: Is "nb_steps_warmup" set for each episode or globally?

texts:
When I configure a DQN agent, nb_steps_warmup can be set. Is this parameter set for each episode or once globally?
What I am trying to ask is, imaging I have a game environment which takes about 3000 max. steps per episode. The DQN is fitted as follows:
dqn.fit(env, nb_steps=30000, visualize=True, verbose=2)

So, as I understand it, the fitting will run approximately 10 episodes (nb_steps / max. steps per episode).
If I set nb_steps_warmup = 5000, what actually happens?
A) nb_steps_warmup=5000, so 5000 out of nb_steps=30000 are used for warming up
B) within each episode the first nb_steps_warmup=5000 steps are used for warm up. As nb_steps_warmup > max. steps per episode, the complete episode is used for warming up, which essentially means the fitting never leaves the warming up phase.
C) sth. else?


texts: Uniform convergence garantee on sample complexity

texts:
I can't understand why the Uniform Convergence guarantees an upper bound and not a lower bound on sample complexity as stated on [1]

Corollary 4.4.
If a class $H$ has the uniform convergence property with a function $m^{UC}_H$
then the class is agnostically PAC learnable with the sample complexity
$$m_H(\epsilon ,\delta) \leq m^{UC}_H(\epsilon/2,\delta)$$
Furthermore, in that case, the $ERM_H$ paradigm is a successful agnostic
PAC learner for $H$.

From what I understood, if we have the sample set $S$ with
$$|S| \geq m^{UC}_H(\epsilon/2,\delta) $$
then, by definition of the UC property, for every $\epsilon,\delta \in (0, 1)$ and for every probability distribution D over Z, with probability of at least $1 − \delta$, $S$ is $\epsilon/2$-representative.
Then, by Lemma 4.3,
$$L_D(h_S) \leq min_{h \in H}{L_D(h) +\epsilon},\ h_S = ERM_H(S)$$
Therefore, by definition, the class $H$ is agnostic PAC learnable
with a sample complexity
$$m_H(\epsilon, \delta) \geq m^{UC}_H(\epsilon/2,\delta)$$
Is that proof correct? Thank you.
[1] Understanding Machine Learning By Shai Shalev-Shwartz and Shai Ben-David, pag 32 (https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/)


texts:
I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2.

How can I automatically generate one graphic for each achievement as in the file attached here.

As I have many KPIs I would like to generate my graphics in a batch process


texts: How to normalize technical skills of IT using machine learning?

texts:
I have a huge collection of skills collected/scraped from various online sources. It was huge effort done by our team.
Now, the biggest challenge we are facing is trying to normalize the skills back to its root form to reduce the duplication.
Here's are few examples,
agile    (or) agile methodologies  (or) agile software
java 2.x (or) java 3.x (or) java

These kind of duplicates are very common is my datasets. So I'm looking for a way to normalize them.
I am not very sure of the way to solve this issue. Can anyone suggest me some very good ways we can solve this problem with some decent accuracy?
Thank you


texts: Antisymmetry of graph of Information vs probability

texts:
The formula for Information given by a data of occurring with probability p is:
I=-log2 p

This formula gives the bits if information needed to know the outcome of the event.

This formula captures the intuition that the information needed to know the outcome of an event with probability 1 is 0 as we already know that outcome of the event.

So shouldn't the formula give the information as 0 for the event with probability 0 as we know the outcome of the event.

So shouldn't the graph of the I vs p be a symmetric as the values of p close to 1 will give similar information as the events with p close to 0?


texts: RUL prediction without failures in historical data

texts:
I have faced in the past some problems of predictive maintenance where I had historical sensor data with failures. With this kind of dataset, you can calculate the RUL (Remaining Useful Life) and label the data accordingly. That allows you to:

Perform a regression, trained to predict the RUL
Label data before the fail and try to predict it (raising a notice like: be careful, the system is going to fail "soon").
Label data performing a multi-class classification. Like the step before, but placing more labels to predict (it should tell you if you are in a healthy, warning, or close-to-error status, for example)

By the way, this time I will have historical data without failures (they did not and will not perform a RTF, Run To Failure). I know that in a specific moment someone did the maintenance program, right before the system breaks. I indeed will have the minimum allowed sensor value.

How can I face this kind of problem, doing predictive maintenance and telling that they could perform their maintenance "a little bit" later without incurring in a failure? I was thinking about survival analysis (I have never done it before), but I do not think I will have technical specs of the hardware showing the estimated life of a particular part.

I do not have the data yet, but I am starting to guess how I will face that problem. I created a simple example of what I should expect (with a single sensor, I guess I will have more).

In my test data I put a vertical line where I assume they performed maintenance, The horizontal line is the limit that the sensor should never cross before getting an error. The data after the vertical green line is the data I will never have in my dataset.

import matplotlib.pyplot as plt
import pandas as pd

min_s1 = 23;
replace_cycle = 20
df_1 = pd.DataFrame(data={'s1':[100,99,98,98,97,95,92,88,84,76,70,64,55,45,40,35,32,30,29,28,27,27,26,25,25,24,24,23,23]})
df_2 = pd.DataFrame(data={'s1':[99,97,98,97,96,95,91,85,80,71,65,57,49,40,35,31,28,27,27,26,26,25,25,24,23,23,23,23,23]})
df_3 = pd.DataFrame(data={'s1':[100,98,98,97,97,96,93,89,86,78,72,67,59,50,43,36,33,31,31,29,30,28,28,27,26,26,25,25,24]})

fig, ax = plt.subplots()

df_1.plot(ax=ax, style='.-');
df_2.plot(ax=ax, style='.-');
df_3.plot(ax=ax, style='.-');
plt.axhline(min_s1, color='red', linestyle='--');
plt.axvline(replace_cycle, color='green', linestyle='--');
ax.legend(['s1 registered run #1','s1 registered run #2','s1 registered run #3','s1 min value','s1 maintenance']);


texts: How to use 5 by 5 or 7 by 7 kernel size for a deep learning network with 3 by 3 kernels?

texts: I am using a U-Net architecture. The visual area of the segmentation mask is very small and after learning it is giving a lot of false positives. I am thinking of changing the kernel size of 3 by 3 to 5 by 5. What precautions/care shall I take while creating the model? I am getting issue of dimensionality mismatch if just change 3 by 3 to 5 by 5 kernel size.