texts: Intuition in Backpropagation (gradient descent)

texts:
I am reading about backpropagation, one of many processes involved in gradient descent, and I couldn't quite grasp the intuition in one of the equations.

http://neuralnetworksanddeeplearning.com/chap2.html#MathJax-Element-185-Frame

  An equation for the error $\delta^l$ in terms of the error in the next layer, $\delta^{l+1}$: In particular

  $$
\delta^l=((w^{l+1})^T\delta^{l+1})\circ\sigma'(z^l),
$$

  where $(w^{l+1})^T$ is the transpose of the weight matrix $w^{l+1}$ for the $(l+1)^{th}$ layer.

  ...

  When we apply the transpose weight matrix, $(w^{l+1})^T$, we can think intuitively of this as moving the error backward through the network, giving us some sort of measure of the error at the output of the $l^{th}$ layer.

I know how each terms are derived algebraically (from further down the text), but I can't grasp the intuitive part. In particular, I don't understand how applying the transpose of the weight matrix is like moving the error backwards.

The mathematics is quite straightforward. The equation is the simplification of the error in layer $l$ expressed in terms of error in layer $l+1$. But, the whole intuition bit doesn't come so naturally.

Could someone please explain, in what sense applying the transpose is like "moving the error backwards"?


texts: Using Residual Plots to combat Simpson's Paradox and Other Misleading phenomenon

texts:
I've read that univariate analysis is a huge part of EDA, pretty much everywhere. What I understand of univariate analysis basically comes down to using Seaborn pairplot and correlation matrices and heatmaps.

However, today I read about "Simpson's Paradox" on a kaggle kernel, then in a Britannica article. It was quite fascinating and seemed to render useless what I believed to be commonly accepted best practices.

I then read that one way to combat this sort of misleading analysis is through partial residual plots:

$$\text{Residuals} + \beta_iX_i \text{ vs } X_i$$

What I can't figure out is how this formula controls for multiple independent variables in an intuitive way. What's the explanation for how this works?


texts: Can anyone help with me how to train new data with already saved pickle file?

texts:
Can anyone help me with the code me to train new data with already saved pickle file?
I've trained the model with RandomForestClassifier from sklearn and saved the model into .pickle
Now I'm trying to retrain the model on new data with same features. I want to use pickle file to train one new data. Can anyone help me with this in terms of code?


texts: Machine Learning Method For Creating Chemical Formula

texts:
We work at chemical company . We have nearly 3000 chemical formulas which is composed with chemical raw materials.
Our chemical formulas is composed of 20-25 raw materials. As you guess, amount of these raw materials are very important for our formulas. As a result we obtain different products via using different raw materials and different amount of raw materials.
Here is some examples:
Sample Formula 1:
%10 Raw material A
%25 Raw material B
%5 Raw material C
%60 Raw material D

Result product has such properties :
Property X: 3
Property Y: 10
Property Z: 2

What kind of machine learning method or methods should we use? I have looked for CNN but as you see there are many input and output properties as parameters.
Our goal is training a network via using our exist formulas then we will want that our network will create new formulas according to input parameters which we will give.


texts: Finding parameters of image filter using classified pairs

texts:
I want to solve the problem of finding a parameter vector for an image filter (let us assume we know nothing about how the filter works, but we can feed it an input image and a set of parameters to produce an output image).

Thus, having a set $\{{I_k, J_k:=F_{\alpha}(I_k)}\}_{k\in\overline{1,N}}$ of $I_k$ images together with their filtered counterparts, $J_k$, what solutions would you recommend for finding $\alpha^\ast$ such that given $I^\ast$ the result $F_{\alpha^\ast}(I^\ast)$ is in the same "style" as the one of the $N$ training correspondence pairs.

I suppose one option  is to use a convnet to transform $I_k$ into a feature vector, $v_k$, and then concatenate $\alpha_k$ to obtain $u_k =(v_k,\alpha_k)$. Once this is done, use a regression method to estimate the $\alpha^\ast$ part of $u^\ast$.

I would like to find an alternative solution to what seems like a candidate for the style transfer approach (e.g. https://arxiv.org/pdf/1703.07511.pdf). That approach seems to solve the problem differently, and I envision situations where I need to simply use a filter rather than let a network "guess the style of that filter".

Additional details and possible assumptions

Given the invoked no free lunch prospects, let us assume, for a paeticular problem from this class, that $F$ is a non-linear kernel-based filter that maps $I$ to $J$ as a result of an iterative and convergent process. More specifically, let $F$ be a mean shift filter with the $\alpha=(\rho, \sigma_s,\sigma_r)$ using a concatenated Gaussian kernel and a Parzen window of size $\rho$. Intuitively, I would be tempted to guess that this filter is not smooth w.r.t. $\alpha$, but a formal investigation is required (I suspect it is not smooth given that infinitesimal changes in the size of the window could shift the output towards another mode, indicating a step function behaviour).

In general, it is correct to assume that $\alpha \in \mathbb{R}^d$, with $d \ll N$.

Given the goal of finding $\alpha$ when both the filter action is known (either via numerical computation in general, or, in closed-form if the filter is a gaussian blur, for example), we can be confident that the $N$ input samples have non-constant $\alpha_k$ vector values to start with.

But for sake of generalizability, it would be more elegant to pursue a solution that does not need to know how the filter operates without actually applying it to an input. The first approach suggested in the comments and based on convnets seems to fit this scenario and the optimization problem is taking into account the filter error. However, it would be interesting to hear more opinions, perhaps involving shallow approaches, even at the expense of designing the solution to address the concrete mean shift filter example from above.


texts:
Goal: Visualize why loans are rejected which can be due to multiple reasons. A loan is either rejected or accepted. Any rejection
  is caused by one or more rejection reasons. There is a list of
  rejection reasons which has about 20 entries. The viewer must
  understand that

  loans are often rejected by multiple reasons and
  which are the most prevalent reasons.

  more than one chart. I also thought of small multiples.This can be achieved in one chart or

What is a good way of visualizing this?

That's how I tried to visualize it

import matplotlib.pyplot as plt
import seaborn as sns

def explode_reasons(reasons):
    all_reasons = [f'reason_{i}' for i in range(20)]
    tmp = {x: x in reasons for x in all_reasons}
    return pd.Series(tmp)

df = pd.concat([df, df['reasons'].apply(explode_reasons)], axis=1)

reason_cols = [f'reason_{i}' for i in range(20)]
sns.heatmap(~df.loc[:, reason_cols].sort_values(reason_cols, ascending=False), cbar=False)

That's how my data looks like.

|   id | rejected   | reasons                                 |
|-----:|:-----------|:----------------------------------------|
|    0 | False      | []                                      |
|    1 | False      | []                                      |
|    2 | False      | []                                      |
|    3 | True       | ['reason_18', 'reason_10', 'reason_13'] |
|    4 | True       | ['reason_15', 'reason_8', 'reason_2']   |

That's how it can be reproduced

import pandas as pd
import numpy as np
import random

# generate dummy data
df = pd.DataFrame({'id': range(100),
                'rejected': random.choices([True, False], weights=[80, 20], k=100)})

def generate_rejections(rejected):
    if not rejected:
        return []

    reasons = random.choices([f'reason_{i}' for i in range(20)],
                            weights=range(1, 21),
                            k=random.randint(1, 5))
    return list(set(reasons))

df['reasons'] = df['rejected'].apply(generate_rejections)


texts: Why is Distributional DQN faster than vanilla DQN?

texts:
Recently I learned about Distributional approach to RL, which is a quite fascinating and break--through algorithm.

I have 2 questions:

What is it that makes it perform so much better during runtime than DQN? My understanding is that during runtime we will still have to select an action with the largest expected value. But to compute these expected values, we will now have to look at distributions of all possible actions at $x_{t+1}$, then select an action with a highest expected value. This would actually mean during extra work during runtime

What is the explanation for its faster converge than that of vanilla DQN? As I understand, the policy hasn't changed, we are still selecting the best action from state $x_{t+1}$, then use its best action's distribution for bootstrapping (adjusting) the distribution of our current state's best action.

Where does the Distributional part come into play and make the network be smarter about selecting the actions? (currently we still always select highest expected action as "the target distrib").


texts:
Is it possible to use the Levenshtein (edit) distance of two strings as the error function my model attempts to optimize?

The inputs are images representing the letters in a text document. The output should be the predicted letters from the image. I want to combine them into a text document, and then adjust the model weights to minimize the levenshtein distance of the input document, and the predicted document. Can I do this by extending nn.Module to create a loss function in pytorch, and then using loss.backward to update the weights in the model?


texts: I would like to ask if the matrix that is returned from the R cor() function applied on a dataset, is considered a confusion matrix. Thank you


texts:
I am trying to train a deep network for twitter sentiment classification. It consists of an embedding layer (word2vec), an RNN (GRU) layer, followed by 2 conv layers, followed by 2 dense layers. Using ReLU for all activation functions.

I have just started using tensorboard & noticed that I seemingly have extremely small gradients in my convolutional layer weights (see figure)

I believe I have vanishing gradients since the distribution of CNN filter weights does not seem to change & the gradients are extremely small relative to the weights (see figure). [NOTE: the figure shows layer 1, but layers 2 looked very similar]

My questions are:

1) Am I interpreting the plots correctly that I do indeed have vanishing gradients & thus my convolutional layers arent learning? Does this mean they are currently essentially worthless?

2) What can I do to remedy this situation?

Thanks!

UPDATE 3/13/18

Few comments:

1) I have tried the network w/ just 1 layer and no layers (RNN-->FC), and having 2 kayers does empirically improve performance.

2) I have tried Xavier initialization and it doesnt do much (the previous default initialization mean value of .1 was very close to the Xander value)

3) By quick math, the gradients seem to change on the order of 1e-5, while the weights themselves are on the order of 1e-1. Thus at every iteration, the weights change 1e-5/1e-1*100% = ~.01%. Is this to be expected? What is the threshold for how much the weights change until we consider them to have converged / consider the changes to be useless in the sense that they dont change outcome?


texts: Is the activation function the only difference between logistic regression and perceptron?

texts:
As far as I know, logistic regression can be denoted as:

$$
f(x) = \sigma(w \cdot x + b)
$$

A perceptron  can be denoted as:

$$
f(x) = \operatorname{sign} (w \cdot x + b)
$$

It seems that the only difference between logistic regression and a perceptron model is the activation function.

Is this correct?


texts: Why do we need to add START  + END  symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

texts:
In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to the inputs and outputs before training the model and before inference/decoding unseen data.

E.g. http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

SOS_token = 0
EOS_token = 1

class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.n_words
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1

Is there a technical definition or academic explanation of why that is necessary?
Or does that need to add the END symbol only applies to Natural Language Processing task where the sentence generation needs to end?
But what does the START symbol do? Other than to take the initial state where the trained network will start inferrring.


texts: What are references for problems where XGboost was successfully used?

texts: I created a successful XGboost model but my boss would like me to give him references on the xgboost, where it was successfully used. Do you happen to know if any company used it for any purpose? Insurance companies for lapse prediction or I don't know. I know xgboost beat the Netflix's original algorithm for movie recommendation but I can't find it anywhere. Could you give me something on it? Thank you in advance!


texts:
I am working on a card game for openai gym and currently I ask myself how to shape the reward function for it. One round of the game consists of each player picking a card from its hand, whereas not every card can be played depending on the card which has been played by one of the players before. For every set of played cards, there is a total order such that the player with the highest card wins the round.
In the situation in which cards are rejected I want to give the agent some reward.

In case of an invalid card, it is hard to say if that card is any nearer to one of the valid cards than any other. Also the agent should learn that this card is not playable at this point.

For completeness, the agent gets a discrete observation of everything it can remember of the game (its own cards, cards played in current round, cards played in past rounds, game mode (defines the total order of cards)). Then it should play a discrete action which either is a game mode in the beginning or a card during the round. Then it either gets a reward because its card got rejected or it gets a reward based on whether it wins the round or not. The game accounts a certain amount of points for a won round depending on the constellation of played cards in that round.

My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?

In case any more details are required, just ask for them.


texts: I am trying to understand this paper about conditional GAN, it says that extra information y (class labels) is given to the network. However, I cannot understand its usage during training or its benefits. As far as I know, GAN is unsupervised learning, in this case of extra information usage, can we say that the architecture is supervised?


texts: Why don't different output weights break symmetry?

texts: My deep learning lecturer told us that if a hidden node has identical input weights to another, then the weights will remain the same over the training/there will be no separation. This is confusing to me, because my expectation was that you should only need the input or output weights to be different. Even if the input nodes are identical, the output weights would affect the backpropagated gradient at the hidden nodes and create divergence. Why isn't this the case?


texts: How to select particular column in Spark(pyspark)?

texts:
testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)

I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute 'select'


texts: Does it make sense to apply clustering on aggregation of data?

texts:
I was wondering if it makes sense to apply clustering techniques on an aggregation of data, like, I have three different sources of data such as S1 S2 and S3 where each of these sources share some common columns but the majority are not shared. Does it make sense to group all the sources with all the columns in a big dataframe and apply clustering techniques whereas some records will only have null values for the columns that are not part of its corresponding service.
Thank you.

Update:
The input is basically logs coming from different services with different columns where some of them are shared between all services. The output should be a cluster of records representing a user having the same behavior.
The logs are gathered by second corresponding of a user action. They are aggregated by hours for each user to derive other features (the granularity of seconds it according to me, too much).
The goal, is to detect anormal behaviors.
And the question is, can I regroup different dataframe having different features while some of them are shared and run a K-mean on it. Because my dataframe would look like this (s* for service, and common_ for common feature):

-------------------------------------------------------------
s1_f1 | s1_f2 | s2_f1 | s2_f2 | s3_f1 | common_f1 | common_f2
  1       OK     NULL    NULL    NULL     midday      less
  2       OK     NULL    NULL    NULL     midnight    more
 NULL    NULL     2       5      NULL     midday      less
 NULL    NULL     8       9      NULL     morning     less
 NULL    NULL    NULL    NULL    777      morning     more
 NULL    NULL    NULL    NULL    888      night       more


texts: Spark development on local machine with PyCharm

texts:
I am wondering what is the best practice other devs are using for their python spark jobs.
I am building a dev environment in which I am looking to write code in PyCharm with SparkContext pointing to a standalone cluster and being able to run my code from PyCharm or at least from the command line onto the cluster. I am using Windows 8.

Any success stories with such set up? what is the recommended setup/practice?


texts: Memory error - Hierarchical Dirichlet Process, HDP gensim

texts:
I am running Hierarchical Dirichlet Process, HDP using gensim in Python but as my corpus is too large it is throwing me following error:

model = gensim.models.HdpModel(corpus, id2word=corpus.id2word, chunksize=50000)

 File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 210, in __init__
    self.update(corpus)
  File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 245, in update
    self.update_chunk(chunk)
  File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 313, in update_chunk
    self.update_lambda(ss, word_list, opt_o)
  File "/usr/cluster/contrib/Enthought/Canopy_64/User/lib/python2.7/site-packages/gensim/models/hdpmodel.py", line 415, in update_lambda
    rhot * self.m_D * sstats.m_var_beta_ss / sstats.m_chunksize
MemoryError

I have loaded my corpus using following statement:

corpus = gensim.corpora.MalletCorpus('chunk5000K_records.mallet')

And the data which I used to load corpus has 5 million records. And this is working for me when I am loading only 50K records. So I have added chunksize option HdpModel but it is still giving me an error.

Please let me know how I can solve this issue. And I am running this on High Performance Computing so I think there should be a solution to resolve this issue as this cluster has really big size memory and disk capacity.


texts: How to predict when and why of hospitalization?

texts:
I have an EHR data source which has info on
a) Patient visit records (Inpatient, outpatient, Emergency etc) and why did he visit hospital (diagnosis codes attached to each visit)
b) Patients drugs data
c) Patients lab test data
d) Patient diagnosis history
e) Patient demographics data
If my objective had just been to predict when will we visit hospital next, I can use Time to event analysis or survival analysis etc.
But, I would also like to know why will he visit hospital next or what might be the reason of his next hospital visit?
While predicting "when" seems to be doable, can experts here guide me on how can I find why/what will be the reason for his next hospital visit?


texts:
I have a pandas table with a start and end time for each entry. They don't intersect, but range from seconds to days. I want to visualize them, ideally the x-Axis being the date on which they occus, and the y-axis the time. However, I haven't found a good way to achieve this.

One thought I had were using Boxplots (one Box with one start and one endtime per entry) and grouping by [year, month, day], however this idea doesn't work when the interval starts and ends on a different day.


texts: In calculating policy gradients, wouldn't longer trajectories have more weight according to the policy gradient formula?

texts:
In Sergey Levine's lecture on policy gradients (berkeley deep rl course), he show that policy gradient can be evaluated according to the formula

In this formula, wouldn't longer trajectories get more weight (in finite horizon situations), since the middle term, the sum over log pi, would involve more terms? (Why would it work like that?)

The specific example I have in mind is pacman, longer trajectories would contribute more to the gradient. Should it work like that?


texts: Evaluation method for multi-class classification problem modeled as binary classification problem

texts:
I should mention that even though I have some basic knowledge regarding ML, it is the first big ML project I am working on and for the proposal of my research project I need to suggest an evaluation metric.

The problem is a multiclass(16 classes) classification problem where one data point can be classified in multiple classes (not ranking based though). I plan to model it as a binary classification problem for each class but for the related evaluation metrics I was not able to find a proper application. So, first of all, should I evaluate individual performance for each class (how well class A classification is working), should I go for a general evaluation (This data point belongs A,B,C but at the end classified for A and B only), or both? Second, what kind of metrics can I have a look at? Finally, I haven't started working on the data yet but I expect an unbalanced distribution for my classes. Would it affect my results?


texts: Cannot feed appropriate datatype to model from CSV

texts:
DESCRIPTION: When training my model and 3/4 of one epoch completes I get an error "ValueError: could not convert string to float" by running dataset( see below ) I can see the first column is int64 second non-null object along with the third. This is a problem because Keras needs the data in a number format.

What I have tried: applying a type change using Pandas does not work I get a similar error " could not convert string to float " and even so I cannot change the the type prematurely do to needing the string data in column 3 tokenized i'm stuck in this situation because I do not know how to proceed in converting the data to the correct format(s)

How I tried to change the data with Pandas: data["State"] = data["State"].astype(float) this column contains just label ed values 0-2
corresponding to sentiment

Data Sample Of My CSV File

itemID,State,Sentence
1, 1,"I feel good"
2, 1,"Im feelingbetter"
3, 1,"I feel so good"
4,  1,"Im feeling great"
5,  1,"Im feeling fine"

CSV OVERVIEW  Output of "data.dtype"

RangeIndex 4651 entries, 0 to 4650
Data columns (total 3 columns )
itemID 4651 non-null int64
State 4651 non-null object
Sentence 4650 non-null object

dtypes int64(1) , object(2)

The Code:

    max_words = 500
data = pd.read_csv("data.csv", sep=',', skip_blank_lines=True)
data = data[data["Sentence"].notnull()]
data.info(memory_usage='deep')
data.info()
X = data.loc[:, "Sentence"]
Y = data.loc[:, "State"]

data["Sentence"] = data["Sentence"].apply(lambda x: x.lower())
data["Sentence"] = data["Sentence"].apply((lambda x: re.sub('[^a-zA-z0-9\s]','',x)))

max_features = 3000
tokenizer = Tokenizer(num_words=max_features, split=' ')
tokenizer.fit_on_texts(data['Sentence'].values)
dictionary = tokenizer.word_index
with open('dictionary.json', 'w') as dictionary_file:
    json.dump(dictionary, dictionary_file)

def convert_text_to_index_array(text):
    return [dictionary[word] for word
            in kpt.text_to_word_sequence(text)]

allWordIndices = []
for text in X:
    wordIndices = convert_text_to_index_array(text)
    allWordIndices.append(wordIndices)

allWordIndices = np.asarray(allWordIndices)
X = tokenizer.sequences_to_matrix(allWordIndices, mode='binary')
X = sequence.pad_sequences(X, maxlen=max_words)
X = X.reshape((len(X), 500, 1))
print('Building STATELESS model...', X.shape)

model = Sequential()
model.add(LSTM(32, input_shape=(500, 1)))
model.add(Dense(3, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer="rmsprop", metrics=['accuracy'])

print(model.summary())
filepath = 'model_sentiment.h5'
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20, random_state=42, shuffle=True)

hist = model.fit(X_train, Y_train, validation_split=0.20, epochs=1, batch_size=32, callbacks=callbacks_list, verbose=1)
loss, accuracy = model.evaluate(X, Y, verbose=0)
print('Accuracy: %f' % (accuracy*100))
print('***** TOTAL TRAINING SET AMOUNT', X_train.shape, Y_train.shape)
print('***** TEST GENERALIZATION', X_test.shape, Y_test.shape)
print(model.summary())

X_val = X_train[:500]
Y_val = Y_train[:500]
partial_X_train = X_train[500:]
partial_Y_train = Y_train[500:]
validation_size = 1500
X_validate = X_test[-validation_size:]
Y_validate = Y_test[-validation_size:]
X_test = X_test[:-validation_size]
Y_test = Y_test[:-validation_size]
score, acc = model.evaluate(X_test, Y_test, verbose=2, batch_size=32)
print('score: %.2f' % (score))
print('acc: %.2f' % (acc))


texts:
Is there a closed form solution for a ReLU that is bounded with a maximum value at 1?
I am trying to produce output values for pixel intensities 0 <= x <= 1, but my outputs are producing values greater than 1. How can I counteract this?


texts: How can I "export" a model from Orange once I have prototyped a solution?

texts:
Using Orange, I would like to use the underlying python model within the control system of my device.  The device is a simple machine which is not connected to the internet.  My intent is to use the model to the machine operating parameters based upon the application requirements.

Is there a way (or could there be) to export the trained model for use?  This seems to be a missing feature or future enhancement for Orange.


texts: How to figure out if two sentences have the same meaning with AI?

texts:
I have two sentences which might be similar in meaning.
Are there a useful and successful (machine learning) algorithms, which is able to determine the semantic similarity?

Are there any approaches which can handle this task and are easy to use?

An approach via Keras or Tensorflow as a Framework would work perfectly in my use case.

Thank you!


texts: How many vectors does paragraph vector generate for each paragraph?

texts: For example,if I have a corpus with two paragraphs, does paragraph vector generate two vectors?Additionally, on Distributed Representations of Sentences and Documents (Q. Le, T. Mikolov) paper I do not understand why paragraph vectors are unique among paragraphs but the word vectors are shared. Why?


texts: How can we create an label, value detector?

texts:
I am trying to implement an text detector using MaskRCNN such that the model detects the label and value as shown in the image below.

Detecting the same is easier for fields like page date and order no. but fails for invoice number and salesman shown in image below. Is there any pretrained models for the same? or some other technique that might help.


texts: how to check all values in particular column has same data type or not?

texts:
I have column 'ABC' which has 5000 rows. Currently, dtype of column is object. Mostly it has string values but some values dtype is not string, I want to find all those rows and modify those rows. Column is as following:

1 abc
2 def
3 ghi
4 23
5 mno
6 null
7 qwe
8 12-11-2019
...
...
...
4900 ert
5000 tyu

In above case, I can use for loop to find out rows which do not have desired dtype. I just wanted to know, is their better way to solve this issue.

Note: I am using Pandas.


texts: Binary classfication vs One-class classification

texts:
Why do we need samples of both classes for the training of binary classification algorithms, if one-class algorithms can do the job with only samples from one class?

I know that one-class algorithms (like one-class svm) were proposed with the absence of negative data in mind and that they seek to find decision boundaries that separate positive samples (A) from negative ones (Not A).

Hence the traditional binary classification problem (between (A) and (B) for example) can be formulated as a classification of (A) and (not A = B).
Is it about better classification results or am I missing something?
Thank you in advance


texts: Creating labels for Text classification using keras

texts:
I have a text file with information that needs to classified based on keywords. The text file contains many number of paragraphs. And the paragraph contains keywords that we want (lets say salary amount, interest rate and so on..)

I want to write a model which will extract the paragraph (or 3 to 4 lines of text) containing the keyword i want. How do i create a label in this case? All i have is a raw text.

I am new to NLP. Any suggestions how i can approach this?


texts:
I've created spectrogramms of different classes of low-hertz signals. They all have a plain blue foreground with hardly other coloured pixels, even for me its not easy to distinguish the classes by human sight. Now I'd like to train a CNN to do binary classification on these spectrogramm images. No matter how I build the network with no matter what configurations and parameters - it doesn't learn, loss doesn't decrease.
I figured out the reason: the convolutional filters of CNNs are good at distinguishing forms and shapes, but not really plain colours.
How can CNNs learn colours? Am I missing something? Perhaps there are more suitable models than CNNs?
Here is an example of the two classes that hardly differ, only in the colour hue:

2 more images:

EDIT1: The two classes have images that not all look the same with the same prevailing colour; some have yellow/greenish stripes in them, some have lighter or darker colour etc. So the classes do not only consist of the feature colour. My goal is to classify these images using CNNs, but somehow CNNs fail at learning from images, that almost only consist of colour, whereas CNNs learn well if there are sharp edges, boundaries, object-like elements in the image.


texts: Branching decision tree exit node and visitation duration classfication

texts: I have a dataset where every record in each data cluster corresponds to a step in a branched tree (from an IVR system). I am still learning some of the more machine learning concepts of data science and was wondering for pure detection of exit nodes and detecting time in system, would a flat file analysis be a better option, or is there a better approach that would be faster considering I am dealing with a large (for me) dataset of 2-4 Mil records?  I apologize if this is asked and answered, as I am definitely in a don't know what I don't know scenario here, and would appreciate any help.  I cannot post any of the dataset as it is sensitive info.


texts:
As the title say I have been looking for some online courses that would teach me about anomaly detection using Unsupervised Machine Learning.

I want to focus only on Scikit-Learn and not go deeper into TensorFlow or other options.

I have been looking at some Udemy courses but not really sure.

Decided to write here since the community here has a lot of knowledge and might lead me into the right path.
I am willing to pay for any course as long as it fulfills what I am looking for.


texts: What does test score provide in terms of real application?

texts:
Initially, I wanted to create a classification tool for some program codes in a specific environment based on the application patterns they show. I had 5 different patterns I wanted to look for and eventually my aim was to come up with some kind of statistics about this programming environment that somehow represents the trends (like %40 of the programs are for voting purposes, there has been an increase in the trading programs in the last month, etc). Now, after months of labeling data and training models, I finally obtained 5 different models each one for a different pattern that has varying scores on different metrics (I had quite an unbalanced training data for each class, so just to keep things somehow consisted, always tried to look at different metrics like f1, mcc, roc auc all at once for comparison).

Now, my question is, what do exactly my test scores tell me about my real life data when I apply my models to data that is not possible for me to label or see? Is it possible to derive different statistics as mentioned above using my trained models or am I only limited to my training & testing data? Is it at least possible to mention some kind of probability on a single program, like this is predicted to be a voting program with %X probability?

I would appreciate if you could help me figure out how to interpret the test results from a perspective of final application.


texts: PyTorch MultiLayer Perceptron Classification Size of Features vs Labels Wrong

texts:
I am getting the following error:

/usr/localnfs/Compiler/python/Anaconda3-2019.07/lib/python3.7/site-packages/torch/nn/modules/loss.py:431:
UserWarning: Using a target size (torch.Size([30, 1])) that is different to the input size (torch.Size([30, 10])).

This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

I think I know why this is happening but I don't know how to fix it. I am using the loss function torch.nn.MSELoss(), with no inputs. My model is the multilayer perceptron, and this is how I'm doing it:

class MLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MLP, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.fc1 = nn.Linear(self.input_size, self.hidden_size)
        self.fc2 = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, features):
        out = self.fc1(features)
        out = self.fc2(out)
        return out

My input size is 100, my hidden size is 50, and my output size is 10, because my dataset has 10 columns with 30 rows (the size of my mini batch). The size of my labels is 30x1.

 predict_label = mlp_model(features.float())
 loss_batch = loss_func(input=predict_label, target=label.float())

I am somehow supposed to predict labels for my 30x10 dataset, and compare them to my 30x1 label vector, and I don't know how to do that. The error shows up when I execute the loss_batch line. Please help, I'm very new to this.


texts: Where can I download the toy benchmark dataset for RNNs?

texts:
I have read the paper:
Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Where can I download the toy benchmark dataset for RNNs this paper mentions?

I need addition problem benchmark.


texts:
I am reading about federated learning and have a quick question

1) I know in federated learning, the model updates are shared to a central server

2) All the parties involved in FL can generate benefits because their model has seen more variation in data (due to different parties involved)

But my question is,

Let's say Site A contributes/has 80% (more data points) of the data and site B has only 20% (less data points).

So we know that in this case, Site B will benefit more from the model because with little data he gets to have a better model whereas Site A even with more data contribution has only very little to gain.

For ex in a commercial setting, let's say they want to build a model to detect fraud/anomalies.

In this who will get the most monetary returns? Is it Site A or Site B?

Can you please direct me to research articles, opinion articles and tutorial regarding this please?


texts: Modelling in python and scoring in MATLAB?

texts: I have model objects either pickled object or H2O POJO. Is it possible to call those objects and do the scoring in MATLAB?


texts: Smoothing Proportions :: Massive User Database

texts:
What are some possible techniques for smoothing proportions across very large categories, in order to take into account the sample size? The application of interest here is to use the proportions as input into a predictive model, but I am wary of using the raw proportions in cases where there is little evidence and I don't want to overfit.

Here is an example, where the ID denotes a customer and impressions and clicks are the number of ads shown and clicks the customer has made, respectively.


texts: Caffe net.predict() outputs random results (GoogleNet)

texts:
I used pretrained GoogleNet from https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet and finetuned it with my own data (~ 100k images, 101 classes). After one day training I achieved 62% in top-1 and 85% in top-5 classification and try to use this network to predict several images.

I just followed example from https://github.com/BVLC/caffe/blob/master/examples/classification.ipynb,

Here is my Python code:

import caffe
import numpy as np

caffe_root = './caffe'

MODEL_FILE = 'caffe/models/bvlc_googlenet/deploy.prototxt'
PRETRAINED = 'caffe/models/bvlc_googlenet/bvlc_googlenet_iter_200000.caffemodel'

caffe.set_mode_gpu()

net = caffe.Classifier(MODEL_FILE, PRETRAINED,
               mean=np.load('ilsvrc_2012_mean.npy').mean(1).mean(1),
               channel_swap=(2,1,0),
               raw_scale=255,
               image_dims=(224, 224))

def caffe_predict(path):
        input_image = caffe.io.load_image(path)
        print path
        print input_image
        prediction = net.predict([input_image])

        print prediction
        print "----------"

        print 'prediction shape:', prediction[0].shape
        print 'predicted class:', prediction[0].argmax()

        proba = prediction[0][prediction[0].argmax()]
        ind = prediction[0].argsort()[-5:][::-1] # top-5 predictions

        return prediction[0].argmax(), proba, ind

In my deploy.prototxt I changed the last layer only to predict my 101 classes.

layer {
  name: "loss3/classifier"
  type: "InnerProduct"
  bottom: "pool5/7x7_s1"
  top: "loss3/classifier"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  inner_product_param {
    num_output: 101
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "prob"
  type: "Softmax"
  bottom: "loss3/classifier"
  top: "prob"
}

Here is the distribution of softmax output:

[[ 0.01106235  0.00343131  0.00807581  0.01530041  0.01077161  0.0081002
   0.00989228  0.00972753  0.00429183  0.01377776  0.02028225  0.01209726
   0.01318955  0.00669979  0.00720005  0.00838189  0.00335461  0.01461464
   0.01485041  0.00543212  0.00400191  0.0084842   0.02134697  0.02500303
   0.00561895  0.00776423  0.02176422  0.00752334  0.0116104   0.01328687
   0.00517187  0.02234021  0.00727272  0.02380056  0.01210031  0.00582192
   0.00729601  0.00832637  0.00819836  0.00520551  0.00625274  0.00426603
   0.01210176  0.00571806  0.00646495  0.01589645  0.00642173  0.00805364
   0.00364388  0.01553882  0.01549598  0.01824486  0.00483241  0.01231962
   0.00545738  0.0101487   0.0040346   0.01066607  0.01328133  0.01027429
   0.01581303  0.01199994  0.00371804  0.01241552  0.00831448  0.00789811
   0.00456275  0.00504562  0.00424598  0.01309276  0.0079432   0.0140427
   0.00487625  0.02614347  0.00603372  0.00892296  0.00924052  0.00712763
   0.01101298  0.00716757  0.01019373  0.01234141  0.00905332  0.0040798
   0.00846442  0.00924353  0.00709366  0.01535406  0.00653238  0.01083806
   0.01168014  0.02076091  0.00542234  0.01246306  0.00704035  0.00529556
   0.00751443  0.00797437  0.00408798  0.00891858  0.00444583]]

It seems just like random distribution with no sense.

Thank you for any help or hint and best regards, Alex


texts: difference between logistic regression and binary logistic regression

texts:
In xgboost R package, there are two objectives given with booster gbtree.

reg:logistic
binary:logistic

See,page 22 (first 2 lines) https://cran.r-project.org/web/packages/xgboost/xgboost.pdf

I was wondering, what is the difference between these two methods?


texts: Finding out which values lead Random tree to a decision

texts:
I have a dataset of machines that produce plastic parts. A camera evaluates whether a plastic part was produced correctly or not (binary classification). I'm trying to figure out which factors influence a part being wrongly produced. E.g. I have different temperature values of the machine parts during the production.

I'm using a Random Forest to classify the data. The test dataset is being recognized quite well. The next step is to figure out which values lead to a wrongly produced part (e.g. when temperature > 150K: Part is broken). I've searched the internet but I couldn't find any information about this.

At the moment I'm trying a brute force method where I simply generate a test dataset where I go through different value ranges. But so far everything is classified as wrongly produced part.

Are there other methods I can use to get the values?

Thank you!


texts: How can realize the evaluation/validation of unsupervised models through unlabeled data?

texts:
I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Recently I re-faced to following fundamental questions and literature review in this regard to updating myself and find clear answers:

Deep Learning (DL) is supervised or unsupervised?
Does splitting data make sense in unsupervised settings?
How can we calculate the loss function (MSE, MAE, etc.) for the unsupervised setting? Is it common?

using python & sklearn:
from sklearn.metrics import mean_squared_error
#Feature Selection
criterion = mean_squared_error(y, predictions)

or numpy:
import numpy as np
criterion = np.mean((y_test - est.predict(X_test))**2)

Personally, I'm targeting Clustering methods, and its validation is based on similarities as well as isolation forest and move on Autoencoders for DL as recommended here. Of course, I'm aware that Cluster analysis is not something to automate fully, and It is an explorative method. On the other hand, not completely reliable I was always imagining the main reason for distinguishing between supervised and unsupervised is the existence of label features in data. My picture about DL is mostly the high number of hidden layers and leave the responsibility of feature engineering for DL models. I do like this vision concerning comparing unsupervised anomaly models. I believe that we need still to splitting data and calculate loss function despite unlabeled data due to the fact that:

calculating the metrics on the training set would likely lead to overfitting, and then we need the testing set to evaluate the model. Generally speaking, to track the level of overfitting while we are experimenting with our network.
get an estimate of how many epochs we should train for

Additionally, I read this online article Issues with Unsupervised Learning and Why still we need them.
I know the above-mentioned questions look naive, but any help or clarification, even in the short form for this trilogy questions, will be highly appreciated and help me get rid of this confusion and shape my vision correctly.


texts: Recognize and predict a periodic pattern

texts:
I need to design a controller that tries to collect as much falling objects (raindrops) from an unknown source. Beneficial to solving the problem could be the recognition of the seemingly random generation of the raindrops.

y  ^
   | 0 0 1 0 0 0 0
   | 0 0 1 0 0 1 0
   | 1 0 0 0 0 1 0
   | 0 0 0 1 1 0 0
   ----------------->
                  t

So, I have a binary vector signal (raindrop generated yes/no over 4 positions) over discrete time steps. I can only see the raindrops when they are 10 steps away from hitting the ground. The basket has a capacity of one raindrop. The control actions available are step left, step right or do nothing. Any source of pattern recognition could help me design a more advanced controller. Are there methods that can recognize periodic patterns in this type of data?


texts: Can Anyone Explain this code piece by piece?

texts:
Function that creates a DataFrame with a column for Cluster Number

def pd_centers(featuresUsed, centers):
    colNames = list(featuresUsed)
    colNames.append('prediction')

    # Zip with a column called 'prediction' (index)
    Z = [np.append(A, index) for index, A in enumerate(centers)]

    # Convert to pandas data frame for plotting
    P = pd.DataFrame(Z, columns=colNames)
    P['prediction'] = P['prediction'].astype(int)
    return P
```


texts: Detecting timestamp and feature where anomaly occurs with autoencoder

texts: Is it possible according to you to use the latent space of a deep autoencoder to detect the index of the sample and the column/s (features) where an anomaly occurs or am I losing information with compression? I have a multivariate time series dataset


texts: No statistical significance but observable trends

texts:
I have a general inference question regarding scenarios when results from data are not statistically significant but there appears to be an observable trend.

For example, treatment A and treatment B are applied to 2 independent populations.  Using a ttest to analyze the resulting data (lets say the data is total revenue), the p value == .2, so the effect of treatment on revenue was not statistically significant. However, the total revenue from treatment A was observably higher in treatment B. What can I say in this regard?

I've had academic advisors recommend saying 'While the effect of treatment was not significant, a trend was observed', and then one would go on describing the trend. Is this an adequate viewpoint, or a statistical folly? What conclusions would a data scientist in industry draw and present to stakeholders from a scenario like this?


texts: How many outputs for CNN when dealing with a multi-label classification problem with OneHot Encoded labels?

texts:
My labels are of type tensor [1 0 0 1] denoting a 4 label multiclass problem for any given 16x16 image. I'm using BCEWithLogitsLoss from torch.optim and the number of outputs in the final layer is 4.
I am concerned if this is the wrong way to approach the problem and if there should be either 16 outputs (every combination of 0 and 1 in a (4,) shape).
Alternatively, would it be more reasonable to approach this as a multi-task problem with each label being regarded as an independent task?


texts:
I'm writing to get advices about my project.

I want to make recommander system for shop with some products. In fact i want to
recommand to shop A to take item X because shop B sell this item and shops A and B are very similar.

The "problem" here is the size of the data : i have around 5TB of raw data (about 8 000 000 000 lines)
So it's very difficult to do something with huge data like this.

So my questions are :

-It is relevant to use database like MongoDB (or NoQSL for my data) ?

-How can I build utility matrix for recommandation ? (10 000+ shops and 1 milion+ items)

-Do you recommand technologies for that (i've heard about Neo4J graph database, it is relevant to store the relation between shops and items ?)

-The data is maybe too small for Hadoop and i dont have enough computers for the nodes

Thanks


texts: What does this symbol means, what operator is it?

texts: I am confused about the $E_{x\sim P_{data}(x)}$, what does $E$ means here. I cannot find an appropriate answer on the internet, and hence I am trying data science stack exchange. Please help.


texts: Database best-practice when sunsetting a feature

texts:
Say, your product team decided to sunset a feature of your product, and there are columns in your database related to that feature. Is it a good practice to keep the columns (so you don't lose this data) or remove the columns (so you simplify your data model)?
Here is a concrete example in case the question is not clear enough: Say, Facebook decided one day to remove the like button. Obviously they have columns somewhere in their database to record who liked what. Now, is it a good data practice to just drop these columns once the like button is obsolete, or keep it, but then future developers in 5 years will end up dealing with obsolete data?


texts: Help with DDP Mining algorithm for Effective Classification of data sets from 2 groups

texts:
I'm trying to implement the DDPmine algorithm from this article as part of some project, and I do not understand where in the algorithm we use the Class Label of each transaction?
We have transactions from 2 different groups spouse group has a class label "0" and group b has the class label "1" and we want to find the Discriminative Patterns that are frequent in each group but not on the 2 groups combined but in which part of the algorithm we consider this?
what I'm missing here?


texts: Need inputs for Food data set to predict consumer ordering behaviour

texts:
I am looking for food order data set that shows the consumer list and type of food they order regularly. Like :
customer_number, gender, day, time, choice_of_food, food_category, quantity(grams), Calories, Carbs, Fat, protein

Purpose is to predict the choice_of_food any customer can make on a given day.
Please help with your valuable comments from where I can find similar data-set.

Thanks in advance :)


texts: Power Bi breaks when I add Sort By Column on a field

texts:
Here's the issue:
prior to choosing Sort by:

After Choosing Sort By:

Here's the table, and this is also the sort by column I am choosing:

The formulas that are changing to 100% are a few measures as follows:

First data column in the pictures:
Average Inventory =  DIVIDE(  SUM('CY Inventory Data'[Inventory Quantity]),[Inventory Quantity for Truck])

Inventory Quantity for Truck:
Inventory Quantity for Truck =  CALCULATE(SUM('CY Inventory Data'[Inventory Quantity]),ALL(Equipment[ProductModel],Equipment[ProductModel (groups)]))

-- Where Product Model and groups are related to the DC, JH, TM, TT that all go to 100% after applying the sort by.  These are not grouped together, but that group level isn't in the visualizations changing to 100%.
What the ever-loving hell?


texts: Kernel engineering, valid kernels, multipliying by constant =0?

texts:
I am reading Bishop, Pattern Recognition and Machine Learning. In the chapter about kernels rules are given for constructing kernels from existing valid kernels.

The first one being let k(x,y) be a valid kernel and c>0 a constant then

l(x,y)=c*k(x,y) is a valid kernel

I was wondering why does c>0? and not c>=0, since given the Gram matrix of K multiplying each entry by 0 will give us L. L is p.s.d since det(L)>=0, so it should be a valid kernel? (it seems like the Gram Matrix being p.s.d is a sufficient condition for a kernel to be valid) Where does this argument go wrong?


texts: Issue with miscount on test train split in Python

texts:
Problem
Hey everyone. I think I am missing something incredibly easy, but because I am wokring in resampling for the first time it is giving me all sorts of inadequacy. I performed an ADASYN up sample to my majorly imbalanced dataset and the numbers look very good to me while I am doing diagnostics.
It appears to me that my X_res and y_res variables are coming in incorrect into the prompt nad it is reading them at the split as being of length [48501, 48501].
The diagnostic code suggests to me that it should be [48501, 24411] however and I cannot seem to be able to find where the discrepancy is.
I'd love to hear if anyone has any thoughts on this.
Code
ada = ADASYN(random_state=42)
X_res, y_res = ada.fit_resample(X, y)
print('Resampled dataset shape %s' % Counter(y_res))
X_res = X_res.fillna(0)

print('Resampled dataset shape for X %s' % len(X_res), 'and y %s' % len(y_res))

Resampled dataset shape for X 48501 and y 48501

AX_train, AX_test, Ay_train, Ay_test = train_test_split(X_res, y_res, test_size = 0.2, random_state=27, stratify=y)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-74-1d75049a692e> in <module>

----> 1 AX_train, AX_test, Ay_train, Ay_test = train_test_split(X_res, y_res, test_size = 0.2, random_state=27, stratify=y)

...
~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_consistent_length(*arrays)

254     uniques = np.unique(lengths)
255     if len(uniques) > 1:
--> 256         raise ValueError("Found input variables with inconsistent numbers of"
257                          " samples: %r" % [int(l) for l in lengths])
258

ValueError: Found input variables with inconsistent numbers of samples: [48501, 24411]


texts: how classification scores are interpreted?

texts:
I would like to know how to interpret  classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are labeled as 1, and -1 for negative ones. Now, is it fair to say that for a score 10 the instance is more likely to be successfully predicted than a score 5, despite the result that can be wrong.
Thanks.


texts: How to use a Neural Network to classify a very large number/bit string?

texts:
How would I set up a NN to classify a cryptographic key for asymmetric cryptography, or generally a VERY large number/bit string?

I could split this bit string into bytes/16bit/32bit/64bit numbers to use as NN inputs, but that does not seem like a good idea, since I am not looking for relationships and patterns between the bytes or blocks. I am looking for structure in the whole key.

In this exact case I want to classify an almost 20000-bit string into 2 classes.

What would be the correct approach?


texts: Gradients vanishing despite using Kaiming initialization

texts:
I was implementing a conv block in pytorch with activation function(prelu). I used Kaiming initilization to initialize all my weights and set all the bias to zero. However as I tested these blocks (by stacking 100 such conv and activation blocks on top of each other), I noticed that the output I am getting values of the order of $10^{-10}$. Is this normal, considering I am stacking upto 100 layers. Adding a small bias to each layer fixes the problem. But in Kaiming initialization the biases are supposed to be zero.
Here is the conv block code
from collections import Iterable
def same_padding(kernel_size):
    """
    Padding value so input feature map retains its shape after conv operation
    """
    return (kernel_size-1)//2

def convBlock(input_channels,output_channels,kernel_size=3,padding=None,activation="prelu"):
    """
    Initializes a conv block using Kaiming Initialization
    """
    padding_par=0
    if padding=="same":
        padding_par=same_padding(kernel_size)
    conv=nn.Conv2d(input_channels,output_channels,kernel_size,padding=padding_par)
    relu_negative_slope=0.25
    act=None
    if activation=="prelu" or activation=="leaky_relu":
        nn.init.kaiming_normal_(conv.weight,a=relu_negative_slope,mode="fan_in")
        if activation=="prelu":
            act=nn.PReLU(init=relu_negative_slope)
        else:
            act=nn.LeakyReLU(negative_slope=relu_negative_slope)
    if activation=="relu":
        nn.init.kaiming_normal_(conv.weight,nonlinearity="relu")
        act=nn.ReLU()
    nn.init.constant_(conv.bias.data,0)
    block=nn.Sequential(conv,act)
    return block

def flatten(lis):
    for item in lis:
        if isinstance(item, Iterable) and not isinstance(item, str):
            for x in flatten(item):
                yield x
        else:
            yield item

def Sequential(args):
    flattened_args=list(flatten(args))
    return nn.Sequential(*flattened_args)

This is the test Code
ls=[]
for i in range(100):
    ls.append(convBlock(3,3,3,"same"))
model=Sequential(ls)

test=np.ones((1,3,5,5))
model(torch.Tensor(test))

And the output I am getting is
tensor([[[[-1.7771e-10, -3.5088e-10,  5.9369e-09,  4.2668e-09,  9.8803e-10],
          [ 1.8657e-09, -4.0271e-10,  3.1189e-09,  1.5117e-09,  6.6546e-09],
          [ 2.4237e-09, -6.2249e-10, -5.7327e-10,  4.2867e-09,  6.0034e-09],
          [-1.8757e-10,  5.5446e-09,  1.7641e-09,  5.7018e-09,  6.4347e-09],
          [ 1.2352e-09, -3.4732e-10,  4.1553e-10, -1.2996e-09,  3.8971e-09]],

         [[ 2.6607e-09,  1.7756e-09, -1.0923e-09, -1.4272e-09, -1.1840e-09],
          [ 2.0668e-10, -1.8130e-09, -2.3864e-09, -1.7061e-09, -1.7147e-10],
          [-6.7161e-10, -1.3440e-09, -6.3196e-10, -8.7677e-10, -1.4851e-09],
          [ 3.1475e-09, -1.6574e-09, -3.4180e-09, -3.5224e-09, -2.6642e-09],
          [-1.9703e-09, -3.2277e-09, -2.4733e-09, -2.3707e-09, -8.7598e-10]],

         [[ 3.5573e-09,  7.8113e-09,  6.8232e-09,  1.2285e-09, -9.3973e-10],
          [ 6.6368e-09,  8.2877e-09,  9.2108e-10,  9.7531e-10,  7.0011e-10],
          [ 6.6954e-09,  9.1019e-09,  1.5128e-08,  3.3151e-09,  2.1899e-10],
          [ 1.2152e-08,  7.7002e-09,  1.6406e-08,  1.4948e-08, -6.0882e-10],
          [ 6.9930e-09,  7.3222e-09, -7.4308e-10,  5.2505e-09,  3.4365e-09]]]],
       grad_fn=<PreluBackward>)
```


texts: Which string distance equation for fuzzy-matching person names is reliable?

texts:
A reproducible example with a small bit of R code is available in this stackoverflow post (link so I dont need to re-type out the code). The fuzzytext library in R has the following available string methods c("osa", "lv", "dl", "hamming", "lcs", "qgram", "cosine", "jaccard", "jw", "soundex"). Our use case is matching (left-joining) basketball player names from 2 different sources. From the stackoverflow post, we have the following concerns to account for when string matching names:

The left join shouldn't get mixed up by long / short names. Michael Gadson is clearly Mike Gadson, not one of the other Mike names in the dataset with a different last name.
The left join shouldn't get mixed up by reversed names. Ricky Smith is Rick Smith, he is not Smith Rickie.
The left join shouldn't get mixed up by III, Jr., etc. suffix to names, or by extra spaces or symbols: eg. De Andre' vs DeAndre)
Certain players (e.g. Johnny Williams) in the left-hand-side dataframe have no match in the right-side table. To catch this, we'll need to rely on a properly selected max_dist value.

A 5th concern is avoiding duplicates in the code (we want only 1 row for each person in the left-hand-side dataframe), however this is handled with the groupby(fullName) %>% filter(dist == min(dist) | is.na(dist)) in the code.
Our question is then: given these concerns, what is a good method and max distance to use for this left join?


texts:
I'm unsure of how to ask a question without making it seem like a code review question. At what point does one question whether they've actually implemented the algorithm and-or model correctly? Getting spot-on results is great and all, but seems highly suspect. Also, what checks can be done to ensure that the algorithm and-or model is being implemented correctly? The reason I'm asking is because I'm getting perfect classification and subsequently accuracy, precision, etc. w/ the implementation of SVM.

I am including the code, but feel free to ignore.

# Make a copy of the df
iris_df_copy = iris_df.copy()

# Create a new column, labeled 'T/F', whose value will be based on the value in the 'Class' column. If the value in the
# 'Class' column is 'Iris-setosa', then set the value of the 'T/F' column to 1. If the value in the 'Class' column is
# not 'Iris-setosa', then set the value of the 'T/F' column to 0.
iris_df_copy.loc[iris_df_copy.Class == 'Iris-setosa', 'T/F'] = 1
iris_df_copy.loc[iris_df_copy.Class != 'Iris-setosa', 'T/F'] = 0

X_svm = np.array(iris_df_copy[['Sepal_Length', 'Sepal_Width', 'Petal_Length', 'Petal_Width']])
y_svm = np.ravel(iris_df_copy[['T/F']])

# Split the samples into two subsets, use one for training and the other for testing
X_train_svm, X_test_svm, y_train_svm, y_test_svm = train_test_split(X_svm, y_svm, test_size=0.25, random_state=4)

# Instantiate the learning model - Linear SVM
linear_svm = svm.SVC(kernel='linear')

# Fit the model - Linear SVM
linear_svm.fit(X_train_svm, y_train_svm)

# Predict the response - Linear SVM
linear_svm_pred = linear_svm.predict(X_test_svm)

# Confusion matrix and quantitative metrics - Linear SVM
print("The confusion matrix is: " + np.str(confusion_matrix(y_test_svm, linear_svm_pred)))
print("The accuracy score is: " + np.str(accuracy_score(y_test_svm, linear_svm_pred)))
print("The precision is: " + np.str(precision_score(y_test_svm, linear_svm_pred, average="macro")))
print("The recall is: " + np.str(recall_score(y_test_svm, linear_svm_pred, average="macro")))


texts: GTX 1660 Ti vs. RTX 2060 for a deep learning pc

texts:
I'm planning to build a PC for Deep Learning, after the launch of AMD Ryzen 3rd gen processors. My budget will allow me to choose between a GTX 1660 Ti and RTX 2060. I'm tabulating the specs of both the cards below.

╔════════════════╦════════════╦══════════╗
║      GPU       ║ GTX-1660Ti ║ RTX-2060 ║
╠════════════════╬════════════╬══════════╣
║ GPU-cores      ║ 1536       ║ 1920     ║
║ Tensor-cores   ║ N/A        ║ 240      ║
║ RT-cores       ║ N/A        ║ 30       ║
║ Texture-units  ║ 120        ║ 120      ║
║ Render-outputs ║ 48         ║ 48       ║
║ Base-clock     ║ 1500       ║ 1365     ║
║ Boost-clock    ║ 1770       ║ 1680     ║
║ Mem-bus-width  ║ 192-bit    ║ 192-bit  ║
║ Mem-bandwidth  ║ 288GB/s    ║ 336GB/s  ║
║ Mem-capacity   ║ 6GB        ║ 6GB      ║
║ TDP            ║ 120W       ║ 160W     ║
╚════════════════╩════════════╩══════════╝

Given this comparison, I'd like to know which one would be a better choice for me. Would the RTX-2060 be worth the extra 70$ for the supposedly better performance? I had an initial look at the usefulness of tensor cores in Deep Learning applications, and found out the following article snippet from Nvidia website.

  Note About Tensor Cores

  Designed specifically for deep learning, Tensor Cores on newer GPUs
  such as Tesla V100 and Titan V, deliver significantly higher training
  and inference performance compared to full precision (FP32) training.
  Each Tensor Core provides matrix multiply in half precision (FP16),
  and accumulating results in full precision (FP32). This key capability
  enables Volta to deliver 3X performance speedups in training and
  inference over the previous generation. All samples are optimized to
  take advantage of Tensor Cores and have been tested for accuracy and
  convergence.


texts:
What is the difference between Gradient Descent and Stochastic Gradient Descent?

I am not very familiar with these, can you describe the difference with a short example?


texts: How to detect outliers documents using python Machine Learning

texts:
I'm currently working on a Machine Learning project. The point of this project is to train a model with a pandas.Dataframe composed with 3 columns:

"Libellé pays" --> that won't be used for now.
"Marchandise"  --> current strings that I want to use.
"label"        --> corresponding labels

The thing is, I have only one "class" of document (labelled 1). I would like
to have at the end a model that will be able to determine if a document could be identified as similar to the trained datasets.
After some research, I was able to tell that outliers detection could be useful, but I can't find documentation that matches my current problem.

Could you help me with that?

Edit : adding code and data.

def class MLModel:

    def __init__(self, logger):
    """MLModel constructor."""

    self.logger = logger
    self.xml_dir = "./DASCP-DASCV_2018_MRS-FOS.xls"
    self.dataframe = None
    self.classifier = None

def get_traindata_from_xls(self):
    """fetch training data from excell file. filepath is in init"""

    xl = pd.ExcelFile(self.xml_dir)
    self.dataframe = xl.parse('Manifest')

and this is what self.dataframe looks like: https://pastebin.com/faduQ6gU

Libellé Pays    object
Marchandise     object
label            int64
dtype: object


texts: Product classification in e-commerce using attribute keywords

texts:
I am working on a product classification problem (E-Commerce) in which I have to identify product category based on keywords.

Say for example, if input is given as 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable Upto 256 GB 5.5 inch Full HD Display' , It should correctly identify it as 'Mobile'.

The problem here is that I have certain key-value pairs on which I have to train the model. Previously, I was doing the same problem by training model on product level using LinearSVC and it was giving satisfactory results.

Details about data:

I have 39 classes/ categories currently, which might increase in future.
I am using a csv file which is of around 10 MB and it has around 7000 rows.(Refer below structure)
I am using LinearSVC from sklearn.svm

I have the following structure for training now:

--------------------------------------------------------------
| Attribute_Names   |  Attribute_Values        |   Category  |
--------------------------------------------------------------
| Brand             |  Samsung, Nokia, OnePlus | Mobile      |
| RAM, Memory       | 2 GB, 4 GB, 3 GB, 6GB    | Mobile      |
| Color,Colour      | Black, Golden, White     | Mobile      |
--------------------------------------------------------------
| Fabric, Material  | Cloth, Synthetic, Silk   | Ethnic Wear |
| Pattern, Design   | Digital, floral print    | Ethnic Wear |
--------------------------------------------------------------

I have the following queries:

1 - Which Model to use for this problem ?

2 - How would you handle such problem ?

3 - Any similar problem you have seen.

Any help is much appreciated.

EDIT:

step 1) - merge column(attribute_names, attribute_values)

step 2) - Cleaned the data(removing stopwords,special characters, steming)

step 3) - Feature extraction using TfidfVectorizer(stop_words='english',lowecase=True)

step 4) - OneVsRestClassifier(LinearSVC(loss='hinge',penalty='l2'))

This are the steps we have performed till now,and the results are not good enough (miss-classification still exist)

format/structure of data is same as we mentioned above.


texts: Tensor Flow Time Series Tutorial Question

texts:
In the tutorial, they normalize the data and say "The mean and standard deviation should only be computed using the training data"

What does this refer to? Why should you only use the training data?


texts: Pandas: Assign back to table from grouping by column and index

texts:
I am trying to implement Exponential Moving Average calculation on a DataFrame. The formula is

An additional complication is that my table is grouped and there is a unique bin number per group. This is what I tried

import numpy as np
import numpy.random as rand

n = 5
groups = np.array(['one', 'two', 'three'])
data = pd.DataFrame({
    'price': rand.random(3 * n) * 10,
    'group': np.repeat(groups, n),
    'bin': np.tile(np.arange(n),3)}, index=np.arange(3 * n))

print(data)

       price  group  bin
0   1.601310    one    0
1   3.190662    one    1
2   4.419421    one    2
3   3.817510    one    3
4   2.440774    one    4
5   6.832265    two    0
6   5.636502    two    1
7   4.630515    two    2
8   5.856423    two    3
9   0.916452    two    4
10  4.247134  three    0
11  7.146746  three    1
12  8.049161  three    2
13  7.852168  three    3
14  0.246720  three    4

This is how I am trying to implement the EMA calculation;

data['EMA'] = np.zeros(len(data.index))
data.loc[data['bin'] == 0, 'EMA'] = data.loc[data['bin'] == 0, 'price']

a = 2 / (n + 1)
for _, group in data.groupby('group'):
    for index, row in group.iloc[1:].iterrows():
        prev = group[group['bin'] == row['bin'] - 1].iloc[0]
        row['EMA'] = a * row['price'] + (1 - a) * prev['EMA'] # nope
        data.loc[index, 'EMA'] = a * row['price'] + (1 - a) * prev['EMA'] # nope

Unfortunately neither of these last lines update the values in the group. On the second iteration, the value of prev['EMA'] is still 0. What is the correct way to assign the values back to the table such that it is updated dynamically? Do I need to write out to a temporary array and write it back afterwards?

Additionally, I could not think of an elegant way to do this using assign or transform. If someone can solve that it might be a very good alternative.

Response

Thank you @DaFanat for your help. Unfortunately your code doesn't work. I tried the following

data.loc[:, 'EMA2'] = map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1))

But I get an error TypeError: object of type 'map' has no len(). I tried changing it to this

data['EMA2'] = list(map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1)))

And I do indeed get some results, but they do not look correct;

       price  group  bin       EMA      EMA2
0   5.407722    one    0  5.407722  5.407722
1   0.495734    one    1  3.770393  3.770393
2   7.911491    one    2  5.150759  2.967653
3   1.085836    one    3  3.795785  5.636272
4   7.326432    one    4  4.972667  3.166035

I am inclined to believe my implementation, since the price goes up from 0.5 to 7.9, how can the moving average go down? I think the indices are getting lost and it is putting the values against the wrong cells. How do I retain the indices when performing this calculation?

Solution

Thanks @DaFanat, you took me a long way. I finally figured out how to do it with a slight modification of your original method;

data['EMA2'] = data.groupby('group') \
                   .apply(lambda x: x['price'] * a + x['EMA'].shift(1) * (1-a)) \
                   .reset_index(level=0, drop=True)
data.loc[data['bin'] == 0, 'EMA2'] = data.loc[data['bin'] == 0, 'price']

print(data)

       price  group  bin       EMA      EMA2
0   3.556171    one    0  3.556171  3.556171
1   5.637241    one    1  4.249861  4.249861
2   3.278771    one    2  3.926164  3.926164
3   7.343718    one    3  5.065349  5.065349
4   6.128884    one    4  5.419861  5.419861

Not using the list(map()) ensures that the result is a DataFrame with indices retained, so it knows where to insert the individual rows.


texts: Synthetic Gradients - what's the practical benefit?

texts:
I can see two motives to use Synthetic Gradients in RNN:

To speed up training, by imediately correcting each layer with predicted gradient
To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.
I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Point 1:
Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.
However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.
Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer
shouldn't improve training speed.

Point 2:
Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.
This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A
----> stop and bkprop!
fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B
----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)".
Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.
But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last
timestep of fwdprop allows to effectively train longer sequences, but
vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.
Question:
Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial


texts: Cosine similarity of averaged random word vectors

texts:
I am trying to find the cosine similarity (using glove vector) of two random words. As expected, the distribution of the similarity concentrated around 0 since it is reasonable to think that two random words will not be similar to each other.

However, when I try to do a similar thing to 2 random sets of 10 words, that is I take the average vector of the 10 words in both sets and calculate the cosine similarity, the similarity tends to concentrate at 0.8.

It seems to suggest that given 2 random sentences of 10 words, they are very likely to be similar semantically. What could be the explanation of this?

Included python code to reproduce the result.

import spacy
nlp = spacy.load('en')

vocab = nlp.vocab
words = np.array([x.orth_.encode('utf8') for x in vocab])

hist1 = []
n = 1000
num_words = 1
for _ in range(n):
    x,y = choice(words, size=(2,num_words))
    x = nlp(" ".join([u.decode('utf8') for u in x]))
    y = nlp(" ".join([u.decode('utf8') for u in y]))
    s = x.similarity(y)
    hist1.append(s)

hist10 = []
n = 1000
num_words = 10
for _ in range(n):
    x,y = choice(words, size=(2,num_words))
    x = nlp(" ".join([u.decode('utf8') for u in x]))
    y = nlp(" ".join([u.decode('utf8') for u in y]))
    s = x.similarity(y,)
    hist10.append(s)

plt.hist([hist1,hist10], label=[1,10])
plt.legend()


texts: Beginner: how to model net change given an incomplete set of input/output figures

texts:
this is my first post on the Stack Exchange site, and I'm looking for some guidance.

I have a set of 10x input numbers (positive), and a set of 10x output numbers (negative), as well as the real net change.

In a perfect world, adding those twenty numbers up would be equal to the 'real net change'. Unfortunately, the set in incomplete, and every week there is an error. If it's of any help, the error is always in the same direction (modelled change too big) - it appears that the set of output numbers is incomplete. Sometimes though, the error is non-existent.

What approach would you recommend to model the real change?

I used Weka for regression analysis, but it appears to me that weka is much more powerful when used with categorising problems rather than continuous data.

I appreciate all comments and apologise if the problem stated is too general.


texts: Tools for reading data from large, irregular csv files (aka excel file hell)

texts:
I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does number of rows). There are also identifying parameters in the cells around each dataset, although exactly where they are relative to a dataset varies.

I want to get this messy excel file into R or python. How should I do it?

(I know I should buckle down and copy and paste the datasets into a standard format, but that seems tedious and error prone. So does writing an ad hoc function to do it. Do I have any other options??)


texts:
I have a stream of emotions (from some audio recordings) extracted by a speech emotion recogniser.  My questions now are how to best display these emotions to the end users?

What is the best way to visualise this stream of emotions?
Is there any tool/library/package/software that I can use a stream of ekman emotions (e.g. happy->surprise->sad->...) as input and get some visualisation as output?

Thanks !!


texts: How can I determine how many signals there are in a mixture?

texts:
Let's say that I have 9 sensors arranged in a 3 by 3 grid. I have multiple objects which emit the same signal and move past the grid of sensors, which are picking up the signals. I have a CSV file which contains readings from the 9 sensors collected every 0.01s for up to a minute and I don't know how many objects are in there. How can I find out how many there were?
I think each of the 9 sensors will basically collect very similar readings to each other, but at different time-points (since an object moves past each sensor at a different time). Any ideas on a general approach to these sorts of problems would be welcome too!


texts: Invariance Property of Vowpal Wabbit Updates - Explaination

texts:
One of the discussed nice aspects of the procedure that Vowpal Wabbit uses for updates to sgd
pdf is so-called weight invariance, described in the linked as:

"Among these updates we mainly focus on a novel
set of updates that satisfies an additional invariance
property: for all importance weights of h, the update
is equivalent to two updates with importance weight
h/2. We call these updates importance invariant."

What does this mean and why is it useful?


texts: Reason for Huge Jump in Loss For One Epoch Only?

texts:
I was wondering what possible reasons there could be for a huge jump in loss for only one epoch during training.  I am getting a result like...

Epoch 1/10
2020-05-13 18:42:19.436235: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 30910 of 40000
2020-05-13 18:42:22.360274: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
200/200 [==============================] - 173s 863ms/step - loss: 0.1844 - val_loss: 0.4250
Epoch 2/10
200/200 [==============================] - 167s 833ms/step - loss: 80890.9766 - val_loss: 0.5157
Epoch 3/10
200/200 [==============================] - 166s 830ms/step - loss: 0.0549 - val_loss: 0.2966
Epoch 4/10
200/200 [==============================] - 170s 849ms/step - loss: 0.0488 - val_loss: 0.2708

Which strikes me as very odd.  This is a 3-lay LSTM network using Keras.

I graphed the data after normalization, but nothing out of the ordinary appears.

Here is the code I used to create the network...

# create new more complicated network
multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                      return_sequences=True,
                                       input_shape=
[past_history,len(features_considered)]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
# output layers are same as prediction count
multi_step_model.add(tf.keras.layers.Dense(future_target))
multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

# run training
multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
                                      steps_per_epoch=EVALUATION_INTERVAL,
                                      validation_data=val_data_multi,
                                      validation_steps=50)


texts:
Background
Having gone through articles.

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Attention in Neural Networks
Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
How does attention mechanism learn?
A Comprehensive Guide to Attention Mechanism in Deep Learning for Everyone
Lecture 13: Attention

However, still not sure of the basic mechanism of attention. I see encoder, decoder, hidden state and  diagrams, and the matrix out of blue.

Question
As a first step, please confirm if below is a correct understanding.
If asked, "Related with crime, cocaine, police, smuggle but not with honest, legitimate. Who is this?". Drug-dealer can pop up. The correlations from drug-dealer to (crime, cocaine, police, smuggle, ...) can be quantified as a vector. Some people have built such vectors by going through wikipedia article sentences, BBC news articles, etc. A vector possibly would be a weight in DNN.

When a NLP parser finds smuggle in a sentence, those correlation vectors help to find which words are more strongly correlated. Is this the basic mechanism of attention, which is, the correlation vectors in a nutshell?
For image to caption, an object detection extracts (frisbee, girl, hair, cloths, ...) and those vectors tell that girl is the word to pay attention to for frisbee.

For translation, e.g. English to Spanish, I suppose there are two sets of vectors. One for English words and the other for Spanish, and association between a English word to a Spanish word is trained as a DNN model where training input is (english-vector, spanish-vector) and someone has prepared labels. Is this correct?
PS. A kind request not to list math formulas, or similar diagrams from the articles. The formulas should be the projections of solid intuitions that even a 6 grader can get. I appreciate such intuitions explained in layman's term as the answers.


texts: time series prediction using arima and non linear trend and too much residuals

texts:
I am working on forecasting a financial index, i tried decomposing the time series using :

from matplotlib import pyplot
from statsmodels.tsa.seasonal import seasonal_decompose
result = seasonal_decompose(dataset, model='multiplicative', freq=12)
result.plot()
pyplot.show()

And i got the following result:

The results show that the time series is not stationary and it has a unit root (I used ADF and KPSS tests) and that the mean and std are constant in time!

I am wondering if i should use ARIMA or SARIMA since they are adapted to linear trend (my trend is not linear as shown in the image) or move to using LSTM, NN ... ?
Or even ARIMA or SARIMA are not adapted to this type of time series?


texts: sklearn.metrics.average_precision_score getting different answers for same data but different formats

texts:
I was trying to learn how average precision (AP) is calculated and implemented in scikit-learn. I have read the documentation, but I don't think I fully understand it yet.

Consider the following two snippet:

import numpy as np
from sklearn.metrics import average_precision_score
y_true = np.array([0, 1, 0])
y_scores = np.array([0.4, 0.4, 0.8])
average_precision_score(y_true, y_scores) # 0.3333333333333333

and

y_true = np.array([[1,0], [0,1],[1,0]])
y_scores = np.array([[0.6,0.4], [0.6,0.4], [0.2,0.8]])
average_precision_score(y_true, y_scores) # 0.45833333333333326

From what I understand, these are the same data but formatted in different ways. The first is only showing the true labels and predicted scores for the positive class, while the second is giving information on both classes.

But why are they giving different answers? In particular, how are these two results calculated? And which one is correct? I was reading this post, but I didn't understand how that precision-recall table in the answer is constructed. Can anyone can go through a similar calculation for my example?


texts: Did anybody ever use mean pooling and publish it?

texts:
I found a couple of sources which mention mean pooling for convolutional neural networks (CNNs) - including all lectures I had about CNNs so far - but I could not find any paper with at least 10 citations which uses mean pooling.

Do you know of a paper which uses mean pooling?


texts:
I can use shap to extract important features for Dense NN. However, for CNN, I encountered two problems:

the feature order may be messed up or combined after the filtering and pooling. Can I still use shap or other approach to show the important
features for CNN?
I tried to use shap, but the shap_summary_plot shews the bar plot to the left and the plot_size does not help to adjust it.


texts:
I have a dataset with 10 columns and 158 rows. I try to predict my test dataset which is 1 columns with 158 rows.

I made cross-validations, grid-search and use ElasticNet algorithm.

Also before the evaluate the model I check the pearson correlation between 10 columns which I used for train with other 1 column which I try to predict. The correlation is not good but when I evaluate model the R^2 score is near 0.98 .

How can I be ensure that this score is confidental ? Because I didn't expect a R^2 like this. This is too high that I expect.

Thanks in advance.


texts:
I'm studying Machine Learning using Sebastian Raschka's book.

Wonder if someone could please help me to confirm if I have the following steps correct if I apply Perceptron Algorithm to Iris dataset as shown in the table below.
What do I set $\eta$ to?
What do I set $θ$ to? I randomly set it to 2.

Perceptron rule - summarized by the following steps:

Initialize the weights to 0 or small random numbers.
For each training sample $x(i)$ perform the following steps:

Compute the output value $\hat{y}$ .
Update the weights.

\begin{array}{c|c|c|}
 & \text{sepal_length, $X_1$ } & \text{sepal_width, $X_2$} & \text{petal_length, $X_3$} & \text{petal_width, $X_4$} & \text{species, $Y$}\\ \hline
\text{Row 0} & 5 .1 & 3.5 &1.4 &0.2 & setosa \\ \hline
\text{Row 1} & 4.9 & 3.0 & 1.4 & 0.2 & setosa \\\hline
\text{Row 2} & 4.7 & 3.0 & 1.4 & 0.2 & setosa \\\hline
\text{Row 3} & 4.6 & 3.1 & 1.5 & 0.2 & setosa \\\hline
\text{Row 4} & 5.0 & 3.6 & 1.4 & 0.2 & setosa \\\hline
\end{array}

set: $θ = 2$

set: $virginica = -1,$ $\>$ $setosa = 1$

let's say if $z \ge  θ,$ $\>$ $θ(z) = 1$,  $\>$ $θ(z) = -1$ if otherwise

Initialize weight vector: $W$ = [ $w_1$ = 0.1, $w_2$ = 0.2, $w_3$ = 0.3, $w_4$ = 0.4]

For observation row 0:

$z = w_1*x_1 + w_2*x_2 + w_3*x_3 + w_4*x_4$
$= (0.1* 5.1) + (0.2*3.5) + (0.3*1.4) + (0.4*0.2)$
$= 1.71$

Since $z = 1.71 \lt θ = 2$, $\>$ then $θ(z) = -1$

As for row 0, the perceptron algorithm incorrectly predict $virginica = -1,$, hence the weights would be adjusted.

$w_j:= w_j + \Delta w_j$

$\Delta w_j = \eta(y^{(i)} - \hat{y}^{(i)}) x_j^{(i)}$

Hence,

$\Delta w_1 = \eta(1-(-1))5.1 = 10.2\eta$

$\Delta w_2 = \eta(1-(-1))3.5 = 7.0\eta$

$\Delta w_3 = \eta(1-(-1))1.4 = 2.8\eta$

$\Delta w_3 = \eta(1-(-1))0.2= 0.4\eta$

Update weight vector: $W$ = [ $w_1 = 0.1 + 10.2\eta$, $w_2 = 0.2 + 7.0\eta$, $w_3 = 0.3 + 2.8\eta$, $w_4 = 0.4 +0.4\eta$]

Using the updated weight to Row 1 and repeat the process above.


texts: What metric does `Scalar test loss` in AutoKeras mean?

texts:
I'm using the following code:

import autokeras as ak

regressor = ak.StructuredDataRegressor(max_trials=100)
regressor.fit(x=train_dataset.drop(columns=['MPG']), y=train_dataset['MPG'])
# Evaluate the accuracy of the found model.
print('Accuracy: {accuracy}'.format(
    accuracy=regressor.evaluate(x=test_dataset.drop(columns=['MPG']), y=test_dataset['MPG'])))

The result of the evaluate function is Scalar test loss. I just want to clarify what metric would that be. What is the default loss function?


texts: How to calculate the gradient for nce_loss in tensorflow

texts:
I need to calculate the gradient of a tensorflow that is stored. I can restore the graph and weights using:

    model1 = tf.train.import_meta_graph("models/model.meta")
    model1.restore(sess, tf.train.latest_checkpoint("models/"))
    sess.run(tf.global_variables_initializer())
    graph = tf.get_default_graph()

    weights = graph.get_tensor_by_name("weights:0")
    biases = graph.get_tensor_by_name("biases:0")

I have also named my loss function in the original function so I can restore it with

    loss = graph.get_operation_by_name("loss") # for operation
    loss = graph.get_tensor_by_name("loss:0") # for the tensor

Basically, I want to get the gradient of the loss with a certain input value using tf.gradients(...). My loss is specifically the nce_loss https://www.tensorflow.org/api_docs/python/tf/nn/nce_loss. I want the gradient of the loss given the inputs function. Specifically, I plug in a new embedding and I want the gradient given that new input and the loss function. However I can't seem to define my input successfully. If I use:

grads = tf.gradients(loss, loss.inputs) #here I use the tensor loss definition

I get:

Traceback (most recent call last):
  File "/tmp/fgsm.py", line 168, in <module>
    main(config)
  File "/tmp/fgsm.py", line 114, in main
    loss = graph.get_operation_by_name("loss:0")
  File "/tmp/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3618, in get_operation_by_name
    return self.as_graph_element(name, allow_tensor=False, allow_operation=True)
  File "/tmp/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3490, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/tmp/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3544, in _as_graph_element_locked
    (repr(name), types_str))
ValueError: Name 'loss:0' appears to refer to a Tensor, not a Operation.

How do I define my gradient here?


texts: The Area Under an ROC Curve (AUC) vs Confusion Matrix for classifier evaluation?

texts: When should I use The Area Under an ROC Curve (AUC) or the Confusion Matrix for classifier evaluation? The clasifier evaluation is for example the prediction of customers for possible future sales.


texts:
I'm performing some simulations, and at the end I get a CSV file with three columns. One column holds the values for the x-axis, which was also input to the simulation and theoretical calculations, second one holds theoretically expected values, and the other column holds the values obtained by the simulation. I was planning to plot something like this:

But that does not look good in my case, as the values in y-axis normally keep doubling, and the values for the x-axis exponentially increase, so most of the points end up getting collected at the lower left part, near the intersection of x-axis and y-axis of the plot. Therefore, I need a different way to plot my data, which will be more visually appealing and inform how close the simulation results are to the theoretical expected ones. For example, some of my values can be seen below (and they keep increasing in such a way):

x         = [2, 4, 8, 16, 32, 64] # partially removed for brevity
expected  = [47.9995, 95.9783, 191.9127, 383.9708, 767.8831] # partially removed for brevity
simulated = [48, 96, 191.8, 383.8, 767.4] # partially removed for brevity

What is a good way to plot such a data that doubles in the y-axis and exponentially increases on the x-axis all the time, and to view how similar the two datasets actually are?


texts: I am trying to scrape some data from a website with very little success. Basically there is a route overlaid on google maps and whenever you mouse over specific sections of the map (about 200 in all) it fetches 7 fields from a database and displays them on screen. Doing a single map manually would take about 30 minutes and be quite imprecise. There are about 10,000 map routes I want to scrape so this is not realistic to do it manually. Is there a relatively straightforward way of automating this process?


texts: How to understand incremental stochastic gradient algorithm and its implementation in logistic regression [updated]?

texts:
I have a difficulty where to start the implementation of incremental stochastic gradient descent algorithm and its respective implementation in logistic regression. I don't quite understand this algorithm and there are few sources to explain it with crystal-clear interpretation and possible demo code. I am quite new with ML algorithm and I have no idea which would be efficient workaround to solve this problem.

In particular, the problem I am studying is to implement hogWild! algorithm for logistic regression, which asks me to program incremental SGD algorithm with a sequential order. Can anyone give me a general idea or possible pipeline to make this happen in python?

logistic loss function and gradient

Here is my implementation:

import numpy as np
import scipy as sp
import sklearn as sl
from scipy import special as ss
from  sklearn import datasets

X_train, y_train=datasets.load_svmlight_file('/path/to/train_dataset')
X_test,y_test=datasets.load_svmlight_file('/path/to/train_dataset.txt',
                                          n_features=X_train.shape[1])

class ISGD:
def lossFunc(X,y,w):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    lossFnc=ss.log1p(1+np.nan_to_num(ss.expm1(-y* np.dot(X,w,))))
    rslt=np.float(lossFnc)
    return rslt

def gradFnc(X,y,w):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    gradF1=-y*np.nan_to_num(ss.expm1(-y))
    gradF2=gradF1/(1+np.nan_to_num(ss.expm1(-y*np.dot(X,w))))
    gradF3=gradF2.resize(gradF2.shape[0],)
    return gradF3

def _init_(self, learnRate=0.0001, num_iter=100, verbose=False):
    self.w=None
    self.learnRate=learnRate
    self.verbose=verbose
    self.num_iter=num_iter

def fitt(self, X,y):
    n,d=X.shape
    self.w=np.zeros(shape=(d,))

    for i in range(self.num_iter):
        print ("\n:", "Iteration:", i)

        grd=gradFnc(self.w, X,y)
        grd.resize((grd.shape[0],1))
        self.w=self.w-grd
        print "Loss:", lossFunc(self.w,X,y)

    return self

Seems my above implementation has some problems. Can anyone help me how to correct that? Plus, I don't have a solid idea how to implement incremental SGD sequentially. How can I make this happen? Any idea?


texts: What probability distribution would be more appropriate for monthly rate of going to the store?

texts: Part of a model I am making includes the frequency with which people go shopping for a given good (e.g. people on average go to the supermarket some n times a month).  I am trying to figure out what probability distribution is most appropriate for the task.  I think that this can be modeled by a poisson process, because it involves a rate across time, but am unsure if occurrences of shopping are independent of each other in this case. I considered a normal distribution with the mean centered around some rate but am also unsure about that.


texts: How to solve the gradient descent on a linear classification problem?

texts:
I have a problem which i have attached as an image.

Problem is in image attached

what I understand

error function is given by: $e(y, \hat y)=0$ if $y \cdot a(x-b) \ge 1$ or $e(y, \hat y) = 1-y\cdot a\cdot (x-b)$ if $y a(x-b) < 1$.

Gradient descent at current $t$ is (1,3).

Gradient Descent($E_{in}(a,b)$) as per definition should be equal to partial derivative of the equation of $E_{in}(a,b)$ wrt $a$ and $b$. ($w$ is equivalent to $[a, b]$, according to me)

My doubt

Please note that when i say sum over N points, i mean to use that greek symbol used for summing of N points

I am not sure how would I calculate the partial derivative in this case. When we say to fix '$a$' and vary '$b$', does it mean to find differentiation only wrt '$b$'? Which would mean that gradient($E_{in}$)= $-1 / N (\sum_{i=1}^N y_i a)$. But this removes dependence on $x$ and that's why I doubt my approach.
The final equation for whom derivative needs to be done is:
$1/N \sum_{i=1}^N 1-ya(x-b)$ for N points misclassified. But as per the dataset, no point is misclassified as the error function for each point with the (a,b)=(1,3) is equal to 0.

For point 1, where x=1.2 and y = -1, as $y \cdot a \cdot (x-b) =
(-1)\cdot (1)\cdot (1.2-3)=+1.8$. This means that $e(y_1, h(x_1)) = 0$.

So what should be the answer to this question?

I hope my doubt in the question is cleare to audience. But in any case please let me know if there is something that I am unable to explain.


texts: The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

texts:
First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type of activators only at the first layers of the CNN and afterwards a simple RELU follows at the end. Regarding this, how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

Secondly, as far as I'm concerned because of the vanishing gradient problem the neurons at the beginning tend to fall into zero more often than those at the top of the network and then it is very difficult (or even impossible) to activate again; Wouldn't be correct to allow the negative gradient at the whole pipeline of the Neural Network?