texts: Variable TimeStep in Spatio Temporal 3DConvNet

texts:
I'm working on a new project on climate data. I suppose my output $Y(H*W*1)$ is a consequence of a given data scenario $X(C=4*T*H*W)$ and an initial state $Y0(H*W*1)$. I've chosen a temporal convolutional network

(more or less this one)

For X spatio-temporal feature extraction. The fact is I have data with different timesteps (for example 1 dataset is 500 years long and as a 10 years timestep, and another is 100 years long ans has a 1 year timestep).

I wonder if someone has an idea of how I can include length and timestep as inputs of my model in order my model will be able to deal with different length and timesteps?

Thank you very much for your help,
Marin


texts: Xgboost rank:ndcg learning per group or for all dataset

texts:
I'm trying to implement xgboost with an objective of rank:ndcg

I want the target to be between 0-3.

In my data for most of the groups, there is only 1 event per group which his target is not 0.

I wonder if the model will learn different about this product when I put target value 3 (and not 1) when this is the only target that larger than 0 from products on a different group which got target value 1. or it is learning per group id and if the in the group there is only one target larger than 0, it doesn't care if it is 3 or 1


texts: What are the use-cases for heapsort option in numpy.sort()?

texts:
Could anybody share experience about use cases of the heapsort option in numpy.sort() (or pandas.DataFrame.sort_values()) in the data science field?
I mean if the quicksort option actually means introsort (hybrid algorithm) which includes heapsort, what is the purpose for the separated heapsort option?
Or is it just a kind of legacy for backward compatibility?


texts: Measuring chance ("risk") of being in some class

texts:
I don't know if this question fits better here or in statistics, but I think here is more appropriate.

I have a dataset with several companies and its features and also I have the information if they "broke" in a near future or not. I want build a model that could measure the probability of a company to break in a near future.

At first glance this seems a classification problem, yet I do not want to have a binary answer, but a number that measures the "risk" of this company to break.

Can anyone suggest me a bibliography, or maybe suggest a (class of) model(s) on this kind of problem? Is survival analysis a good way to approach this problem?

The features I have are not in a time series, though it is obtained from temporal date. There are features like chargeback rate, maturity of chargeback, number of transactions the company did in the past 3 months and its mean value and so on.


texts: Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it without Hadoop"?


texts: Pandas merge column duplicate and sum value

texts:
How to merge duplicate column and sum their value?

What I have

A   30
A   40
B   50

What I need

A   70
B   50

DF for this example

d = {'address': ["A", "A", "B"], 'balances': [30, 40, 50]}
df = pd.DataFrame(data=d)
df


texts: Using word embeddings with additional features

texts:
I have the set of queries for classification task using Gradient Boosting Classifier of scikit learn. I want to enrich the model by feeding additional features along with GloVe. How should I approach scaling in this case? GloVe is already well scaled, however, features are not.
I have tried StandardScaler, but this reduced the performance in comparison with just using GloVe. The problem maybe with the feature itself, however, I need your opinion on scaling starategies in case of glove and dummy variable.


texts: Simple constraints on Logistic Regression

texts:
I am doing with a binary classification problem. I have three features (w1x1 + w2x3 + w3x3 + w4), and I want to get a rule so that there is for sure w1 > 0, w2 > 0, w3 < 0 and any constant. For logistic regression, I use the class (which I found in open sources):
class LogisticRegression:
    def __init__(self, lr=0.01, num_iter=100000, fit_intercept=True, verbose=False):
        self.lr = lr
        self.num_iter = num_iter
        self.fit_intercept = fit_intercept

    def __add_intercept(self, X):
        intercept = np.ones((X.shape[0], 1))
        return np.concatenate((intercept, X), axis=1)

    def __sigmoid(self, z):
        return 1 / (1 + np.exp(-z))
    def __loss(self, h, y):
        return (-y * np.log(h) - (1 - y) * np.log(1 - h)).mean()

    def fit(self, X, y):
        if self.fit_intercept:
            X = self.__add_intercept(X)

        # weights initialization
        self.theta = np.zeros(X.shape[1])

        for i in range(self.num_iter):
            z = np.dot(X, self.theta)
            h = self.__sigmoid(z)
            gradient = np.dot(X.T, (h - y)) / y.size
            self.theta -= self.lr * gradient

            if(self.verbose == True and i % 10000 == 0):
                z = np.dot(X, self.theta)
                h = self.__sigmoid(z)
                print(f'loss: {self.__loss(h, y)} \t')

    def predict_prob(self, X):
        if self.fit_intercept:
            X = self.__add_intercept(X)

        return self.__sigmoid(np.dot(X, self.theta))

    def predict(self, X, threshold):
        return self.predict_prob(X) >= threshold

How do I get logistic regression to find the optimal solution using my constraints? (or any other restrictions like w2 < w3).


texts: How exactly does adding a new unit work in Cascade Correlation?

texts:
I've just read The Cascade-Correlation Learning Architecture by Scott E. Fahlman and Christian Lebiere.

I think I've got the overall concept (or at least the "cascade" part - a 4min YouTube video how I think it works):

Start with a minimal network with input and output units only
Learn those weights with standard algorithms (e.g. gradient descent - they seem to use another training objective which I don't quite understand, so it is gradient ascent in the paper)
When the network doesn't improve, add a single new hidden unit. This unit gets input from all input nodes and all hidden nodes which were added before. Its output goes to all output nodes only.
Repeat step 3

However, I don't understand the details of step 3: The input weights to hidden units are frozen (indicated by boxes in the paper). When exactly do they get frozen? Are they just initialized by random and never learned at all?

I also don't understand this paragraph:

  To create a new hidden unit, we begin with a candidate unit that receives
  trainable input connections from all of the network's external inputs and from
  all pre-existing hidden units. The output of this candidate unit is not yet
  connected to the active network. We run a number of passes over the examples of
  the training set, adjusting the candidate unit's input weights after each pass.
  The goal of this adjustment is to maximize $S$, the sum over all output units $o$
  of the magnitude of the correlation (or, more precisely, the covariance)
  between $V$, the candidate unit's value, and Eo, the residual output error
  observed at unit o. We define S as

  $$S = \sum_{o} | \sum_p (V_p - \bar V) (E_{p,o} - \bar{E_o}) |$$

  where $o$ is the network output at which the error is measured and p is the
  training pattern. The quantities $\bar V$ and $\bar{E_o}$ are the values of
  $V$ and $E_o$ averaged over all patterns.

What is an "residual output error"? Is $V_p$ simply the activation of the unit given the pattern $p$? What does the term $S$ mean and why do we want to maximize it?


texts: Why leaky relu is not so common in real practice?

texts:
As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have.

Yet Leaky relu is less popular than Relu in real practice. Can someone tell why?


texts: How can i find trend time for my articles?

texts:
our article is time-based, that means is my article search more in a specific time.
as you can see in under chart this article search more in specific period time.

if my dataset looks like this(it can be a json,csv,xlsx) :

How can I find the trend time of my article (time between the red line in image 1)

I need to know that period of time to be ready for next year.

  (I need excel solution for that, but if you can solve this problem
  with other ways and programming language it's ok)


texts:
Is Siamese network rotation invariant which means if I train my siamese network on the different rotated versions of the same image so will it treat each image as different image or same.

Also if I pass for example a image of cat and a rotated image of same cat as a pair to siamese network will it give a similarity score of 0 or 1.


texts: How multi-head attention mechanism in Transformers is able to attend to information from different representation subspaces at different positions

texts:
Question partially inspired by this post about the need of multi-head attention mechanism.
For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(so that they potentially can - as stated in the Attention is all you need paper - attend to information from different representation subspaces at different positions) and most importantly preserve this diversity during the training process.


texts: Data-preprocessing for Machine Learning model

texts:
I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the co-domain range. Taking one hot encoding would increase the no. of features significantly. Because, each attribute has co-domain as following:

age: 10-19, 20-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, 90-99.
tumor-size: 0-4, 5-9, 10-14, 15-19, 20-24, 25-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59.
inv-nodes: 0-2, 3-5, 6-8, 9-11, 12-14, 15-17, 18-20, 21-23, 24-26, 27-29, 30-32, 33-35, 36-39.


texts:
Im trying to cluster my raw email data using numpy and pandas only.

Hence Im Trying to create an email dataframe from raw emails that I have. However the issue that I have is that I keep getting an AttributeError: 'DataFrame' object has no attribute 'message'.

I dont know where Im going wrong with my code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

def parse_raw_message(raw_message):
    lines = raw_message.split('\n')
    email = {}
    message = ''
    keys_to_extract = ['from', 'to']
    for line in lines:
        if ':' not in line:
            message += line.strip()
            email['body'] = message
        else:
            pairs = line.split(':')
            key = pairs[0].lower()
            val = pairs[1].strip()
            if key in keys_to_extract:
                email[key] = val
    return email

def parse_into_emails(messages):
    emails = [parse_raw_message(message) for message in messages]
    return {
        'body': map_to_list(emails, 'body'),
        'to': map_to_list(emails, 'to'),
        'from_': map_to_list(emails, 'from')
    }

email_df = pd.DataFrame(parse_into_emails(emails.message))

AttributeError           Traceback (most recent call last)
<ipython-input-21-df1de00c07ed> in <module>
----> 1 email_df = pd.DataFrame(parse_into_emails(emails.message))

~\Anaconda3\lib\site-packages\pandas\core\generic.py in __getattr__(self,
name)
   5065             if
self._info_axis._can_hold_identifiers_and_holds_name(name):
   5066                 return self[name]
-> 5067             return object.__getattribute__(self, name)
   5068
   5069     def __setattr__(self, name, value):

AttributeError: 'DataFrame' object has no attribute 'message'


texts: How should a neural network for unbound function approximation be structured?

texts:
I've heard that a multilayer perceptron can approximate any function arbitrarily exact, given enough neurons. I wanted to try it, so I wrote the following code:

#!/usr/bin/env python

"""Example for learning a regression."""

import tensorflow as tf
import numpy

def plot(xs, ys_truth, ys_pred):
    """
    Plot the true values and the predicted values.

    Parameters
    ----------
    xs : list
        Numeric values
    ys_truth : list
        Numeric values, same length as `xs`
    ys_pred : list
        Numeric values, same length as `xs`
    """
    import matplotlib.pyplot as plt
    truth_plot, = plt.plot(xs, ys_truth, '-o', color='#00ff00')
    pred_plot, = plt.plot(xs, ys_pred, '-o', color='#ff0000')
    plt.legend([truth_plot, pred_plot],
               ['Truth', 'Prediction'],
               loc='upper center')
    plt.savefig('plot.png')

# Parameters
learning_rate = 0.1
momentum = 0.6
training_epochs = 1000
display_step = 100

# Generate training data
train_X = []
train_Y = []

# First simple test: a linear function
f = lambda x: x+4

# Second, more complicated test: x^2
# f = lambda x: x**2

for x in range(-20, 20):
    train_X.append(float(x))
    train_Y.append(f(x))
train_X = numpy.asarray(train_X)
train_Y = numpy.asarray(train_Y)
n_samples = train_X.shape[0]

# Graph input
X = tf.placeholder(tf.float32)
reshaped_X = tf.reshape(X, [-1, 1])
Y = tf.placeholder("float")

# Create Model
W1 = tf.Variable(tf.truncated_normal([1, 100], stddev=0.1), name="weight")
b1 = tf.Variable(tf.constant(0.1, shape=[1, 100]), name="bias")
mul = tf.matmul(reshaped_X, W1)
h1 = tf.nn.sigmoid(mul) + b1
W2 = tf.Variable(tf.truncated_normal([100, 100], stddev=0.1), name="weight")
b2 = tf.Variable(tf.constant(0.1, shape=[100]), name="bias")
h2 = tf.nn.sigmoid(tf.matmul(h1, W2)) + b2
W3 = tf.Variable(tf.truncated_normal([100, 1], stddev=0.1), name="weight")
b3 = tf.Variable(tf.constant(0.1, shape=[1]), name="bias")
# identity as activation to get arbitrary output
activation = tf.matmul(h2, W3) + b3

# Minimize the squared errors
l2_loss = tf.reduce_sum(tf.pow(activation-Y, 2))/(2*n_samples)
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum).minimize(l2_loss)

# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})

        # Display logs per epoch step
        if epoch % display_step == 0:
            cost = sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y})
            print("cost=%s\nW1=%s" % (cost, sess.run(W1)))

    print("Optimization Finished!")
    print("cost=%s W1=%s" %
          (sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y}),
           sess.run(W1)))  # "b2=", sess.run(b2)

    # Get output and plot it
    ys_pred = []
    ys_truth = []

    test_X = []
    for x in range(-40, 40):
        test_X.append(float(x))

    for x in test_X:
        ret = sess.run(activation, feed_dict={X: x})
        ys_pred.append(list(ret)[0][0])
        ys_truth.append(f(x))
    plot(train_X.tolist(), ys_truth, ys_pred)

This kind of works for linear functions (at least for the training data, not so much for the testing data outside of the range):

However, it doesn't work at all for non-linear function $x^2$:

Why does this neural network not work for such simple function approximation? What do I have to change to make the same network topology work for both functions?


texts:
I am trying to synthetize clients Data in order to do clustering. My problem is for 1 customer I have severals rows. I would like to synthetize informations to get 1 row per customer. This clustering is about how customers use fidelity program. Here is a picture of my table : By Column (left to right) :

1) CustomerID 2)Date at which their use their points 3) Category number (Ex: 1 is gift card, 2 is a flight etc) 4) How many points they used 5) How many items they purchased with points

My question is how could I have 1 customer per row without loosing informations.
Maybe Pivot Table? But I don"t know how it work exactly. I am new to statistic btw.

Thank you

Cédric


texts: I have a dataset that gives data on infertitlity and causes. The dataset is mainly 0,1 to represent "yes" and "no". However, some fields have "Sometimes", "Often" which would be represented by -1 or 2. I've only learnt how to do categorical data i.e. 1,0 and Numberical data. So my question is since there are more options other than 1 and 0, which type of regression do I use? Logistic regression or Linear regression?


texts: What is the difference between global and universal compression methods?

texts:
I understand that compression methods may be split into two main sets:

global
local

The first set works regardless of the data being processed, i.e., they do not rely on any characteristic of the data, and thus need not to perform any preprocessing on any part of the dataset (before the compression itself). On the other hand, local methods analyze the data, extracting information that usually improves the compression rate.

While reading about some of these methods, I noticed that the unary method is not universal, which surprised me since I thought "globality" and "universality" referred to the same thing. The unary method does not rely on characteristics of the data to yield its encoding (i.e., it is a global method), and therefore it should be global/universal, shouldn't it?

My primary questions:

What is the difference between universal and global methods?
Aren't these classifications synonyms?


texts: interpreting multi linear regression results

texts:
am very new to all of this and am taking baby steps learning this (so please be merciful).

I have imported my csv file into python as shown below:

data = pd.read_csv("sales.csv")
data.head(10)

and I then fit a linear regression model on the sales variable, using the variables as shown in the results as predictors. the results are summarised below:

model_linear = smf.ols('sales ~ month + weekend + holiday + prod_function + prod_regime + prod_listprice + discount + stockout', data=data).fit()
print(model_linear.summary())

this is the part I have problems with. how should I interpret and make sense of this result? for instance, what does this say about the impact discount have on sales?


texts: Is there any implementation of Recursive Auto Encoders in Tensorflow?

texts: I am looking for the implementation of Recursive Auto Encoders (RAE) in tensor flow python. I want to model English sentence representations from a sequence to sequence neural network model. RAE is proven to be one of the best choice to represent sentences in recent machine learning approaches. I googled and didn't find any tensorflow Recursive Auto Encoders (RAE) implementation resource, please help. And if there is no implementation of RAE in tensorflow then please share the links of Theano.


texts: Leave one out Cross validation using sklearn (Multiple CSV)

texts:
I have 52 CSV files in a folder. I want to build a model based on this data. That's why I want to Leave one out cross-validation on these data. How can I do this using sci-kit learn in python?

I tried from sci kit document and also search many resources.

import glob
import numpy as np
import pandas as pd
from sklearn.cross_validation import LeaveOneOut
path=r'...................\Data\New
design process data'
filelist=glob.glob(path + "/*.csv")
loo=LeaveOneOut()
for train,test in loo.split(filelist):
   print("%s %s" % (train, test))

But it showed errors.

init() missing 1 required positional argument: 'n'

I am new in python as well as sci-kit learn. If anyone can help me, It would be a great convenience.


texts:
I'm trying to mine some data with Rstudio, but I have problem in the results output.

Trying to search big number from 100000 to 900000 the function works well but the output of the result is coded in 4 bits. It looks like this (X9438)
Also, I mentioned that this result a.12458a>a is shown like this a.12458a.a

I tried to extract the output into an external file thinking that the terms will be shown in it original format, but in vain.

Is there any package, configuration or plugin to add to resolve this problem?


texts: Does binning a time series with pd.qcut (using quantiles) create data leakage?

texts:
Let's say I want to predict whether a company will default on it's debt at some point in time (so binary classification) and one of the time series variables I'm using is the "revenue" of that company thought time.
Can I binned this variable "revenue" using quantiles cut (like so => pd.qcut(df['revenue'],bins=10)) without creating a data leakage ?
I'm under the impression that I can not really do that since the quantile cut is made by knowing the entire distribution of the variable "revenue" throughout the period. That is, the bin attributed to "revenue" at any point in time in my training data will carry information about the future.
Am'I correctly assuming that this will create a data leakage for this time series prediction problem ?
If so, can I safely use pd.cut instead with no quantile ?
Thank you


texts: How to begin understanding of audio and music analysis

texts:
I recently have been assigned to do some work with the python libROSA library.  I don't have extensive experience with audio and music analysis and the apis and docs seem to assume a higher level of understanding.  For example, the hello world example says things like:

The example is encoded in OGG Vorbis format,
The variable sr contains the sampling rate of y, that is, the number
of samples per second of audio.
By default, all audio is mixed to mono and resampled to 22050 Hz at
load time.

And I am "why do you need to encode audio?" and "why do you even need to sample? (analog v digital I guess)" and "why do you need to mix to mono?"  "What does that even mean, 'mix'"?

Is there a good book(s) or sites that can help me get a basic understanding of audio and music processing?

Thanks in advance


texts:
In Neural networks [3.8] : Conditional random fields - Markov network by Hugo Larochelle it seems to me that a Markov Random Field is a special case of a CRF.

However, in the Wikipedia article Markov random field it says:

  One notable variant of a Markov random field is a conditional random field, in which each random variable may also be conditioned upon a set of global observations o.

This would mean that CRFs are a special case of MRFs.

Definitions

Markov Random Field

Again, according to Wikipedia

  Given an undirected graph $G=(V,E)$, a set of random variables $X = (X_v)_{v\in V}$ indexed by $V$  form a Markov random field with respect to $G$  if they satisfy the local Markov properties:

  Pairwise Markov property: Any two non-adjacent variables are conditionally independent given all other variables:
  $X_u \perp\!\!\!\perp X_v \mid X_{V \setminus \{u,v\}} \quad \text{if } \{u,v\} \notin E$

  Local Markov property: A variable is conditionally independent of all other variables given its neighbors:
  $X_v \perp\!\!\!\perp X_{V\setminus \operatorname{cl}(v)} \mid X_{\operatorname{ne}(v)}$
  where ${\textstyle \operatorname{ne}(v)}$ is the set of neighbors of $v$, and $\operatorname{cl}(v) = v \cup \operatorname{ne}(v)$ is the closed neighbourhood of $v$.

  Global Markov property: Any two subsets of variables are conditionally independent given a separating subset:
  $X_A \perp\!\!\!\perp X_B \mid X_S$
  where every path from a node in $A$ to a node in $B$ passes through $S$.

Please note if you know a citable source which gives a good definition.

Conditional Random Fields

According to Wikipedia:

  Lafferty, McCallum and Pereira[1] define a CRF on observations $\boldsymbol{X}$ and random variables $\boldsymbol{Y}$ as follows:

    Let $G = (V , E)$ be a graph such that

    $\boldsymbol{Y} = (\boldsymbol{Y}_v)_{v\in V}$, so that $\boldsymbol{Y}$ is indexed by the vertices of $G$. Then $(\boldsymbol{X}, \boldsymbol{Y})$ is a conditional random field when the random variables $\boldsymbol{Y}_v$, conditioned on $\boldsymbol{X}$, obey the Markov property with respect to the graph: $$p(\boldsymbol{Y}_v |\boldsymbol{X}, \boldsymbol{Y}_w, w \neq v) = p(\boldsymbol{Y}_v |\boldsymbol{X}, \boldsymbol{Y}_w, w \sim v)$$ where $\mathit{w} \sim v$ means that $w$ and $v$ are neighbors in G.

  What this means is that a CRF is an undirected graphical model whose nodes can be divided into exactly two disjoint sets $\boldsymbol{X}$ and $\boldsymbol{Y}$, the observed and output variables, respectively; the conditional distribution $p(\boldsymbol{Y}|\boldsymbol{X})$ is then modeled.

Question

What is the relationship between Markov Random Fields and Conditional Random Fields?


texts: Preparing data for a live prediction engine

texts:
I need to build a live prediction engine (either using Ensemble methods/ RNN/ Keras Classifier) that could learn from historical data what kinds of users are likely to transact and, in real-time, generate transaction probabilities for current users on a DAILY basis. I do have their demographics, transactional and app activities data. There is an amount limit that can be transacted by the users in a month. How do I prepare the historical data to predict the next day if he is going to transact or not? Thanks.

Demographics:

CustomerID | CustomerAge | Gender | Nationality | Joining Date | Loyalty Program Opt-in | Address

1222 | 25 | M | British |  21-Apr-2016 | Yes | Old kent

Transactions:

Beneficiary | CustomerID| TransactionDateTime | TransactionAmount

4534534 | 545544 | 19/02/2016 16:02:56 | 1888.85

4353222 | 545544 | 23/02/2016 08:49:49 | 3242.43

2453453 | 535455 | 26/02/2016 10:10:56 | 4776.57

5353532 | 435353 | 11/02/2016 12:43:38 | 4862.16

4532522 | 345353 | 23/02/2016 17:42:22 | 4923.54


texts: Choosing class labels from annotated data

texts:
For a multi label, multi class categorization on a social media dataset, we have collected around 5000 samples from the dataset and have manually annotated them. 5000 samples are labelled by 3 people, and 1500  of them are same posts.  How should I decide the category label now?

For example,

AnnotatorId  Post  Labels
-------------------------
Annotator1: post1: A,B,C
Annotator2: post1: A,D,E
Annotator3: post1: B,D,E

Will the label for post1 be A,B,D through majority voting? Or, is there some better, commonly used approaches?


texts:
With all the hoopla around Data Science, Machine Learning, and all the success stories around, there are a lot of both justified, as well as overinflated, expectations from Data Scientists and their predictive models.

My question to practicing Statisticians, Machine Learning experts, and Data Scientists is - how do you manage expectations from the businesspeople in you company, particularly with regards to predictive accuracy of models? To put it trivially, if your best model can only achieve 90% accuracy, and upper management expects nothing less than 99%, how do you handle situations like these?


texts: What regressors are recommended with text modeling?

texts:
For the sake of my own exploration, I am working on a sales prediction project. I am using text extracted from a set of books to build a predictive model.

With scikit learn, I have created a Tfidf, and together with the numeric sales numbers, I created a SGDRegressor. But I'd like to practice with other models.

I am looking over the options available to me, and I'm wondering, what other algorithms might be useful in this kind of regression scenario? And/or, what other algorithms in scikit learn will take a tfidf as the dataset?


texts: python xgboost DMatrix - get feature values or convert to np.array

texts:
I'm trying to create a custom evaluation metric (feval) function for xgboost.cv. It should process some of the training features, however I can't find a way to extract features from xgboost.DMatrix() object (only labels)..

As I was suggested it can be done with first making a DMatrix slice then saving it to a svmlight file and finally reading it with scikit-learn, but is there a more elegant way if doing it?


texts: Can k-means clustering get shells as clusters?

texts:
Imagine you have $k$ classes. Every class $i$ has points which are follow a probability distribution, such that their distance to 0 is $i$ in mean, but this distance follows a normal distribution. The direction is uniformly distributed. So all classes are in shells around the origin 0.

Can $k$-means get these shells when you choose the "right" distance metric?
(Obviously it can't find it if you take the euclidean metric, but I wonder if there is any metric at all or if this problem is inherently unsolvable by $k$-means, even if you know the number of clusters $k$)


texts: What are the audio features to best describe a music?

texts:
I'm working on the content-based filtering part of a recommender system for an audio streaming project.

I firstly used the k-mean algorithm with music genres and one-hot encoding to  classify musics into different groups.
But, in order to get more precise results I want to change it and use audio features to feed the model instead.

So my questions are:
 - is my approach correct.
 - what are the most relevant audio features I can extract from an audio file.

Thx for your answers.

edit:
right now, I'm extracting those features:

music tempo
zero-crossing rate
duration
spectral centroids
spectral roll-off
MFCC
spectral bandwidth
spectral contrast

I want to know at which degree those audio features are relevant to 'describe' an audio extract.


texts: How to get the mu+0.1sigma from a normal distribution?

texts:
I have a normal distribution and I would like to know what is the confidence level of mu+0.1sigma (where mu is the mean and sigma the standard deviation). How do i get the confidence level?

Using python or R


texts: Parameter estimation using convolutional for exponential functions

texts:
I was inspired by this paper where the authors fed an 1-dimenisonal input waveform to the simple convolutional neural network and got an estimate of four parameters of the analytic model. This inspired me to see if I could use CNNs to estimate parameters of more complex analytic models.
I have a complex non-linear (contains exponential and Bessel functions) set of equations that depend on five parameters:

v1 = 0.1  -> 5
v2 = 1e3  -> 1e4
v2 = 1e3  -> 1e4
v4 = 0.01 -> 0.5
v5 = 0.01 -> 1.0

The output waveform is exponential and can vary 4-6 orders of magnitude over the observed time period.
I generated around 200K sample waveforms and the parameter values used to generate them. I construct the following network in Tensorflow:

Model: "sequential_1"
_____ Layer (type)                 Output Shape              Param #
===== conv1d_2 (Conv1D)            (None, 376, 128)          2048
_____ conv1d_3 (Conv1D)            (None, 372, 64)           41024
_____ dense_7 (Dense)              (None, 372, 512)          33280
_____ dense_8 (Dense)              (None, 372, 512)          262656
_____ dense_9 (Dense)              (None, 372, 512)          262656
_____ dense_10 (Dense)             (None, 372, 512)          262656
_____ flatten (Flatten)            (None, 190464)            0
_____ dense_11 (Dense)             (None, 512)               97518080
_____ dense_12 (Dense)             (None, 512)               262656
_____ dense_13 (Dense)             (None, 5)                 2565
===== Total params: 98,647,621 Trainable params: 98,647,621 Non-trainable
params: 0

When I go to train the network I noticed that the accuracy remains low and loss is large. When I go to predict parameters I notice that I get a rough (~20% difference between predicted and actual) estimate for v2/v3 but the rest of the variables are 0. This leads me to believe that the issue is the many orders of magnitude difference between the input and different outputs.
Hence, I did the following, I normalized the data by taking the logarithm of the input curve to ensure the input is <20. I also took the logarithm of the parameters and added a constant bias so the parameters (network output) are all 0-3. This time when I trained the network, the accuracy was even worse!
I tried a couple different networks and normalizations but so far nothing has shown any promise. So now I'm wondering if any of you guys has an ideas? Should I be trying a different kind of NN (other than CNN)? Is there something I can do to have the NN more amenable to handling exponentially varying inputs/outputs?
Finally does anyone know of any papers that does parameter estimation of exponential analytic functions? The one I linked fits a linear function and I think the complexity might be the major issue here.


texts:
I have a large dataset (around $ 10^6 $ samples) and an algorithm that will surely choke on that much data.

Suppose that I have removed duplicates and near-duplicates. What are the well-known techniques for reducing sample size without losing too much of the information possibly encoded in the initial dataset?

I thought about using some clustering algorithm (which scales well with respect to number of clusters, possibly BIRCH) and use the resulting clusters to find $ N $ nearest points to cluster centroid. However this feels somehow wrong.


texts: How much data warrants building a pipeline?

texts: If I'm doing simple aggregation dashboards, what's the minimum amount of data which justifies building a full blown data processing pipeline? Is it reasonable to build a complicated pipeline with Mysql -> Hadoop -> Redshift and then analytics and visualizations on top of redshift if my database only has about 10million transactions per month. Would it be better to just run something like ReDash visualizations on top of my unaggregated data in MySql?


texts: Why do we do small number of iterations per epoch and large number of epochs rather than one epoch iterate on entire batch until small gradient?

texts: Why do we do many epochs and one iteration per epoch rather than one epoch and iterate while gradient is not within tolerance in Keras deep learning or by default?


texts: XGBoost (R Interface) throwing "label set cannot be empty" error

texts:
I am trying to use the XGBoost model to perform a multi-class classification over 40 classes.

The code is as follows:

xgb_params = list(colsample_bytree= 0.7,
                  subsample = 0.7,
                  eta = 0.05,
                  objective= 'multi:softmax',
                  max_depth= 5,
                  min_child_weight= 1,
                  eval_metric= "mlogloss", num_class = categoryclassnos,
                  nthread=4)

fit.xgb = xgb.train(params = xgb_params,
                    data = dtrain,
                    nrounds = 500,
                    watchlist = list(train = dtrain, test=dtest),
                    print_every_n = 50)

However, I am getting the following error:

  Check failed: (info.labels.size()) != (0) label set cannot be empty

I have reproduced the dataset and the R script here.

Any help/ pointers are deeply appreciated.


texts:
I understood random forest is building a model with multiple decision trees,

Row sampling is based on bootstraping

My question is how feature selection is happening for each tree ?
Any help would be appreciated.


texts: Building a Citation Network to Analyze in R

texts:
I am working on network analysis project at one of our Nation's Service Academies, and I need a little help.

As a starting point, we are looking at a citation network that we build by using the keyword, "Network Analysis", for peer reviewed articles only in Web of Science. We took the top 50 results, and looked at the articles to which those cited and built a citation network.  It outputs the information in Bibtex, Tab-Delimited, HTML, Plain Text, End Note, and a few others.

Here is my question:

What is the best way to create a citation network?

We have these formats, but it is not usable yet.  Ideally, we would have a matrix with article names on the x and y, and binary numbers for the
    connections.


texts: Mutate with custom function in R does not work

texts:
I have a data frame, containing a column called: "Frequency". Frequency has values like "Year", "Week", "Month" etc. Now I want to create a new column based on the Frequency column where Year's new corresponding value will be 1, Month's corresponding value will be 12 and Week corresponding value will be 48. I tried to make a function for this as "getValue" and tried to make a new column applying a mutation (dplyr) on that funcion. But unfortunately I am getting the following warning and all values being transformed by the value of "1".

getValue <- function(input) {

          if (input == 'Year')
          {
            result <- 1
          }
          else if(input == 'Month')
          {
            result <- 12
          }
          else if(input == 'Week')
          {
            result <- 48
          }

          return(result)
        }

      Data =  mutate(gateway, YearlyHit = getValue(gateway$Frequency))
      Data

Here is the warning message I am getting -

Warning message:
In if (input == "Year") { :
  the condition has length > 1 and only the first element will be used

How can I achieve the required result in R?


texts: A feature highly correlated with target variable

texts: What if one of the predictor variables is highly correlated with the target variable (say 0.9), what should we do? Should we drop it or keep it to build the prediction model(classification or regression)?


texts: Predictions become random after loading a custom saved Keras model?

texts:
I did not found a conclusive answer for this problem so I am posting it here. I created a Keras model with custom layers and trained it on a classification task at which it achieves a completely satisfying performance with an accuracy of about 65%. However, after saving and reloading it the performance completely deteriorates to about 25% which is akin to a completely random prediction. I have been trying different tricks (fixing seeds for numpy & tf backend, using alternative library functions keras.models.save_model, pickling) but none of it works to a satisfying degree.
The code I am using is the following. I am using additional custom "GraphConv" layers (c.f. Kipf & Well, 2017):
data_shape = (np.shape(Xp)[1], 2)
num_classes = 4

in_x = Input(shape=data_shape)
x = GraphConv(filters=25, adjacency_matrix=A[0], input_features=2)(in_x)
x = Activation('relu')(x)
x = MaxPooling1D(pool_size=4, padding='same')(x)
x = GraphConv(filters=16, adjacency_matrix=A[2], input_features=25)(x)
x = Activation('relu')(x)
x = MaxPooling1D(pool_size=4, padding='same')(x)
x = Flatten()(x)
x = Dense(10)(x)
x = Activation('relu')(x)
x = Dense(50)(x)
x = Activation('relu')(x)
out_x = Dense(num_classes, activation='softmax')(x)

model = Model(inputs=in_x, outputs=out_x)
model.compile(loss=keras.losses.categorical_crossentropy, optimizer='adam', metrics=['accuracy'])
model.summary()

h = model.fit(X_train, Y_train, epochs=10, batch_size=250, verbose=1, validation_split=0.2)

After training and achieving about 65% accuracy, I save the model with
model.save("test_model")

Based upon the saved model, I create a new one with
test = keras.models.load_model("test_model", custom_objects={'GraphConv': GraphConv})

Now evaluating and comparing the performances I get for the vanilla model:
model.evaluate(X_train, Y_train)

4000/4000 [==============================] - 1s 151us/step
[0.8065748610496521, 0.6245]

And for the new test model:
test.evaluate(X_train, Y_train)

4000/4000 [==============================] - 1s 204us/step
[7.069425365447998, 0.25375]

I have been thinking about whether some of the parameters of the custom layers are not correctly saved or the weights are not the same of both models. But after some testing and comparing model.get_weights() and test.get_weights() as well as the parameters in model.layers and test.layers, I can only confirm that they are the same for both models.
So at this point, I am completely clueless about how I could still solve this problem.
Will be very thankful for any suggestions I receive!


texts:
What are the differences between DQN DDQN and Dueling Q network?
And how they expressed in the agent's behaviour? (For example in an Open AI Gym environment)


texts:
I need a sanity check. I want to create an anomaly detection system.

The logic which I am planning to use is the following:

Find anomalies in the past using Seasonal Hybrid Extreme Studentized Deviate Test.
Binarise the anomalies (1 the anomalies and 0 the trends).
Run several algorithms (Autoencoders, SVM, Logistic Regression, Naive Bayes, Lasso Regression, etc) with variables that are correlated and validate the models and use it.

Does the binarisation process makes sense?


texts:
I am using gridsearchcv to tune the parameters of my model and I also use pipeline and cross-validation. When I run the model to tune the parameter of XGBoost, it returns nan. However, when I use the same code for other classifiers like random forest, it works and it returns complete results.
kf = StratifiedKFold(n_splits=10, shuffle=False)

SCORING = ['accuracy', 'precision', 'recall', 'f1' ]

# define parametres for hypertuning
params = {
    'Classifier__n_estimators': [5, 10, 20, 50, 100, 200]
}

XGB = XGBClassifier()
UnSam = RepeatedEditedNearestNeighbours()

pipe = Pipeline(steps=[('UnderSampling', UnSam ), ('Classifier', XGB)])
# ___________________________________________

mod = GridSearchCV(pipe, params, cv =kf, scoring = SCORING, refit='f1', return_train_score=True)
mod.fit(X_train, y_train)

Here is my code and when I run it, the following results are obtained:
{'Classifier__n_estimators': 5}
__________________________________________________
F1 :  [nan nan nan nan nan nan]
 Recall :  [nan nan nan nan nan nan]
 Accuracy :  [nan nan nan nan nan nan]
 Precision :  [nan nan nan nan nan nan]

Another thing that is weird is that when I apply the same code for tunning the penalty in Logistics Regression, it returns nan for l1 and elasticnet.
kf = StratifiedKFold(n_splits=10, shuffle=False)

SCORING = ['accuracy', 'precision', 'recall', 'f1' ]

# define parametres for hypertuning
params = {
    'Classifier__penalty': ['l1','l2','elasticnet']
}

LR = LogisticRegression(random_state=0)
UnSam = RepeatedEditedNearestNeighbours()

pipe = Pipeline(steps=[('UnderSampling', UnSam ), ('Classifier', LR)])
# ___________________________________________

mod = GridSearchCV(pipe, params, cv =kf, scoring = SCORING, refit='f1', return_train_score=True)
mod.fit(X_train, y_train)

The results are as follows:
{'Classifier__penalty': 'l2'}
__________________________________________________
F1 :  [  nan 0.363   nan]
 Recall :  [   nan 0.4188    nan]
 Accuracy :  [   nan 0.7809    nan]
 Precision :  [   nan 0.3215    nan]


texts: What is the proper order of normalization steps before and after splitting data

texts:
I use a classification model on time-series data where I normalize the data before splitting the data into train and test. Now, I know that train and test data should be treated separately to prevent data leaking. What could be the proper order of normalization steps here? Should I apply steps 1,2,3 separately to train and test after I split data with the help of a sliding window? I use a sliding window here to compare each hour (test) with its previous 24 hrs data (train). Here is the order that I am currently using in the pipeline.

Moving averages (mean)
Resampling every hour
Standardization
Split data into train and test using a sliding window (of a length 24 hrs (train) and slides every 1 hr (test))
Fit the model using train data
Predict using the test data


texts: How to use NLP to determine the normal words in the text

texts:
I have a lot of .txt files. I want to train the model which remove bad words from the document(bad word - random sequence of characters).
For example: bad words - hsA , ubj8 ,t+h ,SSSh ,_^[ ,SVWh ,SSSSSS
normal words : *.exe , copy , write ,write ,ftp.exe


texts: Multiple Output Layers in Neural Networks in Deep Q Learning

texts:
I am trying to train a computer to play with LEGO bricks (simulated) using DQN.
My input is an image with 4 color channels (RGB and depth) and the output of the neural network is the coordinates of the best position where the next brick should be put.

Instead of having $m_x * m_y * m_z$ (where $m_k$ is the size of positions along the k axis) different outputs, I want to have 3 output layers, each of which returns the x, y or z coordinates of the best position, so I have only $m_x + m_y + m_z$ outputs.

My idea is replacing the fully-connected output layer by three fully-connected output layers, but only have one reward value (labeled value) for three layers. Can I do this? Will the three layers returns different outputs if the reward is same?


texts: How to handle image classification network where output classes can be subsets of one another?

texts:
For instance, if I wanted a train network that can output

van
truck
sedan
vehicle
pedestrian

does it make sense to only train it on van, truck, sedan, and pedestrian and then make "vehicle" a synonym of van, truck, sedan? Or does it make sense to train vehicle as its own class made up of the combined training data of van, truck, sedan?

EDIT ---

Here's a more clear example I think, what if there are labels that are not exclusive? For instance, a child can also be a pedestrian. How do I train a network to output both labels if it, for example, sees a child walking down the street? It seems that using a softmax later as the final layer of the network would not allow this to work as all probabilities need to sum to one, in this case it could be 90% child and 80% pedestrian and that's entirely valid.


texts:
I have a large data set(460 Mb) which has a column - Log with 386551 rows.
I wish to use clustering and N-Gram approach to form word cloud. My code is as follows:

library(readr)
AMC <- read_csv("All Tickets.csv")
Desc <- AMC[,4]

#Very large data hence breaking it down for creating corpus
#DataframeSource has been used insted of VectorSource is to be able to       handle the data

library(tm)
docs_new <- data.frame(Desc)

test1 <- docs_new[1:100000,]
test2 <- docs_new[100001:200000,]
test3 <- docs_new[200001:300000,]
test4 <- docs_new[300001:386551,]
test1 <- data.frame(test1)
test1 <- Corpus(DataframeSource(test1))
test2 <- data.frame(test2)
test2 <- Corpus(DataframeSource(test2))
test3 <- data.frame(test3)
test3 <- Corpus(DataframeSource(test3))
test4 <- data.frame(test4)
test4 <- Corpus(DataframeSource(test4))

# attach all the corpus
docs_new <- c(test1,test2,test3,test4)

docs_new <- tm_map(docs_new, tolower)
docs_new <- tm_map(docs_new, removePunctuation)
docs_new <- tm_map(docs_new, removeNumbers)
docs_new <- tm_map(docs_new, removeWords, stopwords(kind = "en"))
docs_new <- tm_map(docs_new, stripWhitespace)
docs_new <- tm_map(docs_new, stemDocument)
docs_new <- tm_map(docs_new, PlainTextDocument)

#tokenizer for tdm with ngrams
library(RWeka)
options(mc.cores=1)
BigramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 2, max =2))
tdm <- TermDocumentMatrix(docs_new, control = list(tokenize = BigramTokenizer))

This is giving me results as follows:

TermDocumentMatrix (terms: 1874071, documents: 386551)>>
Non-/sparse entries: 17313767/724406705354
Sparsity           : 100%
Maximal term length: 733
Weighting          : term frequency (tf)

I then converted it to dgMatrix using following :

library("Matrix")
mat <- sparseMatrix(i=tdm$i, j=tdm$j, x=tdm$v, dims=c(tdm$nrow, tdm$ncol))

While trying to use the following I am getting memory size error:

removeSparseTerms(tdm, 0.2)

Please suggest further as I am new to Text Analytics.


texts: Mining Association rules from a data warehouse and a transactional database

texts: I wonder if it is possible to perform market basket analysis to extract the association rules from a data warehouse and a transactional database in the same time to predict the future purchases of a specific user?


texts: What do I initialise each model in cross validation with in a multi-layer Perceptron?

texts:
So, as far as my understanding goes, cross-validation is used to determine the best model.

I understand that once we determine the best model, we then train it on the entire dataset. I'm supposed to be using cross-validation for the multi-layer perceptron that can classify the MNIST dataset. I don't seem to get how cross-validation fits in training the model.

Let's say I'm using 5-fold cross-validation, which means I will have to make 5 different models but, how will the training of these individual model proceed? In particular, I have the following questions:

Will the training of these individual model be as usual(backward propagation)?
What do I initialise each model with? (Random Weights?)
After completing the cross-validation, I have the best model(say B) with me now, what does it mean to train this model on the entire dataset?
(Does it mean, I initialise the weights of the new model being trained on the whole dataset, with those of B).


texts:
I'm trying to write a program that can take multiple low-resolution images as inputs and output a high-resolution image.

My understanding is that for single-image superresolution, Convolutional Neural Networks work great. I can just take a network with just three convolution layers as described here. I can then train the network with a huge dataset of low-res images and their corresponding high-res images (with mean-squared error cost function), and it should theoretically work.

If I want to achieve greater accuracy, though, I can gain more information about a potential high-resolution image by looking at multiple low-resolution images.

So my question is: How would I modify this neural-network based algorithm to be able to take multiple images (of the same object) as input, and output a more accurate high-resolution image as a result? I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.

(Sorry if anything I said is incorrect; I know virtually nothing about this field, so any advice helps!)


texts:
After running xgboost model with:

objective = 'binary:logistic'
eval_metric = 'logloss'

I have a group of 3 variables that have the highest values of gain. Now, if I replace each one of the 20 more important variables according to this metric by their mean one by one and calculate the kolmogorov smirnov coefficient (KS), I get that the one that reduces the most the ks is not one of those 3, but one that has a relative low gain.

Gain

    Gain    Cover
v1  21.5%   2.5%
v2  12.9%   4.1%
v3  11.1%   1.8%
v4  3.5%    3.4%
v5  2.7%    1.7%
v6  2.4%    2.5%
v7  2.3%    2.2%
v8  2.2%    1.9%
v9  1.9%    4.0%
v10 1.9%    2.0%
v11 1.9%    0.9%
v12 1.6%    4.6% *****

ks of replacing each variable by its mean (one by one)

v12         39% *****
the rest    45%

How is this explained? Thanks.


texts: [under/over]-sampling teaches model the wrong distribution?

texts:
TLDR: Will under/oversampling during the training phase teach the model the wrong distribution and adversely affect accuracy?
Let us assume you want to train a classifier to differentiate between class A and class B. Unfortunately, the population distribution of A and B is unbalanced at a ratio of [1:100]. As such, you utilize under-sampling or over-sampling such that the training and validation sets effectively achieve a [1:1] ratio between A and B. You do nothing to the test set. The distributions of the sets and and the training results are in the below table:
           Train  Val    Test
A Dist.    0.5    0.5    0.99
B Dist.    0.5    0.5    0.01
Accuracy   1.0    0.999  0.85

You have now trained a model which performs worse on the population than a "classify all as A" approach. Does over-sampling or under-sampling teach a model the wrong distribution - causing it to over-confidently predict minority classes? If not - what could be happening in this example?


texts: XGBoost feature importance has all features but decision tree doesn't

texts:
I have used XGBoost to train a model with 400 features. My understanding is that since the max_depth is default at only 6, and 2^6 < 400, not all features will end up in the tree.
How come when I output the feature importance chart, it shows every single feature with above 0 importance? The decision tree output clearly shows that not every feature has been used in the final tree.


texts: Why do we need XGBoost and Random Forest?

texts:
I wasn't clear on couple of concepts:

XGBoost converts weak learners to strong learners. What's the advantage of doing this ? Combining many weak learners instead of just using a single tree ?
Random Forest uses various sample from tree to create a tree. What's the advantage of this method instead of just using a singular tree?


texts: Can anyone provide specific techniques with using ICD-10 codes in Machine Learning? I have usually used a simply approach of creating multiple binary column representing ICD-10 codes… which can get extremely long. Or I have used Hashing features. Are there other techniques or ways to use ICD-10 codes in ML? Anyone can provide a useful link to see various approaching to modeling such features?


texts: I've got predicted results from two different types of neural networks. Now I would like to run significance testing on both of the results to prove that they do not have equal predictive accuracy. I've learnt that the only tool in the game for this is Diebold-Mariano test. What tool I can use to run this testing (Matlab? R?)


texts: boltzmann-exploration(softmax exploration) in reinforcement learning

texts:
I have started learning reinforcement learning and as a part of it I am exploring the action selection strategies available. I am comparing epsilon-greedy vs boltzmann exploration(softmax exploration). I have understood and implemented epsilon greedy the follwing way.

def get_action(state, episode):
# get action from model using epsilon-greedy policy
# Decay in ε after we generate each sample from the environment
    epsilon = epsilon_min + (epsilon_max - epsilon_min) * np.exp(-epsilon_decay*episode / 4)

    if np.random.rand() <= epsilon: # Exploration: randomly choosing and action
        action = [round(random.uniform(0.3, 1.0), 2), round(random.uniform(0.3, 1.0), 2)]

    else: #Exploitation: this gets the action corresponding to max q-value of current state
        state_encod = np.reshape(state, [1, state_size])
        q_values = model.predict(state_encod)
        action = np.argmax(q_values)

    return action, epsilon

I am not completely fluent with boltzmann and facing difficulty with implementation. can some one help me with this implementation. Tried a lot but couldn't. Really stuck at this.

Thanks.


texts: Multiclass classification of timeseries data using NN

texts:
I have the following dataset that I was thinking of using RNN/LSTM for classifying the protocol. Data contains features from packet capture that has only two provided fields: OUI of the MAC address of the device and the inter-arrival time (IAT) of the packets for a specific protocol.

For example, the labeled data for http and ntp from the point-of-view of a specific device type is provided below:

# MAC, ConnectivityIAT  -> protocol
ff:f1:f2, 10, -> http
ff:f1:f2, 20, -> http
ff:f1:f2, 30, -> http
ff:f1:f2, 0, -> http

ff:f1:f2, 3, -> ntp
ff:f1:f2, 6, -> ntp
ff:f1:f2, 9, -> ntp

The task is to predict the correct class when something like:

ff:ff1:f2, 10 -> ?

is seen.

Questions:

How do I convert the categorical value of OUI to numerical entities
How to approch the solution using a NN ?


texts: How to Use Shap Kernal Explainer with Pipeline models?

texts:
I have a pandas DataFrame X. I would like to find the prediction explanation of a a particular model.

My model is given below:

pipeline = Pipeline(steps= [
        ('imputer', imputer_function()),
        ('classifier', RandomForestClassifier()
    ])
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
y_pred = pipeline.fit(x_train, y_train).predict(x_test)

Now for prediction explainer, I use Kernal Explainer from Shap.

This is the following:

# use Kernel SHAP to explain test set predictions
shap.initjs()

explainer = shap.KernelExplainer(pipeline.predict_proba, x_train, link="logit")

shap_values = explainer.shap_values(x_test, nsamples=10)

# # plot the SHAP values for the Setosa output of the first instance
shap.force_plot(explainer.expected_value[0], shap_values[0][0,:], x_test.iloc[0,:], link="logit")

When I run the code, I get the error:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames.

Provided model function fails when applied to the provided data set.

ValueError: Specifying the columns using strings is only supported for pandas DataFrames

Can anyone please help me? I'm really stuck with this. Both x_train and x_test are pandas data frames.


texts: What is a product most frequently bought together with

texts:
I want to find out what people also buy when they buy a "Bike Helmet". There are only 77 different products in my dataframe.

I might solve this using a for loop, and creating a counter. So whenever ProductName == 'Bike Helmet, look at the OrderID and increase the counter by 1 for all those product names. Then find the next instance of a bike helmet purchase. This approach, however, is slow.

  OrderID           ProductName  PricePerOrder  ProfitPerOrder  Quantity  \
239    10337           Bike Helmet     805.200000      178.080000        24
238    10337           Mehmet-Skor       5.200000        0.626667        28
237    10337          Mehmet-Tröja       7.990000        1.540000        20
236    10337          Oyaki Kimono      10.025000        2.558333        25
240    10337           DSW Sandals       3.600000        0.628000        40
40     10262          Runner Shoes      14.362500        1.946250        15
42     10262       O-Man Underwear       0.535680       -0.030720        12
41     10262         Slip-on Shoes       4.606667        0.050667         2
336    10374           Onesy Dress      38.677500       10.800000        15
337    10374  Rossi Bermuda Shorts      13.665000        2.370000        30


texts: Why are we taking the square root of the gradient in Adagrad?

texts:
This is how we update weights with Adagrad:
$$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$
where, $w_i$ is the $i^{th}$  weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all the timesteps and $E$ is epsilon to prevent division by zero.
Now why are we taking the square root of $g_i+E$, can't we just remove the root and just divide.
I know removing the root gives bad performance, but why does it gives bad performance on removing the root?


texts: Multi-class text classification with LSTM in Keras

texts:
I'm quite new to Deep Learning and trying to solve the problem of Multi-Class, multi-label text classification using Deep Learning.

https://github.com/fchollet/keras/blob/master/examples/imdb_cnn_lstm.py .
I've another dataset. int form of a csv file ("text","classifier"), on which i want to perform text classification task. I've tried a few ways to pass my training text to keras but couldn't so I'm stuck at this point. Can anyone suggest me how should I pass my "train.csv" and "test.csv" file to the X_train, y_train and X_test, y_test?

Typically stuck in this line.

(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)

'train.csv' has this format:

"Job Description:An ideal fitment would apply his/ her advanced analytics expertise at a cutting edge Industrial Analytics specialized Data Science organization; primarily, in any of the following areas- automotive/ energy/ oil & gas/ aerospace/ marine/ chemical.  Experience in statistical modeling, predictive modeling, Random forests, Decision tree, Linear Regression, Correlation, Time- series.  BE / MS/ PhD in Mechanical/ OR/ IE/ computer science/ EE/ chemical.    Mentor/ Lead a small team of data scientist",Business Analytics

'test.csv' has same format that is "job_description","category"


texts: Clustering with imbalanced data and groups

texts:
I have a problem that is about identifying clusters of highly correlated items. I initially focused on building a model and features that put similar data items close to each other. The main challenge is that I have a case of imbalanced data, as follows:

Tens of Millions of items are random and not necessarily correlated.
Hundreds of clusters of items (composed of 10-1000s of elements) exist* or may emerge. *I do have partial ground truth for the existing ones.
Clusters are very different, in size and properties.

I'd like to return the identified clusters, and the elements within each cluster. F1 should be a good measure.
To move forward, I can think of threshold-based hierarchical clustering. Are there other techniques to consider?


texts: What exactly does the model generation mean in this diagram?

texts: I've been trying to grasp a research paper on image colorization using neural networks here I am stuck at this diagram. What I need help on, is the Model Generation step  after Feature extraction. What exactly do we do in this step?


texts:
I have 2 column in data frame, X and Y. And I have some string values stored in text, which I want to put in X, Y as shown in the example.

Example :

text=9 10 13 110 14 16
     12 1 6   1  1  2

X    Y

9    12

10   1

13   6

110  1

14   1

16   2


texts:
Matlab is a great tool for some mathematical experiments, Neural Networks, Image Processing ...

I would like to know if there is such a comprehensive and strong tool for data manipulation and NLP tasks? such as tokenization, POS tagging, parsing, training, testing ....

However I am new to NLP and I need a tool which let me experiment, get familiar and progress


texts: Classification model using RNN(action detection)

texts:
1) Could it be useful to use RNN for classification problem?(e.g.
to distinguish which action is taken: car is going, walking,
digging, nothing).
If 1 question is positive, how should RNN structure look like?

I have dataset of 4 actions, many examples for each action, each example includes 124 samples.
So my X_train, X_test are arrays of float(400000, 124, 1); y_train, y_test are arrays of int(0 or 1 or 2 or 3 depends on action).

My data preprocessing:

X_train, X_test, y_train, y_test = np.array(X_train), np.array(X_test), np.array(y_train), np.array(y_test)
X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))

My structure:

regressor = Sequential()
regressor.add(LSTM(units=55, return_sequences=True, input_shape=(X_train.shape[1], 1)))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units=50))
regressor.add(Dropout(0.2))
regressor.add(Dense(4,activation='softmax'))

regressor.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
regressor.fit(X_train, y_train, epochs=5, batch_size=32)
y_pred = regressor.predict(X_test)


texts: When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score normalisation on min-max normalised data? That is if the input data is already normalised to [0, 1], is it a good idea to train on the z-scores of that?


texts:
I need to compare two groups of people, where the independent variable is having a college degree, and the dependent variable is the income.

The problem is that if I divide the whole population of the study into two groups,
one of them has signally more persons, so the mean is affected by the size of each group and the value of the outliers.

How can I demonstrate or reject the theory that having a college degree
among the population of the study, ensures a better income?


texts: Recommendation matrix as a product of User Similarity and Ratings

texts:
For both item-item and user-user collaborative filtering the recommendation matrix $Γ_{m x n}$, which is an (m x n) matrix, can be defined as:

$$Γ(i,j)=r_{ij}$$
For user-user collaborative recommendation r_ij is defined as

$$r_{ij}= ∑_{(x∈ users)}[cosSim(x,i)*R_{xj} ]$$

For item-item collaborative recommendation $r_{ij}$ is defined as

$$r_{ij}= ∑_{(x ∈ items)}[R_{ix}*cosSim(x,j) ]$$

The matrix Γ can thus be defined as:

for user-user filtering: $Γ= S_U * R $
for user-user filtering $Γ= R * S_I$

In which scenarios does the equation Γ holds true. Assuming that we need to pick the k items with highest $r_{ij}$ in order to recommend the top-k items for a user i


texts: Hello I am using keras to develop a neural network model and I have a data of 45 numerical predictor variables, 2 categorical targets that will be predicted each with a different model. As I found, there is no feature importance model in keras. There are three options I can use, correlation ratio between the variables, kendals rank coefficient values and lasso regulation. Which one do you think is suitable to be used for feature selection in neural networks?


texts: Why my neural network does not predict decimal values in range [-1,1]? When it is able to predict the integer values

texts:
I am trying to perform a very simple experiment, predict the input number. The concept is same as an auto-encoder. But with just one layer, which can handle the task of encoding and decoding-

Also, wanted to observe how the network learns, with different training examples.

Initially, I took 10000 training samples, of integers, in range[1, 10000]. So, example if I pass 767676 as one of the test sample it should predict the output as 767676.0
This test passed very well.
The activation function used here was 'softplus', which keeps the values in range [0, inf).
the network -

train_d = numpy.array([i for i in range(10000)], dtype=np.int)

model = Sequential()
model.add(Dense(1, input_shape=(1, ), activation='softplus'))

model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)

Now when I gave it 10000 training sample of decimal values in the range[-1, 1].  E.g. 0.3456 the expected output will be 0.3456 but rather, its giving me 0.5721.
The network -

train_d = numpy.array[round(random.uniform(-1, 1), 4) for i in range(10000)], dtype=np.float)

model = Sequential()
model.add(Dense(1, input_shape=(1,), activation='softsign'))
model.fit([train_d], train_d, epochs=5000, batch_size=256, shuffle=True)


texts: Metrics for each class in a multi-class training/testing dataset

texts:
I am working with the NSL_KDD Dataset for cyber security and training a number of different models. There are five different classes: benign, dos, probe, r2u, and u2l.
(I am using scikitlearn.)
I would like to calculate accuracy for each class, as opposed to the accuracy of the entire classifier.
Can someone please check my understanding?
Accuracy = (TP+TN)/(n-samples)
In a multi-class classifier, is TN the sum of all TPs for the other classes?
E.g.
   A   B   C  D    E
A [10, 0,  0, 0,   1
B  0, 15,  0, 0,   0
C  0,  0, 20, 0,   0
D  0,  0,  0, 10,  0
E  0,  0   0  0,  20]

When calculating accuracy for class B the parameters would be:
TP = 15
TN = (10+20+10+20)
n-samples = 76
Accuracy for class B would be: (15+60)/76 = 0.99%
Is this correct?


texts: I am looking for domain-specific data science programs (as a major, not a minor or specialization). I found programs in bio-statistics (and other disciplines in public health), MS in marketing analytics, MS in learning analytics and MS in urban informatics. Does anyone know any other domain-specific data science field of study?


texts: How would you improve and optimize a manual encoding like this?

texts:
df.features is like a list of features in each row.
First I set all those encoded to 0, and then did this:

for index, row in df.features.iteritems():
if type(row) == str:
    if 'Academia' in row:
        df.academia.iloc[index] = 1
    if 'Churrasqueira' in row:
        df.churrasqueira.iloc[index] = 1
    if 'Cinema' in row:
        df.cinema.iloc[index] = 1
    if 'Espaço gourmet' in row:
        df.espaco_gourmet.iloc[index] = 1
    if 'Espaço verde / Parque' in row:
        df.parque.iloc[index] = 1
    if 'Gramado' in row:
        df.gramado.iloc[index] = 1
    if 'Jardim' in row:
        df.jardim.iloc[index] = 1
    if 'Piscina' in row:
        df.piscina.iloc[index] = 1
    if 'Playground' in row:
        df.playground.iloc[index] = 1
    if 'Quadra de tênis' in row:
        df.quadra_tenis.iloc[index] = 1
    if 'Quadra poliesportiva' in row:
        df.quadra_poli.iloc[index] = 1
    if 'Quintal' in row:
        df.quintal.iloc[index] = 1
    if 'Salão de festas' in row:
        df.salao_festas.iloc[index] = 1
    if 'Salão de jogos' in row:
        df.salao_jogos.iloc[index] = 1
    if 'Aquecimento' in row:
        df.aquecimento.iloc[index] = 1
    if 'Ar condicionado' in row:
        df.ar_condicionado.iloc[index] = 1
    if 'Conexão à internet' in row:
        df.internet.iloc[index] = 1
    if 'Depósito' in row:
        df.deposito.iloc[index] = 1
    if 'Elevador' in row:
        df.elevador.iloc[index] = 1
    if 'Garagem' in row:
        df.garagem.iloc[index] = 1
    if 'Lareira' in row:
        df.lareira.iloc[index] = 1
    if 'Lavanderia' in row:
        df.lavanderia.iloc[index] = 1
    if 'Massagem' in row:
        df.massagem.iloc[index] = 1
    if 'Mobiliado' in row:
        df.mobiliado.iloc[index] = 1
    if 'Recepção' in row:
        df.recepcao.iloc[index] = 1
    if 'Sauna' in row:
        df.sauna.iloc[index] = 1
    if 'Spa' in row:
        df.spa.iloc[index] = 1
    if 'TV a cabo' in row:
        df.tv_cabo.iloc[index] = 1
    if 'Circuito de segurança' in row:
        df.circuito_seguranca.iloc[index] = 1
    if 'Condomínio fechado' in row:
        df.condominio_fechado.iloc[index] = 1
    if 'Interfone' in row:
        df.interfone.iloc[index] = 1
    if 'Segurança 24h' in row:
        df.seguranca_24h.iloc[index] = 1
    if 'Sistema de alarme' in row:
        df.alarme.iloc[index] = 1
    if 'Vigia' in row:
        df.vigia.iloc[index] = 1
    if 'Área de serviço' in row:
        df.area_servico.iloc[index] = 1
    if 'Cozinha' in row:
        df.cozinha.iloc[index] = 1
    if 'Escritório' in row:
        df.escritorio.iloc[index] = 1
    if 'Sacada' in row:
        df.sacada.iloc[index] = 1
    if 'Varanda gourmet' in row:
        df.varanda_gourmet.iloc[index] = 1


texts: What does it mean to take the "average" of two decision trees by 'voting'

texts:
I have heard, in relation to random forest algorithm, that the algorithm will fit many decision trees and take the average of them by votes. (This is related to bagging as well)

I understand what the average means for something example such as $\vec{x}=[1,2,3], \; \bar{x} =2 $. But I don't know what it would mean if I had two decision trees.

Could anyone please provide a simple example / explanation of this averaging process for a couple of decision trees?


texts: Keras EarlyStopping callback: Why would I ever set restore_best_weights=False?

texts:
The point of EarlyStopping is to stop training at a point where validation loss (or some other metric) does not improve.
If I have set EarlyStopping(patience=10, restore_best_weights=False), Keras will return the model trained for 10 extra epochs after val_loss reached a minimum. Why would I ever want this? Has this model not just trained for 10 unnecessary epochs? Wouldn't it make more sense to give me back the model that was trained at the lowest validation loss i.e. with restore_best_weights=True?
Would love to hear situations where doing those extra 10 epochs of training is better than not doing them.


texts: k-means and LDA for text classification: how to test accuracy?

texts:
I have many tweets that I would like classify based on their similarity.
Unfortunately I am not quite familiar with text-classification and nlp, so I had to read a lot of documents before having an idea on the topic.
My tweets have no labels so I cannot classify them: only manually, but it would be time consuming.
I would like to group them by topic, so i have first considered LDA for topic classification, then k-means clustering.
Is it a good approach to proceed?
What are the differences and how I can test the accuracy of the classification?


texts: How to make a hybrid ARIMA and SVMs model in R

texts:
I want to combine 2 awesome models for data prediction / forecast - an ARIMA and an SVMs model, and thus I want to reduce standard error for the hybrid model. Currently, here's a graph showing them in action:

I don't exactly know how to combine both models together. My way is to fit the result of the ARIMA prediction to an SVM classifier. However, my method produces a higher error in all 4 categories (MAE, MAPE, MSE and RMSE):

row 1 - ARIMA fit

row 2 - SVMs fit

row 3 - ARIMA + SVMs combination (my method)

This is my source code:

# Time series data
data_ts <- ts(daily_data[, c('cnt')])

# ARIMA model
arima_seasonal <-
    Arima(
    data_ts,
    order = c(2, 1, 2),
    seasonal = c(1, 0, 1),
    include.drift = TRUE
    )
arima_seasonal_pred <- fitted(arima_seasonal)

# SVM model
svm_tune <- tune.svm(
    data_ts ~ daily_data$date,
     data = daily_data,
     gamma = 2 ^ (-1:1),
     cost = 2 ^ (2:8),
     epsilon = seq(0, 1, 0.1),
     kernel = "radial"
 )
 # - best parameters:
 # gamma cost epsilon
 # 2     256  0.4
 svm_model <- svm_tune$best.model

svm_pred <- fitted(svm_model)

# ARIMA + SVM
arima_svm_tune <- tune.svm(
    arima_seasonal_pred ~ daily_data$date,
     data = daily_data,
     gamma = 2 ^ (-1:1),
     cost = 2 ^ (2:8),
     epsilon = seq(0, 1, 0.1),
     kernel = "radial"
 )
 arima_svm <- arima_svm_tune$best.model
arima_svm_pred <- fitted(arima_svm)

label1 <- "Actual data"
label2 <- "ARIMA model"
label3 <- "SVMs"
label4 <- "Hybrid model"

# Plot them
graph <- ggplot(daily_data) +
    geom_line(aes(x = date, y = cnt, colour = label1), size = .4) +
    geom_line(aes(x = date, y = arima_seasonal_pred, colour = label2)) +
    geom_point(aes(x = date, y = arima_seasonal_pred, colour = label2), size = 0.5) +
    geom_line(aes(x = date, y = svm_pred, colour = label3), size = .4) +
    geom_point(aes(x = date, y = svm_pred, colour = label3), size = 0.5) +
    geom_line(aes(x = date, y = arima_svm_pred, colour = label4), size = .4) +
    geom_point(aes(x = date, y = arima_svm_pred, colour = label4), size = 0.5) +
    scale_x_date("Date") +
    ylab("Bicycle Sales")
graph$labels$colour <- "Legend"

ggsave(
    filename = "arima_svm_pred_1.png",
    plot = graph,
    width = 10,
    height = 6
)

Any help will be greatly appreciated.


texts:
Say you have a binary classification problem, and a dataset with 20,000 observations and 20 columns. The target variable is very imbalanced, there are are missing values, skewed distributions, outliers, etc.

My question is, in a general sense, what order should these data preprocessing steps be performed?

Fill in missing values,
Normalize/standardize data,
Deal with skewness,
Deal with outliers,
Balance target variable classes


texts: Implemented early stopping but came across the error SGDClassifier: Not fitted error in sklearn

texts:
Below is the simpler implementation of early stopping which i came across the book and wanted to try it.

# Implement SGD Classifier

sgd_clf =   SGDClassifier(random_state=42,
                          warm_start=True,
                          n_iter=1,
                          learning_rate='constant',
                          eta0=0.0005)

minimum_val_error = float('inf')
best_epoch = None
best_model = None

for epoch in range(1000):
  sgd_clf.fit(X_train_scaled,y_train)
  predictions = sgd_clf.predict(X_val_scaled)
  error = mean_squared_error(y_val,predictions)
  if error < minimum_val_error:
    minimum_val_error = error
    best_epoch = epoch
    best_model = clone(sgd_clf)

Once the above snippet is executed, best model and best epoch are stored in variable best_model and best_epoch.So, to test the best_model, i ran the below statement.

y_test_predictions = best_model.predict(X_test)

But then i came across the error This SGDClassifier instance is not fitted yet

Any hints on how to solve this, would be greatly helpful. Thanks


texts:
Beginner's question regarding sequences in neural networks: suppose I have classification problem that looks like:
X = very long sequence of varying length.
Y = class (assume for simplicity y=0/1).
The class depends on the whole sequence - it is impossible to determine the class without seeing all of it.
How NN handle this? I have read various methods, such as truncating the sequence (which is a problem - there is no class for truncated part of the sequence, so I cannot label it), using encoder/decoder lstm (again, couldn't completely understand how it solves the problem), etc.


texts: How to determine parameters for diminishing return function in a market mix model

texts: I am working on a market mix model ad was trying to apply diminishing returns to GRP values for my media activities. I was confused on how to go about it ,i.e how to decide the parameters for the for the diminishing fn ex negative exponential function y = α⋅(1 − e−β⋅x) ; β > 0 has 2 parameters alpha & Beta, there are any number of combinations , what is the approach to finding a reasonable estimate for these values then.


texts: The best way to calculate variations between 2 datasets?

texts:
I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database, it will ask the DBA to tune the database. My tracing data looks like this.

The images show the query execution count from 2 different months. According to the images there is a considerable pattern chnage in query execution. Query2 is executed most of the times in the first dataset and Query3 is executed most of the time in second dataset. I want to mathamatically calcuate this pattern difference. One solution I can think of is calculating the percentage differene of execution count of each query and sum it all. If the value is greater than the predefined thresold I can send a notification to the DBA. Is there a better way to do that? Any data analysis technique that can be applied? Any libraries to do that?


texts: More weightage to a categorical feature for an Autoencoder model

texts:
I am using autoencoder for anomaly detection. I don't have any labels already and so its unsupervised. If I have categorical variables, I usually one hot encode before giving it to the model. I would like to know if we there is a possibility to give more weights to one particular categorical feature before giving it to the model.

Any help is much appreciated!!


texts: How to realize a 4 dimensional time series data classification

texts:
Assuming I am doing a speech binary classification.
An audio file contains a different number of sentences.
I am splitting the audio files by their sentences with padding to a fixed number of sentences.
So I get (batch, sentence_number, timestep, features)
For example, (10,5,20,34)
I have 10 audio files, I split each audio file into 5 sentences. If the audio contains less than 5 sentences, pad it to 5 with 0's. So now I have 5 sub-audio files, I extract the acoustic features from the 5 sub-audio files using a timestep of 20 and 34 features for each timestep. I get (5,20,34) data.
My intuition is that audio may have sub-audio containing different classes (0 or 1). I would like to make a prediction for that audio based on the SoftMax of the prediction of its sub-audios.
For example, Audio 1 has 3 of its sub-audios predicted as 0, and 2 predicted as 1. So Audio 1 will be classified as 0 since 3>2.
How shall make this model?
My idea using Tensorflow is something like below:
input1 = Input(shape=(20,34)) #input the sub-audios first
x = Masking(mask_value=0.0)(input1)
x = MultiHead(Bidirectional(LSTM(units=lstm_unit)), layer_num=10)(x) # Apply a multihead-attention or self-attention
x = Flatten()(x)
sub_prediction = Dense(2, activation='softmax')(x) # 5 binary prediction for the sub-audios,softmax because it may be more than 2 classes, for understanding purpose I used an example of 2 classes.

I am not sure how to proceed from here.

How to utilize the model to make a prediction of the audio from the predictions of its 5 sub-audios.
The final expected output is a softmax y. In this case, it is y_pred.shape =(10,2)


texts: How to read data in DSX from Data hub object storage container?

texts:
I am creating a new notebook with DSX. I have a data set that I want to use. It is not CSV. I found that I could go to Data hub by clicking top-right tile near my logged in name, then object storage in dropdown and finally upload a text file.

How do I read this file in my notebook? Do I need to create a connection ?


texts: Memory error on using data generator in keras

texts:
I am using the following augmentations on dataset of size 9 GB:

datagen = ImageDataGenerator(
    featurewise_center=False,  # set input mean to 0 over the dataset
    samplewise_center=False,  # set each sample mean to 0
    featurewise_std_normalization=False,  # divide inputs by std of the dataset
    samplewise_std_normalization=False,  # divide each input by its std
    zca_whitening=True,  # apply ZCA whitening
    rotation_range=30,  # randomly rotate images in the range (degrees, 0 to 180)
    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
    horizontal_flip=True,  # randomly flip images
    vertical_flip=True)  # randomly flip images
datagen.fit(a)

model.fit_generator(datagen.flow(a,b, batch_size=32),
                    steps_per_epoch=len(a) / 32, epochs=epochs, class_weight = sclass_weight, validation_data = [c, d],callbacks = [MetricsCheckpoint('logs')])

When the code comes to the datagen.fit, I get into memory error ( the code doesn't even go into training)

I have 50 gb ram and am training it on a K80 with a batch size of 32, so don't think that will be a problem.

It works fine when I comment all the augmentations.

Can someone tell me where I am going wrong?


texts: Are there any rules for choosing the size of a mini-batch?

texts:
When training neural networks, one hyperparameter is the size of a minibatch. Common choices are 32, 64, and 128 elements per mini batch.
Are there any rules/guidelines on how big a mini-batch should be? Or any publications which investigate the effect on the training?


texts: Gaussian Mixture Models as a classifier?

texts:
I'm learning the GMM clustering algorithm. I don't understand how it can used as a classifier. Here are my thought:

1) GMM is an unsupervised ML algorithm. At least that's how sklearn categorizes it.

2) Unsupervised methods can cluster data, but can't make predictions.

However, sklearn's user guide clearly applid GMM as a classifier to the iris dataset.

If I have to guess, maybe after clustering, each cluster is assigned to a class label based on some kind of majority voting. However, I can't find any documentation. Could someone shed more light on this process from unsupervised to supervised learning?

A related question: when using GMM as a classifier, is it common practice to simply make n_components=n_classes, instead of checking AIC, BIC, etc.?


texts: How to control number of branches for each split in sklearn DT classifier?

texts: I am trying to code a two class classification DT problem that I used SAS EM before. But trying to do it in Sklearn. The target variable is a two class categorical variable. But there are a few continuous independent variables. In SAS I could specify the "Maximum Number of Branches" for each split. So when it is set to 4, some leaf will split into 2 and some in 4 (especially for continuous variables). I could not find an equivalent parameter in sklearn. Looked at "max_leaf-nodes". But that controls the total number of "leaf" nodes of the entire tree. I am sure some of you probably has faced the same situation and already found a solution. Please help/share. I will really appreciate it.


texts: PyTorch Hugging Face - Language generation with torchscript model

texts:
Have a fine-tuned a summarization model following the Hugging Face seq2seq guide (starting from sshleifer/distilbart-xsum-12-6).
We are interested in using AWS elastic inference for deployment for cost reduction. (e.g. similar to this)
Wondering whether there’s any examples or any suggested way to use the beam searching logic in BartForConditionalGeneration with model inference from a torchscript model. Most of the examples for torchscript found are with classification tasks where this isn’t necessary.


texts: Image masking tool to implement mask RCNN using python

texts:
I am trying to understand Mask RCNN.  For that I have to input image with mask in png format while building the model.

I try to follow the article present in this blog.

The blogger used Pixel Annotation Tool.

I tried to follow her steps.  I downloaded all the requirements for this tool.  Like QT, Open CV, CMAKE and VS 2015 +

When I try to update the build script as mentioned here for windows.  I am unable to find the make OpenCV build folder. I am in windows 10 with Python 3.7.

I also tried Lear Tool.  It didn't help me.

I tried VGG annotation tool.  I am able to create bounding boxes but unable to create mask in png format.

Can anyone suggest a tool for this activity.

I have posted the same in Stackoverflow. I thought of posting it here since its more relevant. I am deleting the question from stackoverflow.


texts: What does the classification report interpret? Class 1 indicates abnormal data

texts: How to interpret the report and How is precision, recall values are calculated for individual class labels. What is the significance of macro avg ? Does this report signify a good predictions by the model?


texts: Cat2Vec embedding a categorical value columns with respect to multiple y's

texts:
I try to do some embeddings on categorical columns using Keras. Here is the code:

embedding_size = 7
model = k.models.Sequential()
model.add(k.layers.Embedding(input_dim = 13, output_dim = embedding_size, input_length = 1, name="embedding"))
model.add(Flatten())
model.add(Dense(50, activation="relu"))
model.add(Dense(15, activation="relu"))
model.add(Dense(1))
model.compile(loss = "mse", optimizer = "adam", metrics=["accuracy"])
model.fit(x = df['Release Cause'].as_matrix(), y=df['Jitter'].as_matrix() , epochs = 50, batch_size = 4)

Here y is only one column of the dataset. I want to do embedding with respect to multiple columns. Is it possible and how to achieve it.


texts: What is Hellinger Distance and when to use it?

texts: I am interested in knowing what really happens in  Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use  Hellinger Distance? What are the benefits of using  Hellinger Distance?


texts: Base64 Encoded images in Grpc Client request for tf serving

texts:
I have been succeed to deploy my model on tensorflow serving.

My model takes in a base64 string as input and predicts some output.
I was able to test it using the following restApi

def http_request(img):
    image_as_txt = encode_image(img)
    payload = {
        "signature_name": "predict",
        "inputs":
            {
                "image": {"b64": image_as_txt}
            }
    }
    headers = {
        'Content-Type': "application/json"
    }

    response = requests.request("POST", 'http://10.10.10.2:8501/v1/models/my_model:predict", data=json.dumps(payload),
                            headers=headers)
    print(json.loads(response.text))

The above code is working fine till now

now I am trying to do the same using a gRPC client request for which I am using the below code

channel = grpc.insecure_channel('10.10.10.2:8500')

stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

request = predict_pb2.PredictRequest()
request.model_spec.name = 'my_model'
request.model_spec.signature_name = 'predict'

image_as_txt = //Some base64 image//
request.inputs['image'].CopyFrom(make_tensor_proto(image_as_txt),shape=())
res = result_future.result()
print(result_future.result())

but getting the following error

 [Unable to decode bytes as JPEG, PNG, GIF, or BMP]  [[{{node decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert}}]]","grpc_status":3}"

can anyone help me to send base64 images to my model


texts: Why SVM classifier error is much smaller than KNN classifier error?

texts:
I have a dataset with a binary classification, on which I train a kNN algorithm and a SVM algorithm.

On the tests set I get approximately 75% error when using kNN, but only approximately a 20% error using an SVM.

What does this tell me about the dataset?


texts: Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error

texts:
We want to use cosine similarity with hierarchical clustering and we have cosine similarities already calculated.
In the sklearn.cluster.AgglomerativeClustering documentation it says:

  A distance matrix (instead of a similarity matrix) is needed as input
  for the fit method.

So, we converted cosine similarities to distances as

distance = 1 - similarity

Our python code produces error at the fit() method at the end. (I am not writing the real value of X in the code, since it is very big.) X is just a cosine similarity matrix with values converted to distance as written above. Notice the diagonal, it is all 0.) Here is the code:

import pandas as pd
import numpy as np
from sklearn.cluster import AgglomerativeClustering

X = np.array([[0,0.3,0.4],[0.3,0,0.7],[0.4,0.7,0]])

cluster = AgglomerativeClustering(affinity='precomputed')
cluster.fit(X)

The error is:

runfile('/Users/stackoverflowuser/Desktop/4.2/Pr/untitled0.py', wdir='/Users/stackoverflowuser/Desktop/4.2/Pr')
Traceback (most recent call last):

  File "<ipython-input-1-b8b98765b168>", line 1, in <module>
    runfile('/Users/stackoverflowuser/Desktop/4.2/Pr/untitled0.py', wdir='/Users/stackoverflowuser/Desktop/4.2/Pr')

  File "/anaconda2/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 704, in runfile
    execfile(filename, namespace)

  File "/anaconda2/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 100, in execfile
    builtins.execfile(filename, *where)

  File "/Users/stackoverflowuser/Desktop/4.2/Pr/untitled0.py", line 84, in <module>
    cluster.fit(X)

  File "/anaconda2/lib/python2.7/site-packages/sklearn/cluster/hierarchical.py", line 795, in fit
    (self.affinity, ))

ValueError: precomputed was provided as affinity. Ward can only work with euclidean distances.

Is there anything that I can provide? Thanks already.


texts: Is it good practice to convert columns with a number to a range between 0 and 1?

texts:
Relatively new to data science. I heard something about converting columns which contain integers into a range between 0 and 1. I think the reasoning was that so all the columns will be more similar in their range. I think along with that there might have been a step of removing outliers (very high integers) to that they wouldn't cause all other results be skewed as a low fraction.

Is this accurate?

If yes, is there an easy command to make it happen with a Pandas dataset?


texts: Trained CNN individually on multiple images to classify them, how can I now classify a related "set" of these images that correspond to one object?

texts:
I have a N object classification examples, each example consisting of a set M individual images of the object at different angles. I've trained M CNNs with the dataset of one particular image angle each and their corresponding label. (Thus I have M sets of model parameters I've discovered for each angle)

Now given this information, what is a good approach to classifying a new single object based on it's set of M individual image angles? (i.e I can classify with some decent accuracy what each individual image is in the set, but is there a method I can use the feature-dependency of images that belong to the same unique example set to make an overall "better" prediction than just taking the average of all the individual probabilities?)


texts: I have discrete values in the target variable(Exactly 13 different values in total) . When I am giving that as input to Random forest Classifier ,it gives error that input as continuous. And if I give it to regressor it is predicting a value between the discrete values. How can I treat this problem


texts: Different optimizers for generator and discriminator in GAN

texts: I've seen an advice about GAN implementation, that there should be different optimizers for generator (G) and discriminator (D). As I understand, it depends on how fast each model (G and D) convergences. I'd like to clarify this question. Why is it possible to say, that, for example, GSD has to be chosen for D and Adam for G only and not vice verse? (Intuitive or mathematical explanations are welcomed both)


texts: splitting mechanism with one hot encoded variables (tree based/boosting)

texts:
I am using xgboost and have a categorical unordered feature with 25 levels. So when i apply one hot encoding i have 25 columns. This introduces alot of sparsity. Even more unusual, my feature importance report shows 5 of these one hot encoded column in top 10, with one of them appearing at the top.
I tried to see if there was a diffrence in percentage of these categories between my binary classes (1, 0) but there isn't so i am a little perplexed as to why it is assigning such a high feature importance to them.
i have read online that if we have a categorical variable with q levels, the tree has to choose from ((2^q/2)-1) splits. For a dummy variable, there is only one possible split and this induces sparsity
i am not sure i understand this, say i have a column called color: red, green, blue,yellow , and i implement one hot encoding so is the number of splits that happen is 2^4/2 -1 = 3? if this increases as i have e.g. 2^25/2 -1, more splits means the tree is more likely to find a 'good split' for data at hand and lead to overfitting? But what i don't understand is how this splitting chages with dummy variables.. does that equation hold or not for one hot endoded variables.
am i interpreting this correctly?
sources elemts of statisicatl learning:
https://towardsdatascience.com/one-hot-encoding-is-making-your-tree-based-ensembles-worse-heres-why-d64b282b5769#:~:text=For%20every%20tree%2Dbased%20algorithm,a%20feature%20and%20a%20value.&text=The%20trees%20generally%20tend%20to,values%20(0%20or%201).


texts: How to optimally train deep learning model using output as new input

texts:
I'm trying to train a network to predict the future. My current setup uses 5 time steps as inputs from the past, each consisting of 10 features, resulting in a [5, 10] input matrix (initially generated by another algorithm). Each of these time steps is a dt step further in time. The network outputs 1 output with again 10 features, a dt step in the future w.r.t the last time step from the inputs. This input replaces the oldest time step in the inputs matrix.

The network is trained using a generated input sequence as well as one correct output for that sequence. The MSE error reduces to about 1e-4 for a 100.000 samples training set.

When executing the network using the previously mentioned method. The solution blows up pretty quickly as the network makes an error in the first prediction and uses that as an input, removing a correct but older input. Now, a bigger error is made due a partially incorrect input sequence which it has never seen before. The next time step this becomes even worse and so on.

Instead of training with a completely correct input sequence and its correct output, i was thinking the network should train on its own predicted, not completely correct, inputs as well. I could only think of training the network as mentioned before. Use that to create more samples including its own prediction errors and train it again using that training set. That process could be repeated until a desired point.

Is there a more efficient way to do that training only on a single training set using Keras as my deep learning toolbox?

Edit

I will provide some more context for my problem. I'm trying to predict the future states for a few types (square, triangle and sawtooth) of travelling waves, moving with a certain speed to the left or right. The following figure gives an example of a random travelling wave I could be using during training.

These waves are discretized by 1024 x positions with a y value. Due to their relative high dimensionality, I use an autoencoder to create a latent space with a dimensionality of 10. The following figure gives an example of an autoencoded wave, including the input, encoded latent space and the decoded output.

The lower dimensional latent space can be advanced in time more easily for an RNN network. To do this, I create five time steps of the wave using a generation algorithm. These are encoded to their latent space using the encoder of the autoencoder. Now they are fed into an RNN which predicts the next time step of the latent space. This output is fed back to the inputs to replace the oldest time step input. The output can also be decoded using the decoder of the autoencoder, resulting in a high dimensional prediction of the next time step of the traveling wave.

I have trained the RNN network giving it examples of 5 latent inputs and the correct next time step latent space output, reaching an mse of about 1.5e-4. A result for a new prediction of this RNN can be seen in the following figure.

The first prediction are pretty accurate for most of the inputs I give the network. However, when I use its own outputs as new inputs to keep on predicting next time steps, the error increases. The next figure gives an example of the same wave depicted before but now at prediction step 20.

For the RNN i'm currently using the following Keras setup:

inputLayer = Input(shape = (RNN_numInputs, AE_latentSize))
x = CuDNNLSTM(RNN_units[0], return_sequences=False)(inputLayer)
x = Dropout(0.2)(x)
x = RepeatVector(RNN_numOutputs)(x)
x = CuDNNLSTM(RNN_units[1], return_sequences=True)(x)
x = Dropout(0.2)(x)
x = Conv1D(AE_latentSize, RNN_outputKernelSize, activation='linear', padding = 'same')(x)

rnn = Model(inputLayer, x)

Where RNN_numInputs is 5, AE_latentSize is 10, RNN_units is [200, 400] and RNN_outputKernelSize is 3.

I think that when I can decrease the error of the prediction even further, the long term prediction results can become better.

Does anyone have an idea how I can increase the perfomance of this algorithm?