texts: Input sequence ordering for LSTM network

texts:
When training a LSTM network with time series data, I guess the order in which this data is fed matters, my question is how should this ordering be...

Let's take a time-series vector which will be the input for the LSTM:

\[ X = [x_0, x_{\small(-1\small)}, x_{\small(-2\small)},\ldots, x_{\small(-N\small)}]\]

with negative indexing indicating past values.

Which vector should be fed (theoretical case, not related to any API)?:

\[ X = [x_0, x_{\small(-1\small)}, x_{\small(-2\small)},\ldots, x_{\small(-N\small)}]\]

or

\[ X^r = [x_{\small(-N\small)},\ldots, x_{\small(-2\small)}, x_{\small(-1\small)}, x_0]\]

More precisely, what of these two orderings should be used in TensorFlow LSTM-Cells?


texts: Difference between a target and a label in machine learning

texts:
If I have a supervised learning system (for example for the MNIST dataset) I have features (pixel values of MNIST data) and labels (correct digit-value).

However sometimes people use the word target (instead of label).

Are target and label interchangeable? Is label just used for classification? Target both for classification and regression?


texts:
Data-set has 3 features. The number of clusters are two.

I am figuring out how to print clusters using scatter plot for the data having 3 feature column and clustered into 2 clusters using kmeans.

The train data is in dataframe format and data is of activity data set:

X =  pd.concat([train_data['start'], train_data['end'], train_data  ['duration']], axis=1)
kmeans.fit(X)
Y_pred = kmeans.predict(X)
plt.scatter(X.iloc[:,0], X.iloc[:,1], c=Y_pred, cmap=plt.cm.Paired)
plt.legend()
plt.title('train data')
plt.show()
getting following output:


texts: What does normalizing and mean centering data do?

texts:
Are there any concerns to normalizing data to be within the range 0 - 1 and mean centering the data as well?

Does it matter which comes first?

If you do one, is the other not required?


texts: Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory?

texts:
Consider a very large data set that doesn't fit into memory.  Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?

This code is just hypothetical to illustrate my question.

# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)

# compared to
for x_bat, y_bat in stream_next_batch_from_file():
    x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
    # Append the splits to their respective files.
    append_data(x_train, y_train, "train_set_filename")
    append_data(x_test, y_test, "test_set_filename")
    # etc.


texts:
Having read a lot about class imbalance in classification tasks, I'd like to know what is the methodology for data imbalance in regression tasks.

Particularly,
- What is the procedure to check for it (for example in classification, we'd check if all classes have the same training examples)
- What are possible ways to solve it?


texts:
https://github.com/google-research/bert/blob/master/modeling.py#L491-L520

The code of BERT is one of the implementation. But it is not what I need.

I search a lot but can not judge.

But where is the code which seems uses cos and sin?


texts: GD for logistic regression isn't stable. Why?

texts:
Here's my (incomplete) implementation for linear regression using GD:

from enum import Enum
import numpy as np
from math import exp

class GDType(Enum):
  GD = 1
  SGD = 2

class LogisticRegression:
  def __init__(self, gd_type=GDType.GD, learning_rate=0.1, num_iterations=1000):
    self.gd_type = gd_type
    self.learning_rate = learning_rate
    self.num_iterations = num_iterations

  def __odds(self, x, b):
    return exp(b.dot(x)) / float(exp(1 + b.dot(x)))

  def __gd_step(self, X, y):

    grad = np.zeros(self.p)

    for j in range(self.p):
      for i in range(self.m):
        grad[j] += self.__odds(X[i], self.b) - y[i]
      grad[j] *= X[i, j]

    return grad

  def __gd(self, X, y):
    for i in range(self.num_iterations):
      dl = self.__gd_step(X, y)
      self.b -= (self.learning_rate / float(self.m)) * dl
      print(self.b)

  def fit(self, X, y):
    X = np.column_stack((np.ones(X.shape[0]).T, X)) # 1's column for the bias
    self.m = X.shape[0]
    self.p = X.shape[1]
    self.b = np.zeros(self.p)
    self.__gd(X, y)
    print(self.b)

I'm testing it on the iris data, as follows:

from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data[:, :2]
y = (iris.target != 0) * 1

lr = LogisticRegression()
lr.fit(X, y)

The problem is that the coefficients are getting larger and larger instead of converging. Why?

I doubled checked (correct me if wrong) and it seems to me that __gd_step is correct.


texts:
So, I'm just starting to learn how a neural network can operate to recognize patterns and categorize inputs, and I've seen how an artificial neural network can parse image data and categorize the images (demo with convnetjs), and the key there is to downsample the image and each pixel stimulates one input neuron into the network.

However, I'm trying to wrap my head around if this is possible to be done with string inputs? The use-case I've got is a "recommendation engine" for movies a user has watched. Movies have lots of string data (title, plot, tags), and I could imagine "downsampling" the text down to a few key words that describe that movie, but even if I parse out the top five words that describe this movie, I think I'd need input neurons for every english word in order to compare a set of movies? I could limit the input neurons just to the words used in the set, but then could it grow/learn by adding new movies (user watches a new movie, with new words)? Most of the libraries I've seen don't allow adding new neurons after the system has been trained?

Is there a standard way to map string/word/character data to inputs into a neural network? Or is a neural network really not the right tool for the job of parsing string data like this (what's a better tool for pattern-matching in string data)?


texts:
I know this is probably a really easy fix, but I've been stuck on this for a while. I'm a noobie to both machine learning and programming in general.  My data is from this dataset. https://www.kaggle.com/ozlerhakan/spam-or-not-spam-dataset
I made a model that can predict whether or not an email is spam email, and it does so reasonably well, getting an 84% accuracy with the validation data. But when I try to make it predict on a new string, it seems to be predicting on a character-by-character basis. Below is the code that I used to preprocess the training(and testing) data. I would've copy-pasted the required code, but when I look at the draft, it looks terrible. So here's the link to my notebook. https://colab.research.google.com/drive/1EnoNFR5CFi85hOrW8ihx66QWb6sToKAL?usp=sharing
I tried to debug the function by putting various print statements all over. I've come to the conclusion that the problem spot is when I try to one-hot encode the words. I need those words to be one-hot encoded because that is how I trained the model.  But I don't know how to fix that. I would really appreciate some help with this. Thank you in advance.


texts: Why my Decision Tree Model is not further bifurcated in to Age

texts:
I've created a Decision Tree Model for expected candidates that buyed Villa.

Though the Drilling down Green highlighted box, show data has Age in it.

Question- But why my model is not bifurcating it further into Age?

Deposit & Age both are having content type as Discretized as DicretizationBucketCount as 5 & 10 respectively.

Source Data Link


texts: Normalizing variables with logarithmic shape

texts:
A simple model with two variables [A,B] to train, let's say, a logistic regression or any other classification model:

A: Flat distribution from 0 to 100.
B: A logarithmic distribution from 0
to a few thousands.

What would be the proper way to normalize this? Should I make B flat before? Do I put a limit before the max in B and consider all points above like the max?
I read you carefully. Thanks in advance.


texts: Classification problem with many images per instance

texts:
I am working in the following kind of classification problem: I have to classify every instance as class A or class B using many images of the instance. That is, every training example has not one image (which is the usual thing in image classification), but many images, and the number of images for every training instance is not fixed. That is, instance 1 can have 3 images, and according to these images we have to classify it as A or B, and instance 2 can have instead 5 images.

As any machine learning problem, I am provided with many labelled images and I have to build a classifier.

Although ideas are also welcome, I am looking for a documented way to attack this kind of problem (Kaggles, papers or books, mainly).

My main idea was the following: train a model $f$ that given one image gives a probability of that image being of class A. Then, for every training instance, evaluate $f$ in every image of the instance and compute statistics (aggregate) of the distribution of these probabilities, as the mean, median, maximum and minimum. Then, train a model $g$ that has as inputs these aggregates and use the composition of $f$, aggregates and $g$ as the final model. This idea is a bit simple so I am looking for something better.


texts: Residual Neural Network with 2D input data in tensorflow

texts:
I have a simple feedforward network to approximate a function $f(x,y)$. I'm interested in adding skip-connections to my network (i.e. ResNet). In 1D, it's fairly straightforward. Here's a code snippet for a single block:

def create_model(X, weights, biases, num_layers, num_hidden):

    # first layer
    W = weights[0]
    b = biases[0]
    H = tf.nn.tanh(tf.add(tf.matmul(X, W), b))

    # interior hidden layers
    for l in range(1,num_layers-2):
        W = weights[l]
        b = biases[l]
        H = tf.tanh(tf.add(tf.matmul(H, W), b))

    # residual connection
    W = weights[num_layers-2]
    b = biases[num_layers-2]
    H = tf.add(tf.tanh(tf.add(tf.matmul(H, W), b)), X)

    # output (activation) layer, linear
    W = weights[-1]
    b = biases[-1]
    Y = tf.add(tf.matmul(H, W), b)

    return Y

The above code snippet runs due to broadcasting. In particular, we have
$$
\sigma(\sigma(\mathbf{x}W_{1} + b_{1})W_{2} + b_{2}) + \mathbf{x}
$$

for a block with two hidden layers and a skip connection. Let $n$ be the number of neurons and $W_{1} \in \mathbb{R}^{1\times n}$, $W_{2} \in \mathbb{R}^{n\times n}$, $b_{1}, b_{2} \in \mathbb{R}^{1 \times n}$. Suppose $\mathbf{x} \in \mathbb{R}^{m \times 1}$ (so $m$ is the length of the input X). We have the first term $\sigma(\sigma(\mathbf{x}W_{1} + b_{1})W_{2} + b_{2})$ is in $\mathbb{R}^{m \times n}$, and tensorflow allows broadcasting so that $\mathbf{x}$ is added to each column of the first term. Is this how the skip connection is supposed to work?

I'm interested in extending this to two-dimensional functions, again only with a single block for now:

# initialize weights
def initialize_NN(num_layers, num_hidden):
    weights = []
    biases = []

    # first layer, based on input data
    W = init_weights([2, num_hidden], 'xavier', xavier_params=(1, num_hidden))
    b = init_weights([1, num_hidden], 'zeros')
    weights.append(W)
    biases.append(b)

    # interior hidden layers
    for l in range(1,num_layers-1):
        W = init_weights([num_hidden, num_hidden], 'xavier', xavier_params=(num_hidden, num_hidden))
        b = init_weights([1, num_hidden], 'zeros')
        weights.append(W)
        biases.append(b)

    # output (activation) layer
    W = init_weights([num_hidden, 1], 'xavier', xavier_params=(num_hidden, 1))
    b = init_weights([1, 1], 'zeros')
    weights.append(W)
    biases.append(b)

    return weights, biases

# setup model
def create_model(X, Y, weights, biases, num_layers, num_hidden):

    DATA = tf.concat((X, Y), axis=1)

    # first layer
    W = weights[0]
    b = biases[0]
    U = tf.nn.tanh(tf.add(tf.matmul(DATA, W), b))

    # interior hidden layers
    for l in range(1,num_layers-2):
        W = weights[l]
        b = biases[l]
        U = tf.tanh(tf.add(tf.matmul(U, W), b))

    # residual connection
    W = weights[num_layers-2]
    b = biases[num_layers-2]
    U = tf.add(tf.tanh(tf.add(tf.matmul(U, W), b)), DATA)

    # output (activation) layer, linear
    W = weights[-1]
    b = biases[-1]
    U = tf.add(tf.matmul(U, W), b)

    return U

Of course, I get an error with the shapes now since DATA is two-dimensional. The error I get is

ValueError: Dimensions must be equal, but are 100 and 2 for 'Add_3' (op: 'Add') with input shapes: [?,100], [?,2].

in this line:

U = tf.add(tf.tanh(tf.add(tf.matmul(U, W), b)), DATA)

I've read that I should multiply by some appropriate transformation $W_{s}$, but I'm not sure how to determine this mapping. If I tried something like

U = tf.add(tf.tanh(tf.add(tf.matmul(U, W), b)), tf.matmul(DATA, tf.identity(weights[0]))))

then it certainly fixes the shape issue, but I'm not sure if this is correct. Can someone explain to me or suggest a good tutorial how I can rectify this problem?


texts:
I am studying Bayes probability applied to machine learning, and I have encoutered the concept of likelihood, which I don't understand.

I have seen that the Bayes rule is:

$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

where $P(B|A)P(A)$ is the conditional probability, while $P(B|A)$ is the likelihood, but I don't understand what the likelihood is.

Can somebody please help me?


texts:
In this wiki page, I came across with the following phrase.

  When data is not labeled, a supervised learning is not possible, and
  an unsupervised learning is required

I cannot figure out why supervised learning is not possible?

Appreciate any help to resolve this ambiguity.


texts: How much data are sufficient to train my machine learning model?

texts:
I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.

My colleague (who is a machine learning expert) said that, in his opinion, the arguably most important practical aspect of machine learning is how to understand whether you have collected enough data to train your machine learning model.

This statement surprised me, because I had never given that much importance to this aspect...

I then looked for more information on the internet, and I found this post on FastML.com reporting as rule of thumb that you need roughly 10 times as many data instances as there are features.

Two questions:

1 - Is this issue really particularly relevant in machine learning?

2 - Is the 10 times rule working? Are there any other relevant sources for this theme?


texts: Prediction of Remaining Useful Lifetime (RUL) of an Accelerator Beam

texts:
I have used this LSTM model with different data, and it worked fine and gave accurate predictions.
But now that I am trying to predict the RUL (Remaining Useful Lifetime) of the Accuracy of my companies Ion Beam, it is predicting complete nonsense. Even though I fed it data it already knows from training. Is there something obviously wrong with my Model or could it be that there are most likely no patterns to make sense of, in my companies dataset?
I know that this is a vague question.
Any comment is appreciated.
train_X_reshaped.shape = (1980, 1000, 30)
train_y.shape = (1980, 1000)
TIMESTEPS = 1000
SAMPLE_DIVISOR = 10
EPOCHS = 10

features = train_X_reshaped.shape[2]

# design network
model = Sequential()
model.add(LSTM(130, batch_input_shape=(SAMPLE_DIVISOR, TIMESTEPS, features), stateful=True, return_sequences=False))
model.add(Dense(TIMESTEPS))
model.compile(loss='mae', optimizer='adam', metrics=['accuracy'])
#plot_model(model, show_shapes=True, show_layer_names=True)

# fit network
for i in range(EPOCHS):
    history = model.fit(train_X_reshaped, train_y, epochs=1, batch_size=SAMPLE_DIVISOR, shuffle=False)# callbacks= [tensorboard])
    model.reset_states()

yhat = model.predict(train_X_reshaped, batch_size=SAMPLE_DIVISOR)

plotPred(train_y.flatten(), yhat.flatten())


texts: What is the difference between 8 filters twice and one 16 filters in convolutional layers at CNN?

texts: Why would we use two convolution layers in a row with same spec? E.g. in VGG16 first two layers, they used 3 x 3 filter with 64 depth twice. What is the difference if we use 3 x 3 filter with 128 depth once.


texts: Retrain Tensorflow CNN Model on additional Images

texts:
So I've created a model to classify images of traffic signs. I'm currently using this model to predict classes of images I've gotten from the internet. However, I would like to be able to re-classify or correctly label images in which the model failed to correctly predict their classes.
One way would be to place the images in their correct classes and retrain the model from the last saved checkpoint. I don't think this is a good approach as the model should overfit over time. Hence, I'd like to know if there's a better way to achieve this.


texts: Multi-class cross entropy loss with negative sampling

texts:
So I have multiple networks, each representing a transformation, followed by a multi-class cross entropy loss. For each training sample, there's one right transformation and then the right label.
My question is, if I can do negative sampling for those wrong transformation networks, by e.g make their cross entropy loss negative. The idea is that the wrong transformation shouldn't produce high score for the right label. I've tried it but the negative sampling then dominates the loss. My guess is that cross entropy loss is a log loss so it has a higher gradient near 0. So it's much easier for the model to push the probability of the negative samples to 0 than to push the probability of the positive samples to 1.
I would like to know if there're any solutions for this scenario?


texts: Choosing the size of Character Embedding for Language Generation models

texts:
I am working on a character-based Language Generator, loosely based on this tutorial on the TensorFlow 2.0 website. Following the example, I am using an Embedding() layer to generate character embeddings: I want my model to generate text character by character.

My vocabulary counts 86 unique characters. What embedding size should I choose?

Should I always choose an embedding size that is shorter than the size of vocabulary? The embedding size in the example above is much larger than the vocabulary size, I can't understand how this can build an effective model (but apparently it does, if it's an official tutorial, and if anyone can explain me why it'd be much appreciated).

EDIT:

Another thing I find puzzling is: when we generate word embeddings is because we want a dense representation of a word meaning. Does it make sense to make it larger than the actual one-hot encoded vectors we started with?


texts: What are some machine learning problems that can be attacked with continuous multiobjective optimization?

texts: I am working on continuous vector optimization, and hence continuous multiobjective optimization is a particular case. I am interested in finding applications in machine learning for this problems. Is there any algorithm/application on which you could use Pareto/weak Pareto optimal points as the optimal parameters? I was wondering whether there exists something like a multiobjective version of logistic regression or least squares. Any pointer to the literature will also be very helpful.


texts: SKLEARN Metrics report "Number of classes, 28, does not match size of target_names, 35. Try specifying the labels parameter"

texts:
What's the proper way to define the labels or target names of classification_report?
I have the report like this:
print(metrics.classification_report(twenty_test.target[:n_samples], predicted, target_names=twenty_test.target_names))

I noticed i get the error like in the title or similar once the twenty_test.target[:n_samples] (or possibly also predicted array) is different length than twenty_test.target_names...
It doesn't work just out of box if i simply provide all target names (also those not included in the "target" or "predicted" array. (If i provide it as "target_names" param).
But if I provided it as the "labels" param as hinted i simply see 0 filled report.
Btw this is the output of unique values of "target" array:
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 25}

This is the output of predicted array (unique values):
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 25, 27, 30, 31, 32}

I tried to get the longer list of these and build the list of the longest value like this:
longer = max(max(set(predicted)), max(set(twenty_test.target[:n_samples])))+1
print(metrics.classification_report(twenty_test.target[:n_samples], predicted, target_names=twenty_test.target_names[:longer])

Doesn't work. I would expect all target names also those not include should work out of box.


texts: What algorithms can be used to predict the outcome of a cricket match?

texts:
I am doing a project to predict the outcome of a cricket match, I have the data that states which matches were won by whom for ODIs. [Espn data]

Which algorithm could be used to predict the outcome of the coming matches?
Would Quadratic Regression be a good idea? Or would predicting based on probability algorithms such as the markov's algorithm is what is generally used?
Any other algorithm I should use?

So basically, I want to know which algorithm I should use, I will implement in C++ ultimately but I will do it in R or python first.

P.S.:-

I am a newbie in this field, hence pardon if the question sounds too stupid.
I have learnt regression so far in data analytics.


texts: Deep Learning to estimate what is beyond the edge

texts: I have an image of some data which is approximately 4,000 x 8,000 pixels.  I am interested in finding out if anyone has used a deep learning algorithm to predict what would be on the image if it extended 100 more pixels in each direction.  I would imagine that the data could be trained on smaller rectangles, and then the rules developed would be used to extend beyond the image given.  Has anyone seen a problem like this (and is there a reference)?   Even if not, what deep learning scheme would be best for this?


texts:
I'm training a nmt model with keras. I have kinda memory limitation so I train my model with chunks of data everytime. This is the part of my code related to callbacks:

if not os.path.isfile('./model.h5'):
    # define model
    print("Defining the model for the first time: ")
    model = define_model(answer_vocab_size, question_vocab_size, answer_length, question_length, 256)
    model.compile(optimizer='adam', loss='categorical_crossentropy')

    checkpoint = ModelCheckpoint(filename, monitor='loss', verbose=1, save_best_only=True, mode='min', save_weights_only=False)
    callbacks_list = [checkpoint]
    model.fit(trainX, trainY, epochs=15, batch_size=64, callbacks=callbacks_list)
else:
    print("Updating the previous model ....")
    model = load_model(filename)
    checkpoint = ModelCheckpoint(filename, monitor='loss', verbose=1, save_best_only=True, mode='min', save_weights_only=False)
    callbacks_list = [checkpoint]
    model.fit(trainX, trainY, epochs=15, batch_size=64, callbacks=callbacks_list)

So I have like 10 files and each file have 100000 data. My memory can only afford 10000. So when I run this, one file get picked and my model get trained with whole data in 10 stages. Actually that part of code that I pasted runs in the loop for 10 times.
So I have a little problem, When I run this code for a file, It always start the loss with a number like 7 and in the last stage, decreases to a number like 4. When a pick a new file, the loss at the beginning should be a better one, right? But the same numbers!
I think since in every run, my variable callbacks_list destroys, this can cause problems. Should I save callbacks list in a file?
Any advice would be appreciated.


texts: Python Text Classification - Data that does not fit into any category

texts:
I am having a lot of trouble finding any kind of answers to this problem i am facing.
I have a few text classifiers that i am testing out, and they work well for data that does fit into any predefined category, but if I input, lets just say "fhjakdlfsah", it will still assign it to some category because i guess the predict_proba functionality has to add up to 1 for all of the categories.
Is there something i am missing here? I am having such a hard time finding a solution to this, and I would imagine it is a very common thing to deal with. Right now i am working with gradientboosting from sklearn, and tried wrapping it in a onevsrestclassifier as suggested by others, but it is still having the same thing where all probabilities are adding up to one, and it is getting assigned the highest probability
Basically I am looking for a solution that can say either, yes this fits into one of these categories, or no, this does not fit into any of these categories.
Any help would be greatly appreciated as I am getting quite stuck here
Thanks!


texts:
I am doing multilabel news classification in python language.The dataset I have has two files. First CSV contains articles at each row. Second CSV contains corresponding labels to each article. Here is the snapshot of label file

Here is the dropbox link....https://www.dropbox.com/s/7huzh41je735oqn/labelset.csv?dl=0

Is the dataset imbalanced one?
How do I distribute this dataset properly into training set,
validation and test set?

Note: By properly I mean, Can this unbalanced dataset be split into proper proportions in training,validation and testing set?


texts:
I am trying to create a pollution prediction LSTM. I've seen an example on the web to cater for a Multivariate LSTM to predict the pollution levels for one city (Beijing), but what about more than one city?  I don't really want a separate network for every city, I'd like a single generalised model/network for all x cities. But how do I feed that data into the LSTM?

Say I have the same data for each city, do I...

1) Train on all data for one city, then the next city, and so on until all cities are done.

2) Train data for all cities on date t, then data for all cities on t+1, then t+2 etc.

3) Something completely different.

Any thoughts?


texts:
I'm trying to train a CNN/LSTM classifier that I found online but keep running into the following issue. How do I fix it?

I'm not sure how to reshape the data to properly get the dimensions to work. The model is defined as follows for reference:

Please let me know how to get this to work, I've been stuck on it for hours!! Thanks :)


texts: LSTM Time series prediction for multiple multivariate series

texts:
I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and other related features of all cities to predict the next state. What should be the network architecture for such cases.

I was thinking of the following process:

Normalisation of the data with city specific Min,Max scalar
Feed sliding window data(t_1 to t_60) to LSTM and predict (t+1) value
Take the output value and get the actuals values from step 1.

I have read multiple papers and blogs online but mostly the deal with one multivariate time-series. But, in my case its multiple multivariate time series but one generalised model. Can someone suggest what are common industry practises for these and related papers/blogs. Do I need to model for each city (Its not possible in my case because of scalability issues?


texts: Grid Search Pool Classifiers for Dynamic Classifier Selection Ensembles

texts:
I would like to grid search pool classifiers hyper parameter of OLA() ( Overall Local Accuracy ) model from deslib python package.
from sklearn.datasets import make_classification

from sklearn.model_selection import RepeatedStratifiedKFold

from sklearn.model_selection import cross_val_score

from deslib.dcs.ola import OLA

from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import LogisticRegression

from sklearn.tree import DecisionTreeClassifier

from sklearn.naive_bayes import GaussianNB

Then :
X , y = make_classification( n_samples = 10000 , n_features = 20 , n_informative = 15 , n_redundant = 5 , random_state = 999 )

model = OLA()

cv = RepeatedStratifiedKFold( n_splits = 10 , n_repeats = 3 , random_state = 999 )

grid = dict()

grid[ 'pool_classifiers' ] = [ [ LogisticRegression() , DecisionTreeClassifier() , GaussianNB() ] ,
                               [ LogisticRegression() , DecisionTreeClassifier() ] ]

search = GridSearchCV( model , grid , scoring = 'accuracy' , cv = cv )

search_results = search.fit( X , y )

But the following error message is raised :
NotFittedError: This LogisticRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

It means model in the pool must be fitted prior to grid searching but I thought fitting took place on each train fold of the cv step.
Does it mean I must fit myself each model on training folds ?
Thanks for helping on this topic.


texts: How to determine threshold in Sigmoid function

texts:
Context:

I picked up data-set from here and tried to run Logistic Regression on it. Since I am not very much aware of MATLAB, I converted "Strings" to "Numbers" with my own using "NUMBERS" software.

What I want to achieve:

After running the LR algorithm when I tried to predict the value of existing data points, I am getting values ranging between 0-1 (as it should be), but since my job it to predict whether it is either 0 (yes) or 1 (no), that means I need to find a cut-off line (threshold) in my prediction (This could probably be done by comparing actual value by predicted value).

Question:

How can I figure out the threshold for predicted result such that result is assumed to be 1 if predicted value > threshold, otherwise 0?

Predicted values can be found here.
I am assuming predicted values are correct as cost curve is showing asymptotic nature. I have pushed my work here, you may want to cross-validate and provide me few more key points.


texts: Can anyone explain me the fisher score working

texts: I have been working on feature selection and I wanted to know what does fisher score tell us about the data which helps us in feature selection.


texts: I am getting very minimal mse values and not sure if it is correct?

texts:
Below is the linear regression model I fitted and not sure if I am doing the right way as I am getting neat to 99% accuracy

Fitting Simple Linear Regression to the Training set

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

ln_regressor = LinearRegression()
mse = cross_val_score(ln_regressor, X_train, Y_train , scoring = 'neg_mean_squared_error', cv = 5)
mean_mse = np.mean(mse)
print(mean_mse)

ln_regressor.fit(X_train, Y_train)

** MSE SCORE =-6.612466691367042e-06**

Predicting the Test set results

y_pred = ln_regressor.predict(X_test)

Evaluating accuracy of test data

mse2 = cross_val_score(ln_regressor, X_test, y_pred , scoring = 'neg_mean_squared_error', cv = 5)
mean_mse2 = np.mean(mse2)
print(mean_mse2)

**MSE score = -4.645751512870382e-31**

Please Note:
My data is in log scale & transformed to standard scaling later on

R2= cross_val_score(ln_regressor,X_test, y_pred,cv = 10)

R2.mean()

R2 mean is '0.9999030728571852'


texts: How to treat column with potentially meaningful NaNs

texts:
My data set has a column that indicates the time taken (in days) for members on a site - each with an ID - to sign up for an event. This can range between 1 to 300 days, with about half of the rows filled with NaNs for members who have yet to sign up for any events. Another column indicates the channel through which members initially joined the site, for example through direct referrals, a forum, or other websites.

The question is: If I'd like to try and estimate which groups I should target, as well as the best time, to send an e-mail reminder that increases the chances that members sign up for an event, what would be the appropriate approach if I have no labels for whether past e-mail reminders is successful or not? I was thinking of taking the average (median) of days taken to sign up for an event grouped by the channel through which members joined the site, but that would remove half the data which seems like a waste. Is there an alternative method?


texts:
I have a big data set of fake transactions for a company. Each row contains the username, credit card number, time, device used, and amount of money in the transaction. I need to classify each transaction as either malicious or not malicious and I am lost for ideas on where to start. Doing it by hand would be silly.

I was thinking possibly checking for how often a credit card is used, if it is consistently used at a certain time, or if it is used from lots of different devices (iOS AND Android, as an example) would be possible starting places. I'm still fairly new to all this and ML. Would there be some ML algorithm optimal for this problem?

Also, side question: what would be a good place to host the 600 or so GB of data for cheaps?

Thanks


texts: How to obtain the final values of a DecisionTreeRegressor in Scikit-Learn?

texts: This page shows the paths in the decision trees in scikit-learn. After reaching the leaf nodes of the decision tree, where do we obtain the final resultant value?


texts: Modelling data with Machine learning without a target variable

texts:
I want to understand the required steps that need to be taken into account while handling a dataset that does not have a target variable.
I can do machine learning on top of a labeled dataset having a target variable, but not sure what would be the best way to start with a dataset where is there is no target variable.

I need a step by step guide to achieve an efficient clustering at the end.

Do I need to do the following in order to achieve that?:

Data Cleansing
EDA
Encoding and scaling
Model build
Validation

Or are there any more steps that I need to take care of while dealing with an unsupervised class of data. I am doing this in python


texts: Correlation Study to Determine Weights Of Fields

texts:
I have several input fields, and the content for each field can either be correct or incorrect. These fields are then sent to a black-boxed function (which I can’t control), and the output of the function can either be correct or incorrect.
I want to do a regression study to determine how influential the correctness of each of the fields is in the correctness of the result, and then assign weights to each of the fields based on the study to maximize correctness of the output.
I have done simple studies of correlation between two variables before but do not know how to approach this problem. Could you give me some direction of the steps I would need to take? Once I have them, I can figure out how to do each. I work in Python.
Thanks in advance!


texts: Setting class weights for categorical labels in Keras using generator

texts:
I'm working on with an imbalanced dataset in Keras, and would like to give a higher weight to the samples from my minority class. The fit() function has a nice sample_weight argument, however because of the size of my data I have to use fit_generator().

fit_generator() has a class_weight argument, which seems useful for this purpose and is already discussed in Another question. However, in this case the labels are not one-hot-encoded/categorical and I could not find whether using class_weight also allows for categorical data.

Can use the class_weight argument for one-hot-encoded/categorical labels and if so how? Or do I have to resort to a custom weighted loss function?


texts:
I am looking for a way that could help me create more precise models.

Let's say these are real estate prices for different areas. Only in the data I do not have a clear division into these areas I suppose this is the relationship.

At present I have one model (red line) and I would like to have n models, eg 3. Additional two green lines. And use them for points that are closer to this line.

How convenient to go about it?

What measure should be used to divide this data and apply linear regression so that the variance is as low as possible?

May I have some inspiration :)?


texts: When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc?

texts: I have a data set with categorical and continuous/ordinal explanatory variables and continuous target variable. I tried to filter features using one-way ANOVA for categorical variables and using Spearman's correlation coefficient for continuous/ordinal variables.I am using p-value to filter. I then also used mutual information regression to select features.The results from both the techniques do not match. Can someone please explain what is the discrepancy and what should be used when ?


texts: How to pass database name as parameter to sql statement in python?

texts:
I have several SQL statements (update, delete, insert,...) in my Python code and all of them refer to one database. Now I want to pass the database name as a variable to my query so that I can change the database name one time in my code instead of manually updating the database name everywhere in SQL statements and also connection strings.

How can I pass Database name as a parameter to both SQL statement and connection string? for example in this code I want to pass DBNAME to connection string and SQL query instead of using the database name which is mainDB.

import pyodbc
DBNAME='mainDB'

cn = pyodbc.connect ( 'Driver={SQL Server};'
                    'Server=myserver;'
                    'Port=1433'
                    'Database=mainDB;'
                    'Trusted_Connection=yes;')
cursor = cn.cursor()

SQLquery = """ SELECT distinct [ID],[NAME]
                FROM [mainDB].[dbo].[customer]

            """
cursor.execute(SQLquery)
for row in cursor:
    print (row)
    break


texts: Automatic legal document reviewing advice

texts: I am lawyer from SK. I also like programming and Maths. Which technology should i use for this task? Automatic legal document reviewing. I have a big data set of specific contracts in slovak. Now I want to build my model. Model will tell good/bad contract + bold probable errors based on data set. I don´t know much about NLP, so I´am just asking for advice. Which tech should I study for this task? Word2Vec, keywords extraction algos etc. Thank you guys for your time.


texts: Working back and forth with DataFrame and nparray in Pipeline transformers

texts:
Little backstory : I've recently purchased the book Hands-On Machine Learning with Scikit-Learn and Tensorflow and I've started trying to apply what I've read so far in the book. Before this, I tested my knowledge on the Kaggle Titanic "competition". I now want to refactor my code to use some good practices I learned while reading the book.

At some point, the author talks about sklearn.pipeline.Pipeline. I started writing my own Transformers to populate my pipeline.

The problem I face is that, in some transformers, I'd use some pandas functions, in other numpy functions and I keep converting my dataset back and forth.

I did some research and din't find anything useful, it would be of great help if someone can answer the following questions:

Are there good practices concerning Pipeline?
Should I stick to pandas, numpy, is it a good practice to convert back and forth?


texts: Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

texts:
I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation.
When performing EDA I find missing values in the dataset. The missing data is completely at random and hence it's suggested to just remove the NA's.

Do I remove the NA's before performing univariate and bivariate analysis? Or do I remove NA's before modelling?  Is there a general rule?

I have 20 explanatory variables and one target. 2000 rows. The NA's are only present at 70 rows, for only 2 numerical variables. One variable has 20 missing values, and the other has 50. So no rows with 2 NA's.
I think I should remove the NA's only after performing the bivariate and univariate analysis, since this way I will have more info present for the variables which don't have any missing values. Because I will lose some information if I remove the NA's before the uni\bivariate analysis, and hence have a worse representation of the correlations.
But then if I'm planning on removing the NA's, why don't I just remove them, and perform EDA on the updated dataset, with which i'm planning to build my model?


texts: Support Vector classifier perform well with input features rather than transformed features in contrast to ANN-BP, random forest (other classifiers)

texts:
I am working on stock data with 5 raw features (OHLCV). Using few transformations used by technical analysts, have created 20 more features giving different kinds of indications.
When trying to predict the direction of the next few days, the behaviour shown by SVM is opposite to other classifiers. (SVM is giving better results when 5 raw inputs are considered while other methods (as expected) give better results when generated features are being used)

I completely understand that market data is very random and noisy. This is a theoretical exercise being done to understand the behaviours of different ML methods.


texts: Time series prediction using ARIMA vs LSTM

texts:
The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values. So far I have come across two models:

LSTM (long short term memory; a class of recurrent neural networks)
ARIMA

I have tried both and read some articles on them. Now I am trying to get a better sense on how to compare the two. What I have found so far:

LSTM works better if we are dealing with huge amount of data and enough training data is available, while ARIMA is better for smaller datasets (is this correct?)
ARIMA requires a series of parameters (p,q,d) which must be calculated based on data, while LSTM does not require setting such parameters. However, there are some hyperparameters we need to tune for LSTM.
EDIT: One major difference between the two that I noticed while reading a great article here, is that ARIMA could only perform well on stationary time series (where there is no seasonality, trend and etc.) and you need to take care of that if want to use ARIMA

Other than the above-mentioned properties, I could not find any other points or facts which could help me toward selecting the best model. I would be really grateful if someone could help me finding articles, papers or other stuff (had no luck so far, only some general opinions here and there and nothing based on experiments.)

I have to mention that originally I am dealing with streaming data, however for now I am using NAB datasets which includes 50 datasets with the maximum size of 20k data points.


texts: Guided topic modeling: generating words from topics

texts: I need to generate lists of words related to specific topics for a project. I am familiar with clustering methods of topic modeling such as LDA, but I have something else in mind. Are there any methods to generate lists of related words from a root word? For instance, where I could type "medicine" and it would return a list of words or terms related to medicine, such as "doctor", "malpractice", "office", etc.?


texts: What would be the best way to combine several denoised images and achieve superior denoising results?

texts: I have generated denoised images using several models and would like to create an ensemble of these individual images to achieve superior denoising results. What would be the best way to combine (averaging, max voting, weighted averaging, etc.) these denoised images to achieve a superior denoised image?


texts: Plotting scikit-learn confusion matrix returns no values in the last class

texts:
I am attempting to create a confusion matrix using Scikit-Learn for a multiclass classification CNN, and it works well except for the fact that it does not provide values for the last class (i.e. all of them are zero), as illustrated in the image below:

Here is the function that I used to generate the confusion matrix and plot:
# Function to Calculate Ensemble Confusion Matrix for Given Dataset
    def getConfusionMatrix(self, y_true, y_pred, speaker_labels, title):

        # Calculate Confusion Matrix
        cm = confusion_matrix(y_true, y_pred)

        print(cm)

        # Instantiate Plot Variables
        cmap = plt.cm.Blues # Color map for confusion matrix
        title = title # Plot title
        ticks = np.arange(len(speaker_labels)+1)
        fmt = 'd' # Data format
        thresh = cm.max()/2. # Treshold

        # Plot Confusion Matrix
        plt.figure(figsize=(15, 10))
        plt.imshow(cm, interpolation='nearest', cmap=cmap)
        plt.title(title)
        plt.colorbar()
        plt.xticks(ticks, speaker_labels, rotation=45)
        plt.yticks(ticks, speaker_labels)

        for (i, j) in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
            plt.text(j, i, format(cm[i, j], fmt),
                     color='white' if cm[i, j] > thresh else 'black')

        plt.ylabel('True Label')
        plt.xlabel('Predicted Label')

How can I ensure that the confusion matrix calculates TP/FP/TN/FN for the final class (i.e. ARA NORM in the image)? Is this possibly an issue with input into the confusion_matrix() function?


texts:
I am trying to classify 6 classes time-frequency domain signal (STFT spectrogram) with a size of 3601x217 pixels. Assume that for each classes have 70 training samples, 20 validation samples, and 10 test samples. Each sample is a PNG image of size 3601x217x1 (grayscale), while in the future I am thinking to expand this into 3601x217x4 (3 color channels + alpha). But right now, I am focusing on grayscale first.

After trying pure LSTM, I found out that the model overfits quickly (training accuracy > 90%, but val_acc stucks at 20%). ow I'd like to try out Time Distributed CNN+LSTM.

Following is the code:

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 6
nb_train_samples = 70
nb_validation_samples = 20
epochs = 50
batch_size = 5

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(32, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(64, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(units=128, return_sequences=False))
model.add(LSTM(units=64, return_sequences=False))
model.add(Dense(32))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

model.fit_generator(
    train_generator, steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs, callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

The code gives following error:

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py",
  line 496, in _make_train_function
      raise RuntimeError('You must compile your model before using it.')

  RuntimeError: You must compile your model before using it.

What's wrong with the code? I actually compiled the model before calling model.fit_generator:

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])


texts: To which category does this algorithm belongs?

texts:
I have came across the Catboost package. Among the classes in categories in Sklearn, Catboost seems to belong to Ensemble methods.
What are then the advantages of Catboost over AdaBoost, Bagging etc.?


texts: Is there any way to analyze the format of text strings?

texts:
I have a lot of data which basically consists of alphanumeric text on individual lines which can very in length and contain delimiters.
Since there are many thousands of lines of text, I'm looking to see whether there is an automated way to determine the different formats of text.
A sample of which is:
90665013-163
90731046-103
90840069-009
90847069-009
90880046-103
90889046-103
90897-051
9089744-103
9089844-103
90901-46909
90901-lep
9091046-103
9091046-909
90764046-1037
can10043E
can90065-op016
9094344-103
90669j4-4438718
90666ie79
90664046-103
90710-077
004-919
4A1900935
can90064-op016
can90066-E016
9094544-103
9094646-103
4A1900597
4A1900588
4A9443198
4A94431

So, from this sample, we can see that there are several lines that are the format of 8 numbers, a dash, and then three numbers.
Another format is 3 numbers, a dash, and then another 3 numbers.
I'm looking for a way to determine all of the unique formats available...
Not sure whether this is possible either by some code, an online web service, or perhaps a feature of some software but I'm asking on the off chance this actually possible.


texts: Predicting churn - deal with missing dates in time series and improve modelling result

texts:
This is the follow up question for General approach on time series for customer retention/churn in retail.

I have a time series of data in the following form:

| purchase_date    |    cutomer_id  |   num_purchases | churned |
   2018-10-31            id1              39             0
   2018-11-31            id1              0              0
   2019-01-31            id1              6              0
                         ...
   2019-03-31            id1              88            1
   2019-03-31            id2              300            0
   2018-04-31            id2               2             1
   2019-02-31            id3               1             1
   2019-07-31            id4               100           0
     ...                 id5

I grouped the data by month and summed num_purchases by month. The churned column for user id1 for example represents in which month customer churned. So id1 in my case churned in March. Before this, to label who has churned or not, we sampled customers based on 2 months of inactivity  period from the churn date. I need to predict if a user is going to churn in a 2 months from now.

I am getting very bad prediction results using logistic regression for example and the churned column as a class column. I suspect this is because some users  like id3 and id4 appear only once (or very few number of times) and other users like id1 appear a lot. I am not sure how to approach imputation in this case because these users just didn't exit before or after and I am not sure if it would make sense to impute them. Does anyone have idea on how to improve my model result?  I am getting 0.85 for accuracy, and 0 for precision, recall and F1.


texts: Face Recognition using Eigenfaces and SVM

texts:
I am new to machine learning. I want to develop a face recognition system using scikit-learn. This is the example given in the tutorials of scikit-learn.
I am not getting how the input is being provided to the program. How should I load a particular image and make my program run to predict the label for that?


texts: Different approaches for categorical non-ordered data clustering in R

texts:
I'm trying to find different clustering approaches for only categorical data in R, so far I found:

klaR for kmode
cba for rock
Hierarchical clustering (agglomerative or divisive) with a categorical data distance: gower.

In this question, they suggest kmode, which I've already included

I'm searching for different new approaches, as much as I could find.


texts: Character-based word representation using bi-lstm

texts:
In this blog, it teaches us how to get a word embedding using bi-lstm in character level like the image below:

I am wondering how to optimize the word vector from character-based approach. Is there any loss function or any strategy to update or optimize the word vector trained by a word's characters?


texts: Division of numbers from contours in opencv in python for cnn

texts:
I want to separate numbers in suppose 7638 into different images which can be predicted individually using cnn.
By finding contours how can I divide each contour into separate image in python.
To be more clear: How can I divide https://www.letter2word.com/products/americana-numbers into 0,1,2,3..9 into individual images using the concept of contours in opencv in python so that the resulting images can be predicted by cnn classifier and later all the results written together to give the number in image(initial) as output.
Also, is there any other method to do so.
PS.-I tried searching in google but everywhere it is given to extract 1 particular part of the image while I need more than 1 parts of the image for correct prediction


texts: How do you work with Latent Dirichlet Allocation in practice

texts:
One need to provide LDA with a predefined number of latent topics. Let say I have a text corpus in which I hypothesize there are 10 major topics, all composed of 10 minor subtopics. My objective is to be able to define proximity between documents.

1) How do you estimate the number of topics in practice ? Empirically ? With another method like Hierarchical Dirichlet Process (HDP) ?

2) Do you build several models ? For major and minor topics ? Is there a way to capture the hierarchical structure of the topics ?


texts: Getting to other parameter sets from GridSearchCV in sklearn

texts:
I am using GridSearchCV to train a dataset with several parameters.  Using the methods on the object I am able to get to the "best" parameter set (based on the scoring function).  I would like to be able to get to one (or more) of the other models.  Is there a way to do that?  Is that saved?  If so what is the way to get to those?

import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import GridSearchCV

bc = datasets.load_breast_cancer()
X,Y = bc["data"],bc["target"]

sgd = SGDClassifier(loss='log', penalty='l1', max_iter=1E6, tol=1E-6, shuffle=True, random_state=123456)
param_grid = [{'alpha':np.append(10.**-np.arange(3,6), 10.**-np.arange(3,6)/2)}]

sgd_gs = GridSearchCV(sgd, param_grid, scoring='roc_auc', n_jobs=4, cv=4)
sgd_gs.fit(X, Y)
results_pd = pd.DataFrame(sgd_gs.cv_results_).iloc[:,2:7].sort_values(by='rank_test_score').reset_index(drop=True)
results_pd

scores_train = sgd_gs.best_estimator_.predict_proba(X)[:,1]
print sum(scores_train)

#**code below does not seem to work**
params = results_pd.loc[2, 'params']
scores_train = sgd_gs.best_estimator_.set_params(**params).predict_proba(X)[:,1]
print sum(scores_train)


texts: Keras: confusion matrix with predict_generator

texts:
I'm not using ImageDataGenerator because i'm using hdf5 files. I used DataGenerator class (1) to feed data to the model.fit_generator. How can I get the confusion matrix given that I have batches of random data? I know how to get it with model.predict using sklearn.

(1) https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly#


texts: Should I prevent augmented data to leak to the test/cross validation sets

texts:
I have been working with the cats vs dogs dataset from kaggle which consist on 25000 images of cats and dogs labelled accordingly (btw, great dataset, totally recommended!)

One of the things I did was to augment the data by simply fliping the images 180 degrees, so an image like this

will became this

So far so good, now I have 50000 images, but the question is: should I make SURE that an image that has been augmented remains totally into the training set? I mean, with the previous two images, what would happen if I have one of them in the training set and another one in the test (or cross validation) set? does it mean that I am leaking training data into test data?

I understand that technically they are DIFFERENT images, but still my intuitions seems to resist such idea.

Am I correct to assume that augmented images should not leak to the test set?


texts: Machine Learning model to find items that are frequently bought together using Hadoop Spark

texts:
I need to do my master thesis under Big Data Analytics - Machine Learning subject. I was challenged to create a Market Basket Analysis project on retail industry.

My dataset contains transactions of a supermarket over a year and I will have to predict which products are associated.

The requirements for my project are:
- We must visualize the evolution of the model (Learnable Evolution Model)
- Must be done using Hadoop tool (+ Spark)

I already look for multiple source codes/tutoriais to learn how can I do this but I can't find a machine learning implementation project for find patterns using Spark...

Anyone have an idea how can I find some good tutorials to learn this skills for me to do my master thesis algorithm?


texts: how to represent location-code as a feature in machine learning model?

texts:
I am trying to predict the damage to a buildings after earthquake on a dataset which contains "district number" as feature. I think the feature will have a significant importance in predicting the label but I am not sure how to best represent it.

Any thoughts?


texts: Keras + Tensorflow CNN with multiple image inputs

texts:
I have a CNN that needs to take in 68 images that are all 59x59 pixels. The CNN should output 136 values on the output layer

My training data has shape (-1, 68, 59, 59, 1).

My current approach is to use concatenate to join multiple networks like so:

input_layer = [None] * 68
x = [None] * 68
for i in range(68):
    input_layer[i] = tf.keras.layers.Input(shape=training_data.shape[1:][1:])
    x[i] = Conv2D(64, (5,5))(input_layer[i])
    x[i] = LeakyReLU(alpha=0.3)(x[i])
    x[i] = MaxPooling2D(pool_size=(2,2))(x[i])
    x[i] = Model(inputs=input_layer[i], outputs=x[i])

combined = concatenate(x)

However, this always gives the error:

ValueError: A `Concatenate` layer should be called on a list of at least 2 inputs

Is this approach a suitable approach or am I doing this completely wrong?


texts: Non-linear Support Vector Regression issue - Sklearn Python 3.6

texts:
I am fairly new to Sklearn and machine learning and have encountered an issue when using SVR with an RBF kernel. Below is my predicted data compared directly with my real data:

I do not know what I am doing wrong and was wondering if any of you could help me.

Below is my code:

scaler=MinMaxScaler(feature_range=(0,1))
data=scaler.fit_transform(df['close'].values.reshape(-1,1).astype('float32'))

def split_data(data, test_size=0.2):
    split=len(data)-int(test_size * len(data))
    train=data[:split]
    test=data[split:]

    return train,test

xtrain,xtest=split_data(data)
ytrain,ytest=np.roll(xtrain,-1)[:-1].ravel(), np.roll(xtest,-1)[:-1].ravel()
xtrain,xtest=xtrain[:-1], xtest[:-1]

svr_poly=SVR(kernel='poly', C=1e3, degree=2)
svr_lin=SVR(kernel='linear', C=1e3)
svr_rbf=SVR(kernel='rbf', C=1e3, gamma=0.1)

lin_regr=LinearRegression()

svr_poly.fit(xtrain,ytrain)
svr_lin.fit(xtrain,ytrain)
svr_rbf.fit(xtrain,ytrain)

lin_regr.fit(xtrain,ytrain)

Thanks!

EDIT

The reason I only have one feature is because I want to include it in the following loop to predict prices given the past prediction:

def predict(iterations, data, buy_threshold=0.2, sell_threshold=0.2):

    preds=[data[-1]]
    lin_preds=[data[-1]]
    mean_preds=[]

    buysell=[]
    for d in range(iterations):
        pred=svr_poly.predict(preds[-1].reshape(-1,1))
        lin_pred=svr_lin.predict(preds[-1].reshape(-1,1))

        ### BUY/SELL/KEEP
        # MEAN AVERAGE OF LINEAR AND POLYNOMIAL PREDICTIONS
        mean_pred=(pred+lin_pred)/2

        # NEXT AVERAGE PREDICTION
        next_mean_pred=(svr_poly.predict(mean_pred.reshape(-1,1))+lin_regr.predict(mean_pred.reshape(-1,1)))/2

        if(next_mean_pred >= mean_pred*(1-buy_threshold)):
            buysell.append('BUY')
        elif(next_mean_pred <= mean_pred*(1+sell_threshold)):
            buysell.append('SELL')
        else:
            buysell.append('KEEP')

        ### APPEND PREDICTIONS TO LISTS
        preds.append(pred)
        lin_preds.append(lin_pred)
        mean_preds.append(mean_pred)

    gap=preds[0]-preds[1]
    for i in range(len(preds)):
        preds[i]+=gap

    return np.concatenate(tuple(preds[1:])), np.concatenate(tuple(lin_preds[1:])), np.concatenate(tuple(mean_preds)), buysell

Also here is my training data outputs plotted:


texts: How do you choose a kernel for a discontinuous function in Gaussian Process Regression?

texts:
I'm doing Gaussian Process Regression and created a series of functions by gluing other functions together on random places. Here's an example:

Perhaps this one is to complicated, but all the functions come from the same "family", they're all variations of gaussians. Is there anything standard that can be done with this?


texts: Best way to visualize huge amount of data

texts: I have a data set of around 3M row. I has only 2 category (category- 2:1 ratio). Now i want to visualize(scatter plot) it's distribution to understand can the data linearly separable or not(In order to choose model type).I already try this and the plot is not understandable. What will be the best way to visualize this data set?


texts:
I am trying to display a date in my result after running the below program in SAS. It runs properly but in the sas data table under DOB column I don't get anything except a period .

Below is my code what am I doing wrong?

data sample;
Input ID name $ Dob;
Format DOB mmddyy10. ;
datalines;
1 abc 22jan1996
2 xyz 25aug1996
;
run;
Proc print data = sample;
run;

Also I would like to know what does a period means at the end of this line before the semicolon:

Format DOB mmddyy10. ;


texts: Pecularities of classification of Hidden Markov Models?

texts:
Assume that we have set of Hidden Markov Models (Bayesian networks) Set{(n, m, P, A, B)} (n - number of hidden states, m - number of observable states, P - initial probabilities, A - transition matrix, B - emission matrix). Are the peculiar algorithms or approaches how to classify those HMMs into different subsets of HMMs? If n and m are equal for all the HMMs in the set then classification can be build from the ranges of elements in A and B. If n and m are different for different HMMs then one classify hidden and observable states first...

Are there some research or trends for this? Google is not helping here because it returns results for classification methods using HMMs, but we want to classify (or find clusters) of HMMs themselves.


texts: Sentence embeddings with LSTM to classify the sentences is not working

texts:
I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution works, on the similar lines I wrote the below code which uses Universal Sentence encoder to generate the embedding of the entire sentence and use that with LSTM NN to classify the sentences but it is not working even after 200 epochs the model doesn't converge.
Please find the code below
import tensorflow as tf
import keras
from keras.layers import Input, LSTM, Dense, Activation, Dropout,Embedding
from keras.models import Model
import pandas as pd
import tensorflow_hub as hub

encoder=hub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

word = "Elephant"
sentence = "I am a sentence for which I would like to get its embedding."
paragraph = (
    "Universal Sentence Encoder embeddings also support short paragraphs. "
    "There is no hard limit on how long the paragraph is. Roughly, the longer "
    "the more 'diluted' the embedding will be.")
messages = [word, sentence, paragraph,word, sentence, paragraph]
labels = [0, 1, 2,0, 1, 2]
reviews = tf.one_hot(labels, depth=3)
train_embed=encoder(messages)

embedding_layer = Embedding(6,512,trainable=False)
embedding_layer.build((None,))
embedding_layer.set_weights([train_embed])

input=Input(shape=(512,),dtype="float64")
X=embedding_layer(input)
X=LSTM(units=128,return_sequences=True)(X)
X=Dropout(rate=0.5)(X)
X=LSTM(units=128)(X)
X=Dense(units=3)(X)
X=Activation("softmax")(X)
model=Model(inputs=input,outputs=X)
model.compile(loss="categorical_crossentropy",optimizer="adam",metrics=["accuracy"])
model.fit(train_embed,reviews,epochs = 200,shuffle=True)

The out put looks like below
Epoch 196/200
1/1 [==============================] - 1s 534ms/step - loss: 1.1557 - accuracy: 0.3333
Epoch 197/200
1/1 [==============================] - 1s 501ms/step - loss: 1.0919 - accuracy: 0.5000
Epoch 198/200
1/1 [==============================] - 1s 518ms/step - loss: 1.2014 - accuracy: 0.0000e+00
Epoch 199/200
1/1 [==============================] - 1s 501ms/step - loss: 1.1008 - accuracy: 0.3333
Epoch 200/200
1/1 [==============================] - 1s 512ms/step - loss: 1.0509 - accuracy: 0.5000

Why is the model not converging? does sentence encoding doesn't work with LSTM?


texts: I'm learning ML from Bishop's book. But I don't know that How should I calculate w* in the below picture.


texts:
I am having around 1000 DAG(Directed Acyclic Graph) of different files showing java.io.BufferedReader usage. Following is representation of one of the graphs

digraph G {
  9 [ label="9 : ROOT:setup()#0" ];
  10 [ label="10 : START IF" ];
  12 [ label="12 : java.net.URL.openConnection()#1" ];
  11 [ label="11 : END IF" ];
  13 [ label="13 : java.net.URL.openConnection()#0" ];
  14 [ label="14 : START IF" ];
  16 [ label="16 : java.net.HttpURLConnection.setRequestProperty()#2" ];
  15 [ label="15 : END IF" ];
  17 [ label="17 : START IF" ];
  19 [ label="19 : java.net.HttpURLConnection.addRequestProperty()#2" ];
  18 [ label="18 : END IF" ];
  21 [ label="21 : java.net.HttpURLConnection.setReadTimeout()#1" ];
  22 [ label="22 : java.net.HttpURLConnection.setConnectTimeout()#1" ];
  23 [ label="23 : java.net.HttpURLConnection.setUseCaches()#1" ];
  24 [ label="24 : <static>java.net.HttpURLConnection.setFollowRedirects()#1" ];
  25 [ label="25 : START IF" ];
  27 [ label="27 : java.net.HttpURLConnection.setRequestMethod()#1" ];
  28 [ label="28 : java.net.HttpURLConnection.setDoOutput()#1" ];
  29 [ label="29 : java.net.HttpURLConnection.setDoInput()#1" ];
  30 [ label="30 : java.net.HttpURLConnection.getOutputStream()#0" ];
  31 [ label="31 : java.io.DataOutputStream.<init>()#1" ];
  32 [ label="32 : java.io.DataOutputStream.writeBytes()#1" ];
  33 [ label="33 : java.io.DataOutputStream.flush()#0" ];
  26 [ label="26 : END IF" ];
  34 [ label="34 : java.net.HttpURLConnection.getInputStream()#0" ];
  35 [ label="35 : java.io.InputStreamReader.<init>()#1" ];
  36 [ label="36 : java.io.BufferedReader.<init>()#1" ];
  9 -> 10;
  10 -> 12;
  12 -> 11;
  10 -> 13;
  12 -> 13;
  13 -> 11;
  11 -> 14;
  14 -> 16;
  12 -> 16;
  13 -> 16;
  16 -> 15;
  15 -> 17;
  17 -> 19;
  12 -> 19;
  13 -> 19;
  16 -> 19;
  19 -> 18;
  18 -> 20;
  16 -> 20;
  19 -> 20;
  20 -> 21;
  12 -> 21;
  13 -> 21;
  19 -> 21;
  21 -> 22;
  12 -> 22;
  13 -> 22;
  16 -> 22;
  19 -> 22;
  22 -> 23;
  12 -> 23;
  13 -> 23;
  16 -> 23;
  19 -> 23;
  21 -> 23;
  23 -> 24;
  24 -> 25;
  25 -> 27;
  12 -> 27;
  13 -> 27;
  16 -> 27;
  19 -> 27;
  21 -> 27;
  22 -> 27;
  23 -> 27;
  27 -> 28;
  12 -> 28;
  13 -> 28;
  16 -> 28;
  19 -> 28;
  21 -> 28;
  22 -> 28;
  23 -> 28;
  28 -> 29;
  12 -> 29;
  13 -> 29;
  16 -> 29;
  19 -> 29;
  21 -> 29;
  22 -> 29;
  23 -> 29;
  27 -> 29;
  29 -> 30;
  12 -> 30;
  13 -> 30;
  16 -> 30;
  19 -> 30;
  21 -> 30;
  22 -> 30;
  23 -> 30;
  27 -> 30;
  28 -> 30;
  30 -> 31;
  31 -> 32;
  32 -> 33;
  31 -> 33;
  33 -> 26;
  26 -> 34;
  12 -> 34;
  13 -> 34;
  16 -> 34;
  19 -> 34;
  21 -> 34;
  22 -> 34;
  23 -> 34;
  27 -> 34;
  28 -> 34;
  29 -> 34;
  30 -> 34;
  34 -> 35;
  35 -> 36;
}

I want to cluster these graphs.I am trying to implement K-Medoids.

How to start with K-Medoid implementation?
How can i find distance between two graphs?


texts: Recreating sklearn linear regression from coefficients and intercept

texts:
I am attempting to write my own linear regression function using the coefficients and intercept achieved using the sklearn LinearRegression model. I have 11 features. I am using model.coef_[0] to extract the coefficient for feature 1 etc.. When I recreate the regression function the results are vastly different.

Does model.coef_[0] always correspond to the coefficient for feature 1 etc or do I need to do something to check this? Any tips on this would be greatly appreciated

Update

This was my mistake. I copy and pasted the values directly from print(coef_) and this resulted in a rounding error!


texts: How does ResNet bottleneck architecture's input size is possible to change from 56x56x64 to 56x56x356?

texts: In ResNet papaer, First residual block's input size is 56x56x64 caused by 7x7x64 filter in first layer. But, in the paper, they showed residual block that has 56x56x256 input size. How does it is possible to change from 56x56x64 to 56x56x256?


texts: Keras most important features for text classification

texts:
I am working on a problem where I need to classify phrases in one of the two categories (let's A & B). I used the Keras SepCNN model (similar to this) for that and it is giving me some results.

Now, I want to analyse the predictions and more specifically I want to know why the model classified a certain phrase in category A or B, which set of features played an important role in labeling that phrase as category A or B?

I am not sure if this is even possible to do but need inputs on how to approach this.

#----Model Structure for your reference--
Vectorize text (tfidf)
SelectKBest features (using scikit-learn feature_extraction)
Fit the model
Predict on test set*

I looked on the internet and found nothing helpful. It would be very helpful if someone can point me the right direction.
I also explored eli5 library but that does not seem to work for Keras SepCNN model (at-least I could not get it to work, let me know if anyone has used successfully it?)


texts: Linear regression with non-symmetric cost function?

texts:
I want to predict some value $Y(x)$ and I am trying to get some prediction $\hat Y(x)$ that optimizes between being as low as possible, but still being larger than $Y(x)$. In other words:
$$\text{cost}\left\{ Y(x) \gtrsim \hat Y(x) \right\}  >> \text{cost}\left\{ \hat Y(x) \gtrsim Y(x) \right\}  $$

I think a simple linear regression should do totally fine.
So I somewhat know how to implement this manually, but I guess I'm not the first one with this kind of problem. Are there any packages/libraries (preferably python) out there doing what I want to do?
What's the keyword I need to look for?

What if I knew a function $Y_0(x) > 0$ where $Y(x) > Y_0(x)$. What's the best way to implement these restrictions?


texts: What features used by CNN model should a feature store actually store?

texts:
According to MLOPs principle, it is recommended to have a feature store. The question is in the context of doing image classification using deep learning models like convolutional neural networks which does automatic feature engineering(using convolution layers) as part of the training process.
Questions

Does it make sense to have a feature store for pure image classification/segmentation models?
What features should be stored in the feature store? Output of convolution layers? but then they cannot be reused during the training since during training these will be rebuild by the convolution layers.


texts: Training Gaussian Restricted Boltzmann Machines with Noisy Rectified (nrelu or ssu) linear hidden units

texts:
I'm not sure how to implement this architecture. I'm following this thesis (pages 17-19) or this paper but I'm not sure how to train it. I want to use this to extract features from raw audio.
I know I have to compute the positive and negative correlations, but I don't know how to do this exactly since I can not find any detailed documentation of this.
What I have done so far is:
Positive correlation
To compute it I do a matrix multiplication:
$$\langle xh\rangle^+ = \frac{x_{data}^Th_{act}}{batchsize}$$
Where $x^T_{data}$ is be my batch data transposed and $h_{act}$ are the hidden units activations. $h_{act} = relu(x_{data}W + bias_{hidden})$. Not sure if I have to add noise in this point.
Negative Correlation
To do this I use contrastive divergence (no tag??) with $k=1$
Same thing:
$$\langle xh\rangle^- = \frac{x_{free}^Th_{free_{act}}}{batchsize}$$
but I'm really not sure how I have to compute these terms.
So for me $x^t_{free}$ would be the samples of $\mathcal{N}(ΣWh+b;Σ)$, where $h$
here is activation from previous step ($h_{act}$) without noise.
Lastly $h_{free_{act}}$ is computed as the same as the previous $h_{act}$ but using $x^t_{free}$
Is this correct? Because I don't know when I have to use samples or activations or maybe meanfields


texts: Expand Seaborn FacetGrid layout horisontally

texts:
I have the following code that shows 10 dataframe column values separated by categories PREFERENCE = GOOD and PREFERENCE = BAD.
Is it possible to expand the layout horisontally so that instead of 2 images on one row there are four of them (i.e. ACOUSTIC and DANCE)?

for audio_feature in ['ACOUSTIC', 'DANCE', 'DURATION', 'ENERGY', 'INSTRUMENTAL', 'LIVE', 'LOUD', 'SPEECH', 'TEMPO', 'VALENCE']:
    count, bin_edges = np.histogram(features_df[audio_feature])
    g = sns.FacetGrid(features_df, col='PREFERENCE')
    g.map(plt.hist, audio_feature, color="steelblue", bins=bin_edges) #color="coral"
    plt.show()


texts: Which memory usage is more accurate in nvidia-smi?


texts: Trying to analyze market data in R, but companies continually show very low average growth over large time intervals

texts:
I think the title explains it all.
In more detail, I'm using the data set from this Kaggle page.
The data set doesn't come with a daily change in percent, so I copied the dataset and made a new column for the daily percentage change, which I calculate by $$\Delta daily = \frac{Opening Price-Closing Price}{Opening Price}.$$
I went to do a basic sanity check, and make sure that nothing is going wrong with my column for daily percentage change.
I first picked a few dates and companies and calculated the $\Delta daily$ by hand, then compared it to the value my code gives; everything matched up there. However, I then tried to look at the mean $\Delta daily$ for a few stocks that have been wildly successful over the last few years (Facebook, netflix, amazon, etc). When I take the average $\Delta daily$, a lot of these companies are coming up with an average daily percentage change that is very near zero, or even slightly negative. Worth noting, I tried taking the average growth over several different time intervals; I'd even restrict myself to stretches of time where a stock had very few dips, but still the average $\Delta daily$ for the restricted time windows. For example AMZN was selling at roughly 300\$/share in january of 2015 and roughly 700\$/share in january of 2016, but when I look at $\Delta daily$ for this period of time, I get $-.0003913$. More than double the growth over the year, but an average daily change of $-.4$%? That seems way off.
I began wondering if there was something wrong with the dataset, but then I chose a few of the tech giants and plotted their growth against time over the last few years, and the graph is a near perfect resemblance of the growth chart that I see elsewhere, so the data seems to be fine.
I'm wondering if anyone has any thoughts on what's going on here. Is my intuition just wrong? Is it possible to have an average daily return that's near zero or negative, but still very strong growth in the long run? Is there something about the way I'm building my percent.delta vector that's incorrect?
There's really not much code to share, but here's what I have, in case it's useful. Sorry if it's a little sloppy; I was just messing around with some stuff and so didn't make it as tidy as I would for a more formal project.
Thanks in advance for any thoughts.
allstocks<-read.csv(file='all_stocks_5yr.csv')
allstocks2<-read.csv(file='all_stocks_5yr.csv') #copy data to alter
percent_delta <-as.vector(numeric(619040)) #no. observations; vector that will contain daily %change

for (i in 1:619040){
  percent_delta[i]<-(allstocks2$open[i]-allstocks2$close[i])/allstocks2$open[i]
} #fill vector with daily %change

allstocks2$percent.delta<-percent_delta

#just a check that trends in R match trends seen elsewhere. Can use smth other than NFLX
plot(subset(allstocks2, Name=="NFLX")$date, subset(allstocks2, Name=="NFLX")$close)

#avg percent.delta for, e.g., AMZN between, e.g., jan2015-jan2016
mean(subset(subset(allstocks2,Name=="AMZN"),as.Date(date)>=as.Date("2015-01-23")
& as.Date(date)<=as.Date("2016-01-04"))$percent.delta)
#avg percent_delta for, e.g., AMZN between, e.g., jan2015-jan2016


texts: Installing Prototype Widgets in Orange running on Mac

texts:
I am having trouble downloading some add-ons for the Orange data mining software that are not available through the normal add-ons menu. To do this, I am attempting to install a particular set of add-ons from the GitHub page here.
I am very novice when it comes to using Terminal and terminal commands. Could anyone help provide some insight on how to install this package?
Some people have suggested that they should appear in the prototypes list, but they do not. Does anyone have Orange running on Mac so that they can try this to see if it works?
Specifics for the exact code that I need to implement in terminal to get this to work would also be helpful.
Thanks!


texts: How to calculate perplexity of language model?

texts:
In one of the lecture on language modeling about calculating the perplexity of a model by Dan Jurafsky in his course on Natural Language Processing, in slide number 33 he give the formula for perplexity as

Then, in the next slide number 34, he presents a following scenario:

"If a system has to recognize
• Operator (1 in 4)
• Sales (1 in 4)
• Technical Support (1 in 4)
• 30,000 names (1 in 120,000 each)"

Perplexity in this case is 53.

Can anyone explain how the answer 53 came ?


texts: Why does my LSTM perform better when randomizing training subset vs. standard batch training?

texts:
I am training a simple LSTM network using Keras to predict time series values. It is a simple 2-layer LSTM.

I get the best performance when I train on subsets of the training set that start at random points. Each subset has a training size of 100 samples and a validation size of 30 samples. At each sample the model has a batch size of 16, trains for 100 epochs with an early stop after 20 epochs of little improvement. I run this training 10 times.

How is this different from me training me simply training my model as such:

model.compile(loss='mean_squared_error', optimizer='adam')

results = model.fit(X_train, y_train, epochs=100, batch_size=32, shuffle=False,
                validation_split=0.3, verbose = 1, steps_per_epoch=10,
callbacks = [EarlyStopping(monitor='val_loss', min_delta=5e-5, patience=20, verbose=1)])

Doesn't Keras automatically select random subsets of the training data for training and validation through the mini batches? Is it simply because I am training the model so many more times?

Is this effectively a k-fold-esque training implementation?
However, using sk-learn's KFold or Repeated Kfold doesn't get results as good.

NB: I do not want to use shuffle as this is time series data and that would distort the model training.


texts: Can we use median to replace all the missing values from a column

texts: If we can use the median to replace all of the missing values from a column, then what is the advantage of using median over mean for replacing the missing values in a column with numeric values?


texts: mathematical accurate definition of the binary independence model

texts:
I have a hard time understanding the exact mathematical meaning behind the binary independence model. On wikipedia we can see the following definition  or similarly in the book from Manning and Schütze,

it claims that

  The probability P(R|d,q) that a document is relevant derives from the probability of relevance of the terms vector of that document P(R|x,q). By using the Bayes rule we get:

$P(R|x,q) = \dfrac{P(x|R,q)P(R|q)}{P(x|q)}$

Now, Bayes rule is as follows:
$P(A|B) = \dfrac{P(B|A)P(A)}{P(B)}$

If you set $A := R$ and $B := x,q$ you get:
$P(A|B) = P(R|x,q) = \dfrac{P(x,q|R)P(R)}{P(x,q)}$

If you compare the terms, you'll notice why I am confused:

$x=x,q$
$R,q=q$
$R|q=R$
$x|q=x,q$

This result has nothing to do with the initial claim.
I think that I miss the definition of the 'comma' in this context. I am not aware of multi-dimensional probabilities. As I understand, a probability $P$ is always defined over a $\sigma$ algebra of an event space $\Omega$

In order to understand what's the idea behind the formula above, here a few things that could help:

What does the comma  precisely mean in the formula above (in mathematical notation)?
What is the underlying $\Omega$ ? If there is a probability $P$, then there must be an Omega $\Omega$ which serves as the space on which we define probabilities. It's not clear at all what this space is. If a document is a vector $x \in \{{0,1}\}^n$ and the query is a vector $q \in \{{0,1}\}^n$, then defining a space like $\Omega := \{{0,1}\}^n \times \{{0,1}\}^n$ could make sense. In this case it's not clear what $R$ is. Maybe the intent is to use $\Omega := \{{0,1}\}^n \times \{{0,1}\}^n \times \{{relevant, nonrelevant}\}$
Do $R$, $x$ or $q$ have anything to do with random variables? If yes, then it would help to see their domain, e.g: $R : \Omega \mapsto {0,1}  $
Because the conditional probability is defined between 2 sets, $R$ and $x,q$, as well as $x$ and $R,q$ or $R|q$ should represent sets. If $R$ is a random variable, then maybe in the formula above the term $R$ represents the set $\{\omega \in \Omega\ | R(\omega)=relevant\}$. What is then the set for $x$ or for $q$ ?


texts: Dueling DQN - Advantage Stream, why use average and not the tanh?

texts:
For Dueling DQN (page 5), why do authors use an average for Advantage stream, and don't simply "activate" the Advantage stream (with a $tanh$ for example)?

Would "activating" work in theory, and is it a similar idea to what the authors intended to achieve, or am I missing the point?

To remind, this is the equation, which Produces a Q-value for the taken action $a$ by combining the Value stream with the Advantage stream:

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

where

$s$ is the current state we are in
$a$ is the action we've decided to take
$a'$ is one of any action we could have taken (including the action we've taken)
$\theta$ are parameters (weights) of the network before the "splitting" into two separate streams
$\alpha$ are the parameters of Advantage stream
$\beta$ are the parameters of the Value stream


texts: What loss function does the 'multinomial' distribution with the gbm package in R use?

texts: All distributions in the gbm package in R are associated with a loss function. For example, when we set distribution = 'binomial', the loss function chosen internally is the logistic loss function. Can anyone explain how multi-class classification works with gbm and the loss function that is being used for it i.e. when we set distribution='multinomial'? Is it using one-vs-all or all-vs-all under the hood for doing its multi-class classification?


texts: Doc2vec(gensim) - How to calculate the most similar sentence and get its label?

texts:
I'm trying to use doc2vec(gensim) to identify the most similar sentence and get its label. That is, for example, when the data is composed of 36 types of TVs (each sentence explains a specific product and its labeled to that product), the doc2vec categorizes the user input and decides what TV the user is referring to.

I only know how to get the most similar word: model.most_similar('red/noun')
How can you, instead of words, get the most similar sentence and its label?

Doc2Vec - How to label the paragraphs (gensim) (this tells that the above method is actually possible in doc2vec)

Thank you :)


texts: I read this paper and understood it very well, but I miss negative impact that will happen if we don’t fix the 5 types of messy data introduced in the paper (page 5), or even how fixing them will make the analysis easier. Can any one give examples for this?


texts: Reading values from a column into a variable and then correlating using Python

texts:
I want to be able to correlate values from various IDs, where the date is the same with one another.

The data looks something like this;

ID      Time(secs)   Date
AAAA    1            01/01/1990
AAAA    6            02/01/1990
AAAA    5            03/01/1990
AAAA    2            04/01/1990
AAAA    4            05/01/1990
BBBB    2            01/01/1990
BBBB    4            02/01/1990
BBBB    6            03/01/1990
BBBB    3            04/01/1990
CCCC    3            01/01/1990
CCCC    4            02/01/1990
CCCC    1            03/01/1990
CCCC    6            04/01/1990
DDDD    7            01/01/1990
DDDD    4            02/01/1990
DDDD    5            03/01/1990
DDDD    3            04/01/1990

I want to find the correlation coefficient between each combination of these IDs, where there are matching dates. N.B. Not all of the IDs have the same date;

ID             CorrCoef
AAAA>BBBB      ????
AAAA>CCCC
AAAA>DDDD
BBBB>CCCC
BBBB>DDDD
CCCC>DDDD

I think I need to feed the data from each ID into a variable and then run the following;

data1.corr(data2)


texts: Is "adding the predictions to the real data for new training and prediction" a good idea for LSTM?

texts:
Considering we have trained our model with a lot of data for "many-to-one" prediction. Then we like to forecast the future data of next 10 days. So we use last 60 of existent data and predict the single next day. From here there are 2 approaches:

We can put our model.predict() function in a for loop for 10 times and do predictions like this(adding our predictions to end of our real data).
We can put all of our model(consisting training part, not just predict part), in a for loop and this means we train our model 10 times whenever we do a new prediction and adding it to our real data.

EDIT:
Thinking you have X_train = (100,60,1) array that means 100 examples, 60 time-steps(hidden units) and 1 feature for each example. Also you have y_train array of size (100,1,1) that means 100 labels with time-steps = 1 and 1 feature. Then you train your network to read 60 of inputs and predict the next single output. Also you create a X_test array like this: X_test = X_train[len(X_train - 60):] that means you use last 60 numbers of your series to predict the next number.  So you use the new_number = model.predict(X_test) for that and you predict the time-step 61 that is not a real number. It's your prediction. Then you want to continue your predictions. So what do you do is adding the 61'th predicted number to the last of your X_test = np.append(X_test, new_number) and do new number = model.predict(X_test) again. But the difference is that the last number in your new X_test is your previous prediction. And you keep this way for 10 times to predict 10 next numbers. (This was the first approach).

The other approach(2) has a difference. After doingnew_number = model.predict(X_test) for the first time, you add the predicted number to x_train instead of X_test, like this X_train = np.append(X_train, new_number) and train your model again model.fit(X_train , y_train) with the new predicted number. Then you use new number = model.predict(X_test) and again adding predicted number into the X_train, then train your model again(this time, with 2 new predicted numbers that you have added to the end of your X_train) and so on for 10 times!


texts: What skills do I need to become a data scientist? And how to show them?

texts:
I have fimiliarized myself with the recommended most important concepts (Linear Algebra, Analysis, Phython, Numpy, Pandas, a bit of Statistics, Linear regression). For the last two, I don't know how deep it should go. I know what things mean and how to get them working in python.

But the question is what now? I guess I could argue that this is a starting point and I can apply to a bad data analysis or visualisation position if I learn tableau and present myself well. But what would I do to even prove what I can do before an interview? Putting a notebook on github where I imported a dataset, cleaned it a bit, did a .desribe(), .plot() and a linear regression isn't very impressive nor interesting to anyone. So what would I do instead?

Also, this clearly isn't data science area yet. If I look at kaggle challenges, I either don't know what to do or think to myself "Clean data, LinRegression". So what should I take a look at next?

Note that I'm taking classes, but not in Data Science but in Chemistry right now.


texts: Amount of data needed for deep learning vs support vector machine

texts:
I often read about the fact, that the amount of data to train and get a generalizing model for a deep learning algorithm is much higher in comparison, e.g. to a support vector machine. It makes sense, because of the huge amount of parameters in a deep learning approach, which potentially leads to overfitting.

However: Are there any systematic studies on this? Do deep learning approaches really need more data?

Best regards,
Gesetzt


texts: R - time series decomposition without detection of seasonality

texts:
I have a time series dataset with 200 data points. I have decomposed it using the function below:

dat2 = ts(dat1, frequency = 4)
decomposeDat = decompose(dat2, "multiplicative")

I get 4 components: trend, seasonal, cyclic and irregularity. But when I check if there is seasonality present in the dataset with frequency "4", Rstudio says that there is no seasonality for this frequency. The check is performed with the following code:

dat2 = ts(dat1, frequency = 4)
fit <- tbats(dat2)
seasonal <- !is.null(fit$seasonal)
seasonal

seasonal returns FALSE meaning that there is no seasonality with frequency 4.

Can someone explain, why can I decompose it into a seasonality component when no seasonality is present from the check mentioned above?


texts:
Do you think it's normal for data science projects to have some amount of "what should we do" -time? Or does it mitigate by experience?

By "what should we do" -time I refer to time being spent on reading about and experimenting on "possible ways to do things, when many alternatives exist". This kind of "time" has bothered me, because someone could think it's some kind on inefficiency or failure to concentrate.

On the other hand I've rationalized that it could be a normal part of a data science project, because projects may be different, they may have special considerations or boundaries that are different from other projects. This would lead to time having to be allocated to "particularizing from general and other solution methods". Thus also, to expect that one would instantly understand the "best way" to solve a problem, would be too optimistic and forget that "one may not know, without experimenting, what the best solution is". Since a data science project is not necessarily an "exact mathematical problem".


texts: Find the Weight Values using Linear Regression Analysis in Python

texts:
I have the equation below, which is a noise quality metric of an image:

If the BIQS is 1, it means the image is clean. Else, if it is between 0 and 1, it means the image may contain blur and noise. The smaller the value of BIQS, the dirty is the image. The task is to get the weights value for w1, w2,w3 and w4, given the set of values Blur_mean, Blur_noise, Noise_mean and Noise_ratio, which reflects the image quality. The weights reflects how much each term (Blur_mean, Blur_noise, Noise_mean and Noise_ratio ) contributes to the total image quality score. For Example:

     Blur_mean     Blur_noise    Noise_mean       Noise_ratio
1.   0.8398212     0.8948329     0.9849308        0.89428493
2.   0.8989332     0.8989432     0.7576812        0.35546632
3.   0.4324123     0.3123232     0.2123332        0.33213233
n.   .........     .........     .........        ..........

I don't know the basis for identifying the appropriate values or weight for w1,w2,w3 and w4. Is it possible to use linear regression analysis to find the values of w1, w2,w3 and w4 with almost/ close to 1 R-square or small standard error ? From what I observe, there are 4 independent variables here:

`Blur_mean`
`Blur_noise`
`Noise_mean`
`Noise_ratio`

Dependent Variable is BIQS and the unknowns are w1, w2, w3 and w4. Is there any library that takes care of it in Python?


texts: Datasets for software defined network (SDN)

texts:
I am working on a project with Software Defined Networks (SDN), and new to SDN area. I was checking around internet to find a benchmark dataset, which is a flow based dataset inorder to apply ML techniques. I havenot found one yet. SDN is  a TCP/IP based network. But SDN offers a way to collect the statistics from dataplane devices. I am searching whether there is a flow based statistics benchmark dataset that is publically available for SDN research. I can find the packetcaptures over traditional networks, but I am looking for flow based datasets collected over SDN.
The other option is to simulate a network and inject traffic with tools, but the datasets created like this has less traffic diversity. Another option is to create flows from the existing packet captures, but this needs  packet parser.


texts: Using neural network for "features matching" binary classification

texts:
We have a dataset of numerical features from two images and we want to check if these images match or not using only these features.
Basically we have have these columns:

fA1, fA2, ..., fA14: 14 features from image A
fB1, fB3, ..., fB14: 14 features from image B

We want to predict if image A match image B (y=1) or not (y=0). So it's a features matching problem.

The main usecase is for face recognition using this framework:
BERND HEISELE

So is there any neural networks architecture known for this situation (we have a 7 million annotated training set) ?

N.B: we don't have any images, we have only numerical features.


texts: Displaying network error as a single value

texts:
I've been writing a neural network from scratch. I've completed the feedforward, backpropagation, and mini-batch gradient descent methods, so I can train the network. Other neural networks I've worked with usually display the error/loss after each batch as a single decimal value, and I'd like to implement this functionality but I'm not sure how.

I understand squared error is given by $(y - \hat{y})^2$, and that for an output layer with $m$ neurons, you should have an error vector of size $m$. However, how is the error vector displayed as one value?


texts: how to update column in data frame based on condition

texts:
How to update column IsLCap column in dataframe based on Lvalue column whether it is capitilized or not.

df.loc[df.Lvalue.istitle(), 'IsLCap'] = 1  # need to be corrected

getting an error -AttributeError: 'Series' object has no attribute 'istitle'


texts: Is AUC a good metric for evaluating the performance of a multi-class classification?

texts: Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?


texts: Can I use SVC() as a base_estimtor for ensemble methods?

texts: I am currently testing out a few different ensemble methods on my dataset. I've heard that you can also use support vector machines as base learners in boosting and bagging methods but I am not sure which methods allow it or not. In particular, e.g. for XGB i tried out trees and SVMs as base learners and got the exact same result for 5 different performance metrics which made me question the results and/or that the option can only take trees as base learners. I didn't find much info in the documentation or at least not in all of the documentations. I would be interested about AdaBoostClassifier(), BaggingClassifier() and XGBClassifier(). Does anybody know the details and whether or not I can use SVMs here as base learners?


texts: What is "Policy Collapse" and what are the causes?

texts:
I saw the term "policy collapse" on the comments of a tutorial for  reinforcement learning.

I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad hyper-parameter, be it the learning rate, batch size, etc., but I couldn't find anything explaining it in clearly and in detail.